strix/TESTING_STRATEGY.md

Migration Testing Strategy

Seven-layer safety net to prove the SDK-migrated harness produces behavior identical to the legacy harness, with no silent feature loss. Centered on behavioral parity diffing (the only test that doesn't require knowing in advance what could break) and a feature inventory matrix (every feature has a row, every row has a test, no row → no proof).

---

Table of contents

1. Threat model — what we're afraid of
2. Testing layers (cheap → expensive)
3. Feature inventory matrix
4. Behavioral parity diffing — how it actually works
5. Replay infrastructure (deterministic LLM + sandbox)
6. Live shadow mode (production-grade canary)
7. Per-correction test mapping
8. CI gating and cutover criteria
9. Manual smoke checklist
10. Post-cutover monitoring
11. What to do when a parity diff fails

---

1. Threat model

Concrete categories of regression we want to catch — every test below targets at least one row.

#	Threat	Detection difficulty	Where it hides
T1	A tool stops being invocable (typo, registration miss)	Easy	Agent runs, never calls it
T2	A tool's args don't validate the same way	Medium	LLM gets different error string; no exception
T3	A tool's output format changes (XML vs structured, truncation cap, screenshot extraction)	Hard	Findings still happen, but different shape
T4	An LLM provider quirk lost (Anthropic cache, Bedrock timeout, vision strip)	Hard	Cost+latency drift; no functional break
T5	A side effect lost (wiki note auto-update, vulnerability dedup, PII scrub)	Critical-silent	Reports persist but a feature stops firing
T6	An event type stops being emitted to events.jsonl	Hard	Run completes, dashboards have gaps
T7	Cancellation cascade incomplete (Ctrl+C leaves orphan tasks)	Medium	Looks fine on success path; only visible on cancel
T8	Memory leak / resource leak (orphan inboxes, stale sessions)	Hard	Long runs degrade
T9	Concurrency regression (parallel tools collide, message ordering breaks)	Critical-silent	Pass once, fail on second concurrent run
T10	Schema drift in persisted artifacts (events.jsonl, vuln JSON, report MD)	Medium	Downstream consumers break
T11	Config / env var stops being read	Easy	Default kicks in instead of user override
T12	Exit-code change	Easy	CI integrations break
T13	TUI / CLI output format change	Easy	User experience drift
T14	A skill / prompt section silently dropped	Critical-silent	Agent's behavior subtly worse on specific vuln types
T15	Subagent crash silent (parent waits forever)	Critical-silent	Run hangs without diagnosis
T16	LLM provider routing wrong (strix/foo → wrong model)	Easy	API error within seconds

Critical-silent rows are the dangerous ones — the run looks fine but a feature is gone. Layer 4 (parity diffing) is specifically designed to catch these.

---

2. Testing layers

Bottom-up. Each layer catches a different class of bug; together they're complementary.

┌────────────────────────────────────────────┐
│  Layer 7: Manual smoke (humans, TUI, etc.) │  rare, high signal
├────────────────────────────────────────────┤
│  Layer 6: Live shadow / canary             │  prod-grade, expensive
├────────────────────────────────────────────┤
│  Layer 5: Recorded replay (deterministic)  │  end-to-end, reproducible
├────────────────────────────────────────────┤
│  Layer 4: Behavioral parity diffing        │  **most powerful**
├────────────────────────────────────────────┤
│  Layer 3: Integration (modules together)   │
├────────────────────────────────────────────┤
│  Layer 2: Unit (one module, mocked deps)   │
├────────────────────────────────────────────┤
│  Layer 1: Static (mypy, ruff, signatures)  │  cheapest, fastest
└────────────────────────────────────────────┘

Layer 1 — Static / pre-runtime

Catches type-level mismatches without running the code.

• mypy --strict against strix/ after migration. With openai-agents[litellm]==0.14.6 installed, mypy verifies our subclasses honor SDK's ABC contracts. Catches: F1, F2, F3 type-fix regressions, future SDK signature drift.
• ruff + pyright as secondary type checkers; they sometimes catch what mypy misses (e.g., Pyright's stricter overload resolution).
• Import-surface test (tests/static/test_imports.py): a test that just imports every module in strix/ and instantiates every RunHooks/Capability/Model/Session subclass with valid kwargs. If any abstract method is unimplemented, instantiation raises. Catches T2, T11.
• Inventory completeness test (tests/static/test_inventory.py): asserts every row in tests/inventory/features.csv (see §3) has a non-empty test_id field. Prevents adding a feature without a test.
• SDK version pin guard (tests/static/test_sdk_version.py): asserts agents.__version__ == "0.14.6" exactly. We duplicate _create_container body; an SDK bump must be intentional. Fails CI on accidental upgrade.

Layer 2 — Unit

Each module tested in isolation with mocked dependencies. One file per source file.

• tests/orchestration/test_bus.py — AgentMessageBus register/send/drain/cancel_descendants/total_stats. Concurrency stress: 1000 concurrent send/drain ops, FIFO assertion.
• tests/orchestration/test_filter.py — inject_messages_filter with synthetic CallModelData. Empty inbox → passthrough; 3 messages → 3 user items; user-from-user no XML wrap; bus exception → return unchanged (C14).
• tests/orchestration/test_hooks.py — StrixOrchestrationHooks. Crash detection (output=None or agent_finish_called=False); turn warnings at 85% / N-3; bus errors don't propagate (C15).
• tests/llm/test_anthropic_cache.py — AnthropicCachingLitellmModel._patch. Anthropic model → cache_control present; non-Anthropic → passthrough; system message wrapped correctly.
• tests/llm/test_multi_provider.py — StrixModelProvider.get_model. Known alias → correct concrete model + base URL; unknown alias → UserError (C17).
• tests/llm/test_session.py — StrixSession. Compression triggers above 90K; compressor exception → uncompressed history (C10); subsequent calls skip compression after first failure.
• tests/runtime/test_strix_docker_client.py — StrixDockerSandboxClient._create_container with mocked docker_client. Assert cap_add ⊇ {NET_ADMIN, NET_RAW} and extra_hosts["host.docker.internal"] == "host-gateway".
• tests/sandbox/test_caido_capability.py — CaidoCapability.process_manifest injects proxy env; tools() returns 7 tools; instructions() returns non-empty string; bind() spawns healthcheck task.
• tests/sandbox/test_session_manager.py — create_or_reuse cache hit returns same session; cleanup removes container.
• tests/telemetry/test_processor.py — StrixTracingProcessor. Concurrent writes, JSONL is line-valid; PII patterns scrubbed; OSError doesn't propagate (C16).
• tests/tools/test_decorator.py — strix_tool factory. Default 120s timeout applied; sync function auto-threaded; error_as_result returns string.
• tests/tools/test_sandbox_dispatch.py — post_to_sandbox. 401 → error string; size cap enforced (C18); timeout returns error string.
• tests/tools/test_<each>.py — one per ported tool. Each: smoke test (valid args → expected output shape) + one edge case (bad args, timeout, network error).

Layer 3 — Integration

Multiple modules wired together; LLM and sandbox still mocked.

• tests/integration/test_single_agent_mocked.py — Build root Agent, run with mocked Model that emits scripted tool calls; assert tracer captured expected events; assert RunResult.final_output populated.
• tests/integration/test_multi_agent_mocked.py — Root spawns 2 children via create_agent tool; mocked Model scripts each child's responses; bus messaging between children; both agent_finish; root finish_scan. Assert: stat aggregation correct; messages delivered FIFO; bus.statuses all completed.
• tests/integration/test_cancellation_cascade.py — Build a tree, then bus.cancel_descendants(root); assert all child tasks cancelled(); assert leaves cancelled before parents (C9).
• tests/integration/test_subagent_crash.py — Mock child raising RuntimeError; assert <agent_crash> arrives in parent inbox via filter (C8).
• tests/integration/test_compressor_fallback.py — Mock compressor LLM raising; assert run continues with uncompressed history (C10).
• tests/integration/test_finish_scan_blocks_with_running_children.py — Root calls finish_scan while child still running; assert error returned (C22).
• tests/integration/test_jsonl_concurrent_writes.py — 50 concurrent agents writing to events.jsonl; assert every line is valid JSON (C7); same for notes (C6).

Layer 4 — Behavioral parity diffing

The most important layer. Run the same scenario through legacy and SDK harness; diff every artifact. Detail in §4 below.

Layer 5 — Recorded replay

Deterministic end-to-end tests using captured LLM and sandbox traces. Detail in §5.

Layer 6 — Live shadow / canary

Run both harnesses in production for a small slice of real users; diff results in real time. Detail in §6.

Layer 7 — Manual smoke

Humans operating the TUI, headless mode, multi-target scans. Detail in §9.

---

3. Feature inventory matrix

The single artifact that prevents silent feature loss. One CSV file (tests/inventory/features.csv) with one row per feature. CI fails if any row is missing a test reference.

Schema

feature_id,subsystem,description,source_ref,test_id,owner,phase,status

• feature_id — stable opaque identifier (e.g., F-AGENT-LOOP-001).
• subsystem — agents | llm | tools | sandbox | telemetry | interface | config.
• description — one sentence.
• source_ref — path:line to the legacy implementation OR path:line to the migration playbook spec.
• test_id — name of the test (or test marker) that proves it survives. Empty = blocker.
• owner — engineer responsible.
• phase — Phase 0–6 in which the feature lands.
• status — legacy-only | both | sdk-only | parity-verified.

Sample rows (~250 features expected)

F-AGENT-LOOP-001,agents,Agent loop with max_iterations=300,strix/agents/base_agent.py:152,test_runner_max_turns_300,allam,1,parity-verified
F-AGENT-LOOP-002,agents,85%/N-3 turn warnings,strix/agents/base_agent.py:186,test_turn_warnings_injection,allam,3,parity-verified
F-AGENT-LOOP-003,agents,Streaming early-truncate at </function>,strix/llm/llm.py:212,,allam,defer,legacy-only
F-LLM-001,llm,Anthropic prompt cache control,strix/llm/llm.py:371,test_anthropic_cache_present,allam,0,parity-verified
F-TOOL-BROWSER-001,tools,launch action,strix/tools/browser/browser_actions.py:75,test_browser_launch,allam,2,parity-verified
F-TOOL-BROWSER-002,tools,goto action,strix/tools/browser/browser_actions.py:80,test_browser_goto,allam,2,parity-verified
... (24 browser actions)
F-TOOL-CAIDO-001,tools,list_requests with HTTPQL filter,strix/tools/proxy/proxy_actions.py:9,test_caido_list_requests,allam,2,parity-verified
... (7 caido tools)
F-MULTIAGENT-MSG-001,orchestration,send_message_to_agent FIFO,strix/tools/agents_graph/agents_graph_actions.py:495,test_bus_send_drain_fifo,allam,3,parity-verified
F-MULTIAGENT-MSG-002,orchestration,inter_agent_message XML wrap,base_agent.py:491,test_filter_xml_wrap,allam,3,parity-verified
F-EVENT-001,telemetry,run.started event in events.jsonl,strix/telemetry/tracer.py:87,test_event_run_started_emitted,allam,1,parity-verified
F-EVENT-002,telemetry,tool.execution.started event,strix/telemetry/tracer.py:300,test_event_tool_execution_started,allam,1,parity-verified
... (every event type)
F-CLI-FLAG-target,interface,--target,--t accepts URL/repo/path,strix/interface/main.py:267,test_cli_target_inference,allam,5,parity-verified
F-CLI-FLAG-scan-mode,interface,--scan-mode quick|standard|deep,strix/interface/main.py:295,test_cli_scan_mode,allam,5,parity-verified
... (every CLI flag)
F-OUTPUT-EXIT-CODE-2,interface,exit code 2 on findings in headless,strix/interface/main.py:640,test_headless_exit_code_2,allam,5,parity-verified
F-OUTPUT-VULN-JSON,telemetry,vulnerabilities/vuln_*.json schema,strix/telemetry/tracer.py:365,test_vuln_json_schema,allam,2,parity-verified
F-OUTPUT-REPORT-MD,interface,penetration_test_report.md template,strix/telemetry/tracer.py:400,test_report_md_template,allam,5,parity-verified
F-PII-001,telemetry,scrubadub OpenAI key pattern,strix/telemetry/utils.py:87,test_pii_scrub_openai_key,allam,1,parity-verified
F-PII-002,telemetry,scrubadub bearer token pattern,strix/telemetry/utils.py:91,test_pii_scrub_bearer,allam,1,parity-verified
... (every regex pattern)
F-SKILL-NoSQL,prompts,NoSQL injection skill,strix/prompts/vulnerabilities/nosql_injection.jinja,test_skill_nosql_loadable,allam,2,parity-verified
F-SKILL-K8s,skills,Kubernetes security skill,strix/skills/cloud/kubernetes.md,test_skill_k8s_loadable,allam,2,parity-verified
... (every skill file)

Workflow

1. Bootstrap: a script (scripts/build_inventory.py) walks the legacy code and emits a draft CSV. Engineer fills test_id per row.
2. CI gate: tests/static/test_inventory_completeness.py asserts every row has test_id non-empty. Empty row → CI fails.
3. Coverage check: a separate test asserts every test_id listed in the inventory actually exists as a test function (no typos, no orphans).
4. PR review rule: any code change touching strix/ must update the inventory if it adds/removes/modifies a feature.
5. Cutover gate: ≥98% of inventory rows must be parity-verified. The remaining ≤2% are legacy-only (intentionally dropped) with explicit owner approval recorded in the row.

This matrix is the single artifact that proves "we didn't lose anything." It's the audit trail.

---

4. Behavioral parity diffing

The strongest non-trivial test we have. Same input, same env, both harnesses, diff every output. If the diff is empty, parity is proved.

What we diff

For a fixed scenario (same target, same instruction, same model, same env):

Artifact	Diff strategy
events.jsonl	Per-line JSON normalized + sorted by event_type + key fields; some fields ignored (timestamps, UUIDs, span IDs); deep diff
vulnerabilities/*.json	Group by stable identifier (target+endpoint+CVE), normalize, deep-diff each group's contents
penetration_test_report.md	Length within ±10%; heading set identical; CWE histogram identical
notes/notes.jsonl	Sorted by category + title; content body fuzzy-match (Levenshtein > 0.95)
wiki/*.md	File set identical; per-file content fuzzy-match
Tool call sequence (extracted from events)	Tool name multiset identical; arg signature shapes identical
Token usage	Within ±5% (LLM nondeterminism + caching variance)
Wall-clock duration	Within ±50% (allow for SDK overhead and parallel tool gains)

Normalization (the careful part)

LLMs are nondeterministic. We must normalize away noise before diffing or every diff fails.

# tests/parity/normalize.py
def normalize_events(events_jsonl_path: Path) -> list[dict]:
    out = []
    for line in events_jsonl_path.read_text().splitlines():
        evt = json.loads(line)
        # Strip noise
        evt.pop("timestamp", None)
        evt.pop("trace_id", None)
        evt.pop("span_id", None)
        evt.pop("agent_id", None)            # different IDs in old vs new
        evt.pop("scan_id", None)
        # Normalize content fields
        if "payload" in evt and "content" in evt["payload"]:
            evt["payload"]["content"] = _normalize_text(evt["payload"]["content"])
        out.append(evt)
    # Sort by event_type, then by stable shape hash
    out.sort(key=lambda e: (e.get("event_type", ""), _shape_hash(e)))
    return out

The diff library is deepdiff; failures point to specific keys.

The runner

# tests/parity/run_parity.py
def run_parity(scenario_id: str) -> ParityResult:
    """Run scenario through both harnesses with identical inputs and recorded LLM."""
    inputs = load_scenario(scenario_id)  # target, instruction, env, model

    legacy_run = run_legacy(inputs, recorded_llm=RECORDED[scenario_id])
    sdk_run = run_sdk(inputs, recorded_llm=RECORDED[scenario_id])

    return ParityResult(
        events_diff=diff_events(legacy_run.events_jsonl, sdk_run.events_jsonl),
        vulns_diff=diff_vulns(legacy_run.vulns, sdk_run.vulns),
        report_diff=diff_report(legacy_run.report, sdk_run.report),
        tool_call_diff=diff_tool_calls(legacy_run.events, sdk_run.events),
        usage_drift=compute_usage_drift(legacy_run.usage, sdk_run.usage),
    )

Scenarios to fix as parity baselines

A small, hand-picked, stable set:

Scenario	Target	Mode	Why
S01-static-blackbox	https://juice-shop.local	quick	Standard OWASP target; many findings; broad tool exercise
S02-static-whitebox	./examples/vulnerable-flask-app	deep	Whitebox path; semgrep+ast paths; wiki notes
S03-multi-target	repo + url combined	standard	Multi-target coordination
S04-diff-mode	./examples/vulnerable-flask-app --scope-mode=diff	quick	Diff scope injection
S05-cancellation	juice-shop, kill at iter 10	quick	Cancellation cascade
S06-multi-agent-explicit	DVWA, instruction triggers create_agent	deep	Subagent flow
S07-empty-target	nonexistent.local	quick	Failure path
S08-large-codebase	./examples/big-repo	standard	Compression triggered

Every scenario has a recorded LLM trace (§5). Every scenario runs in CI. Failure on any scenario blocks cutover.

What "diff is empty" means

After normalization, an empty diff on every artifact means: every event the legacy harness emits, the SDK harness emits; every finding the legacy harness reports, the SDK harness reports; every tool call sequence is the same set; the report has the same structure. It does NOT mean every byte is identical — that's impossible with LLMs.

Bidirectional comparison

We diff in both directions:

• "What does legacy have that SDK doesn't?" — the dangerous direction (silent feature loss).
• "What does SDK have that legacy doesn't?" — safer (new behavior), but worth review.

A row like event_type=agent.created missing in SDK → blocker. A row like event_type=tool.guardrail.rejected only in SDK → review.

---

5. Replay infrastructure

LLM and sandbox calls are nondeterministic (LLM) or environment-dependent (sandbox). For deterministic CI, we record once and replay forever.

LLM recording

A RecordedLLM model that intercepts Model.get_response/stream_response, looks up the request in a fixture file, returns the recorded response.

# tests/replay/recorded_llm.py
from agents.models.interface import Model

class RecordedLLM(Model):
    def __init__(self, recording_path: Path):
        self.recordings = json.loads(recording_path.read_text())
        self.cursor = 0

    async def get_response(self, system_instructions, input, model_settings, tools, ...):
        key = _hash_request(system_instructions, input, model_settings)
        if key not in self.recordings:
            raise ValueError(f"No recording for request hash {key[:8]}; capture with --record")
        recording = self.recordings[key]
        return ModelResponse(
            output=[_deserialize_item(it) for it in recording["output"]],
            usage=Usage(**recording["usage"]),
            response_id=recording.get("response_id"),
        )

    async def stream_response(self, ...):
        # Same lookup; replay chunks one by one
        ...

How requests are keyed: hash of (system_instructions, input_items, model, tools, model_settings excerpts). PII-stripped before hashing. Hash collisions are vanishingly rare; if they happen, append a sequence counter.

Capture mode: pytest --record-llm runs scenarios against a real LLM with a flag set; all acompletion calls are intercepted at a wrapper layer and serialized to tests/replay/recordings/<scenario_id>.json. PII scrubbed via existing TelemetrySanitizer before write.

Replay mode (default): pytest uses RecordedLLM instead of real LLM; deterministic.

Sandbox recording

Same pattern for sandbox HTTP calls. Wrap post_to_sandbox:

class RecordedSandbox:
    def __init__(self, recording_path: Path):
        self.recordings = json.loads(recording_path.read_text())

    async def post(self, agent_id, tool_name, kwargs):
        key = _hash_call(agent_id, tool_name, kwargs)
        if key not in self.recordings:
            raise ValueError(f"No sandbox recording for {tool_name} hash {key[:8]}")
        return self.recordings[key]

Inject via test fixture; production code path unchanged.

Recording vs replay in CI

• Replay (default, fast): every PR runs scenarios against recorded LLM + sandbox. Catches behavioral regressions in the harness itself.
• Re-record (manual, scheduled): a recurring CI job (or operator command) re-records scenarios against real LLM + real sandbox. Generates fresh recordings if the model output drifts.
• Drift detection: if a re-record produces output that fails the parity diff, the migration broke (or the legacy harness changed in main; check git history).

This is exactly the pattern the SDK uses internally (inline_snapshot library). We can adopt it directly.

---

6. Live shadow mode

For the highest-confidence pre-cutover signal: run both harnesses in production, on real user runs, diff results.

Shadow runner

A CLI mode strix --target ... --shadow that:

1. Spawns the legacy harness against the target.
2. Spawns the SDK harness against the same target (separate sandbox container).
3. Both run to completion independently.
4. After both finish, computes parity diff and emits a report.

User sees the legacy result (no UX disruption); engineering team sees the diff.

# strix/interface/shadow.py
async def run_shadow(args):
    legacy_task = asyncio.create_task(run_legacy(args, run_dir=Path("strix_runs/shadow_legacy")))
    sdk_task = asyncio.create_task(run_sdk(args, run_dir=Path("strix_runs/shadow_sdk")))
    legacy_result, sdk_result = await asyncio.gather(legacy_task, sdk_task)

    diff = run_parity(legacy_result.run_dir, sdk_result.run_dir)
    write_diff_report(diff, Path("strix_runs/shadow_diff.json"))
    upload_diff_to_telemetry(diff)
    return legacy_result  # user gets the legacy answer; safe rollback

Sampling

Not every run shadows — too expensive in tokens. Configurable sampling:

STRIX_SHADOW_SAMPLE_RATE=0.1   # 10% of runs go through shadow mode
STRIX_SHADOW_FORCE=1           # always (for engineering / staging)

What we look for

• Parity rate: % of shadow runs where diff is empty. Target: ≥99% before cutover.
• Drift class histograms: which fields differ most? Track per-field over time.
• Resource drift: token cost, wall-clock, sandbox memory. Plot distributions.

Shadow runs don't gate cutover by themselves (too noisy with 1 sample), but a sustained drop in parity rate is a stop signal.

---

7. Per-correction test mapping

Every one of the 25 corrections from AUDIT.md + AUDIT_R2.md + AUDIT_R3.md needs a test that would have caught the bug. Defensive code without proof-of-defense is theater.

#	Correction	Test	Layer
C1	Tool-server slot serialization vs SDK parallel calls	test_parallel_tool_calls_safe_default_no_collision (Layer 3) + test_tool_server_relax_phase6_concurrent_works (Layer 3)	3
C2	Anthropic cache_control on system message	test_anthropic_cache_control_on_system_message (Layer 2) + test_anthropic_cache_hit_rate (Layer 5, recorded)	2, 5
C3	DockerSandboxClient subclass injects caps	test_strix_docker_client_caps_injected (Layer 2, mocked) + test_strix_docker_client_nmap_works (Layer 7, live)	2, 7
C4	Subagent tool_use_behavior	test_subagent_exits_on_agent_finish (Layer 3)	3
C5	StrixStreamAccumulator parity	test_stream_accumulator_event_coverage (Layer 4 vs Layer 5 baseline)	4, 5
C6	Notes JSONL write lock	test_notes_jsonl_concurrent_writes_no_corruption (Layer 3)	3
C7	events.jsonl write lock	test_events_jsonl_concurrent_writes_no_corruption (Layer 3)	3
C8	Subagent crash detection	test_subagent_crash_emits_agent_crash_to_parent (Layer 3)	3
C9	Cancellation cascade	test_cancel_descendants_walks_tree_leaf_first (Layer 3)	3
C10	Compressor try/except	test_compressor_failure_returns_uncompressed (Layer 2)	2
C11	Retry policy excludes 401/403/400	test_retry_policy_does_not_retry_401 (Layer 2)	2
C12	Stats snapshot under lock	test_total_stats_consistent_under_concurrent_writes (Layer 2)	2
F1	LitellmModel signature positional-first	mypy / test_anthropic_caching_litellm_model_overrides_correctly	1, 2
F2	RunHooks AgentHookContext + result:str	mypy / test_hooks_signatures_match_sdk	1, 2
F3	TracingProcessor methods sync	mypy / test_processor_methods_are_sync	1, 2
C13	Bus.finalize cleans up state	test_finalize_clears_inbox_parent_name (Layer 2)	2
C14	Filter try/except	test_filter_exception_returns_unmodified (Layer 2)	2
C15	Hooks try/except	test_hooks_exception_does_not_propagate (Layer 2)	2
C16	Processor catches OSError	test_processor_oserror_caught_run_continues (Layer 2)	2
C17	Model alias validation	test_unknown_alias_raises_user_error (Layer 2)	2
C18	Sandbox response size cap	test_sandbox_response_too_large_returns_error (Layer 2)	2
C19	Assert ≥1 enabled tool	test_agent_build_fails_with_no_tools (Layer 2)	2
C20	Per-tool timeout_behavior	test_critical_tool_timeout_raises + test_idempotent_tool_timeout_returns_string	2
C21	RunConfig override + context fields	test_run_config_override_merges + test_context_has_is_whitebox	2
C22	finish_scan checks children	test_finish_scan_blocks_with_running_children (Layer 3)	3
C23	Diff-scope injection	test_diff_scope_in_first_user_message (Layer 4)	4
C24	Run-name + Docker preflight	test_collision_detected + test_docker_unavailable_clear_error (Layer 7)	7
C25	Cancel mode mapping	test_ctrl_c_immediate + test_tui_stop_after_turn (Layer 7)	7

Every test name above is a placeholder for a real test function. The grid is the contract. CI runs all of these.

---

8. CI gating and cutover criteria

Per-PR CI (every commit)

Check	Layer	Failure means
mypy --strict	1	Type contract drift
ruff check + format	1	Lint failure
pytest tests/static/	1	Inventory incomplete or imports broken
pytest tests/<module>/ (unit)	2	Module-level regression
pytest tests/integration/	3	Cross-module regression
pytest tests/parity/ against recorded scenarios	4, 5	Behavioral drift
Inventory completeness	1	A feature row has no test_id
Inventory test existence	1	A test_id is missing the test
SDK version pin	1	Accidental SDK upgrade

Nightly CI (scheduled, longer-running)

Check	Purpose
Re-record scenarios against real LLM+sandbox	Detect drift in legacy or SDK behavior over time
Mutation testing on critical modules (bus, filter, hooks)	Verify tests actually catch bugs
Multi-platform PyInstaller build + smoke	Catch packaging regressions on macOS arm64/x86_64, Linux, Windows
Memory-pressure soak (300-turn run)	Catch leaks

Cutover criteria

To flip STRIX_USE_SDK_HARNESS default from 0 to 1:

• All 25 corrections (C1–C25 + F1–F3) have green tests in the grid above.
• Inventory matrix: ≥98% of rows are parity-verified. Remaining ≤2% are explicitly legacy-only with owner sign-off.
• Layer 4 parity diffing: 8 baseline scenarios all empty-diff (post-normalization).
• Layer 5 replay: all recorded scenarios green.
• Layer 7 manual smoke: TUI works on macOS + Linux; headless mode produces correct exit codes.
• Shadow mode (Layer 6): ≥99% parity rate over a 7-run sample at minimum.
• Mutation testing: ≥80% mutation kill rate on critical modules.
• Memory soak: 300-turn run completes; memory growth < 1 GB; no orphan containers.
• Engineering team signoff via PR review.

Post-cutover gates (kept for one release after flip)

• Legacy harness still ships (gated by STRIX_USE_SDK_HARNESS=0).
• CI continues to run parity diffing on every PR.
• Nightly re-record runs detect any post-cutover legacy/SDK drift.
• Production telemetry on parity rate from real users (sampled).

If parity rate drops below 95% in production: emergency rollback.

---

9. Manual smoke checklist

Things a human verifies before cutover. Run these on macOS and Linux at minimum.

TUI mode

• strix --target https://juice-shop.local launches splash screen.
• Agent tree renders; root expands to show subagents when spawned.
• Streaming text appears in real time as agent generates.
• F1 opens help screen; ESC closes.
• Vulnerability popup appears when first finding logged.
• Ctrl+C → confirm dialog → ESC dismisses, Y stops agent.
• Tab cycles panels; arrow keys navigate agent tree.
• Ctrl+Q quits cleanly; container removed; run dir intact.
• After agent completes, prompt for follow-up message; user can type.

Headless mode

• strix -n --target ./examples/vulnerable-flask-app --scan-mode quick runs.
• Rich panels render: startup, live stats, vuln-found.
• Final summary panel shows.
• Exit code 2 if findings; 0 if none.
• Ctrl+C exits 130 cleanly; container removed.

Multi-target

• strix -t ./repo -t https://app.local handles both.
• Each target gets its own /workspace/<subdir> mount.
• Findings tagged by target.

Diff scope

• strix -n --target ./repo --scope-mode diff --diff-base origin/main includes diff block in instruction.
• Agent's first turn references the diff.

Config override

• strix --config /path/to/custom.json --target ... overrides defaults.
• Env vars from custom config apply; default config vars cleared.

Resilience

• Kill the sandbox container mid-run (docker stop); agent surfaces error, exits gracefully.
• Run with invalid STRIX_LLM=strix/typo; clear error message naming valid aliases.
• Run without Docker daemon; preflight error message tells user to start Docker.

---

10. Post-cutover monitoring

Once SDK harness is the default, watch for slow regressions.

Daily metrics

• Parity rate from shadow sample (rolling 7-day).
• Token cost per scan (rolling 7-day per scan_mode).
• Wall-clock per scan (rolling 7-day).
• Findings count distribution by CWE.
• Crash rate by category (LLM, sandbox, tool, agent).

Alerts

• Parity rate drops below 95% → page engineer.
• Token cost rises >20% week-over-week (with same scan mix) → review.
• Crash rate rises >2× baseline → page.
• Any new error class appears in events.jsonl that wasn't present pre-cutover → review.

Telemetry (existing PostHog + custom)

• Per-scan: legacy-vs-sdk path used, total cost, total findings, exit code, duration, scan_mode.
• Per-tool: invocation count, error rate, p50/p99 latency.
• Per-error: category, count, first/last seen.

---

11. What to do when a parity diff fails

Standard incident playbook. Don't let a failed diff sit; chase root cause.

1. Examine the diff — tests/parity/run_parity.py outputs a structured diff. Identify the specific field/event that drifted.
2. Classify the drift:
   • Code bug in our migration → fix, re-test.
   • Acceptable behavior change (e.g., SDK emits a new event the legacy didn't) → update normalizer to ignore the field, document in tests/parity/normalization_notes.md.
   • Recording staleness → re-record affected scenario; investigate why output changed.
3. Add a new test if the drift wasn't caught by an existing assertion. Update inventory matrix.
4. Don't suppress — never # noqa a parity failure. The matrix is the contract.

Common drift causes (anticipated)

Drift	Root cause	Fix
Tool call sequence differs	LLM nondeterminism on retries	Recording captures multiple valid sequences; diff accepts any
Event count off by 1	SDK emits an extra agent.start for handoff	Normalizer filters handoff events
Token count drift > 5%	Anthropic cache hit/miss timing	Replay against recorded; if still drifts, investigate cache wrapper
Vulnerability missing	Dedup decided differently	Check dedup LLM call recording; verify same prompt produces same answer
Wiki note shape differs	Update format changed	Normalize whitespace; check append_note_content invocation

---

TL;DR

Five mechanisms catch the categories of regression we're worried about:

1. Layer 4 parity diffing on 8 baseline scenarios — catches every silent regression in tool calls, events, findings, reports. The diff itself is the signal; we don't need to enumerate failures.
2. Feature inventory matrix (~250 rows) — every feature has a test; CI fails if the matrix has gaps. Prevents adding features without tests; prevents losing features without notice.
3. Recorded LLM + sandbox replay (Layer 5) — deterministic CI; same input always produces same output; no token burn per PR.
4. Shadow mode in production (Layer 6) — real-world parity validation; sampled at 10%; strongest signal pre-cutover.
5. Per-correction tests (25+3 = 28 specific tests) — every audit finding has a test that proves the fix works.

CI gates everything. Cutover criteria are explicit and measurable. Rollback is a flag flip. Post-cutover monitoring catches slow drift.

The migration cannot silently lose a feature unless: (a) the feature isn't in the inventory matrix, AND (b) it isn't exercised by any of the 8 baseline scenarios, AND (c) it doesn't appear in production within the shadow sampling window. That's a narrow gap, and it gets narrower with every scenario added.