strix/AUDIT.md

Migration Audit — Pre-Execution Verification

Source-verified against openai-agents v0.14.6 at /tmp/openai-agents and Strix at 9fb1012. Five parallel deep-dives covering: agent loop, tool execution, sandbox, LLM/sessions/tracing, and Strix internals.

Verdict: GO. No architectural blockers. Five concrete corrections to the plan must be applied before Phase 1; all are <300 LOC total. Migration today is feasible if we apply the corrections below in the listed order.

---

1. Verified bridges (no change needed)

These claims from MIGRATION_EVALUATION.md were confirmed against source — proceed as planned:

Plan claim	Verification	Source
call_model_input_filter runs before every model call	✓ Confirmed at turn_preparation.py:55-80. Bonus: filter runs ONCE per turn — its output is captured in a lambda closure for retries (model_retry.py:34-35). Inbox messages will NOT be drained twice on retry. Open question #1 in plan §11 is resolved.	run_internal/turn_preparation.py:48-82, run_internal/run_loop.py:1363-1369, 1803-1809
asyncio.create_task(Runner.run(...)) is isolation-safe	✓ Each task gets own RunContextWrapper; contextvars properly isolated. No global state mutation inside Runner.run.	run.py:486-615, run_context.py:42-51
Shared SandboxRunConfig(session=...) across parallel runs	✓ SDK does NOT tear down sandbox sessions at run end; caller owns lifecycle. Safe to reuse one session across N children.	run_config.py:115-138, docker.py:1372-1401
RunContextWrapper.context mutable across turns	✓ Dict is by-reference; mutations persist. Bus + agent_id stash will work as designed.	run_context.py:42-51, run.py:615
RunHooks.on_agent_end fires once per Runner.run	✓ Single fire when final_output established.	run_internal/turn_resolution.py:204-255
Custom Docker image accepted	✓ DockerSandboxClientOptions(image=str) is verbatim pass-through to containers.create(image=...). No assumed binaries.	sandbox/sandboxes/docker.py:106-122, 1340, 1444-1456
Manifest.environment reaches container	✓ Resolved via await manifest.environment.resolve() and passed to containers.create(environment=...).	sandbox/sandboxes/docker.py:1448-1450
Manifest entries are a strict superset (LocalDir, GitRepo, mounts)	✓ Direct LocalDir maps to our tar-pipe with concurrency limits.	sandbox/entries/__init__.py, artifacts.py:127-179
Capability lifecycle (clone, bind, tools, instructions, process_manifest)	✓ Per-run cloned; bound after container start; can hold mutable state. CaidoCapability viable.	sandbox/capabilities/capability.py:15-100, sandbox/runtime.py:180-256
MultiProvider with custom prefix routing	✓ MultiProviderMap.add_provider("strix", StrixModelProvider()) works exactly.	models/multi_provider.py:16-49, 138-232
Custom ModelProvider interface	✓ Just get_model(model_name) -> Model. Post-prefix-strip name received.	models/interface.py:127-151
LitellmModel reasoning effort priority	✓ Exact: reasoning.effort > extra_body["reasoning_effort"] > extra_args["reasoning_effort"].	extensions/models/litellm_model.py:162-199
LitellmModel streaming + tool-call assembly across providers	✓ ChatCmplStreamHandler.handle_stream() unifies provider-native streaming (Anthropic, OpenAI, etc.) into common stream format.	extensions/models/litellm_model.py:315-351
add_trace_processor() / set_trace_processors()	✓ Both exist; can disable defaults entirely.	tracing/__init__.py:94-130
RunHooks 7-hook surface area	✓ All 7 hooks fire as documented. RunHooks + AgentHooks both fire (gathered).	lifecycle.py:13-99, 102-199
Per-tool timeout default is None	✓ Confirmed. Our strix_tool() factory will re-impose 120s.	tool.py:337-338
RunState.to_json()/from_json() resumable across processes	✓ Schema v1.9; full serialization.	run_state.py:1-200
tracing_disabled per-RunConfig	✓ Disables ALL tracing for that run.	run_config.py:186-188
OPENAI_AGENTS_DONT_LOG_MODEL_DATA env	✓ Logging only; independent from tracing.	_debug.py:12-21
Sync function tools auto-offload via asyncio.to_thread	✓ Plan was wrong — SDK DOES auto-thread sync @function_tool bodies. We can drop the manual asyncio.to_thread wrapping in our libtmux/IPython tools and just write sync functions. ~30 LOC saved.	tool.py:1820-1829
ToolGuardrailFunctionOutput.reject_content("nope") continues run	✓ Model sees "nope" as tool output and proceeds. Run NOT halted.	tool_guardrails.py:79-105
Multi-agent stat aggregation via hooks	✓ on_llm_end + on_agent_end fire on each child Runner.run; bus aggregation works.	lifecycle.py, run_internal/turn_resolution.py:204-255

---

2. Critical corrections (must apply before / during Phase 1)

Five corrections to the plan. All are concrete and small.

2.1 [BLOCKER] Strix tool server slot serialization vs SDK parallel tool calls

The collision. SDK fires N tool calls in one turn as N concurrent asyncio.create_task (run_internal/tool_execution.py:1414, :1424-1430). Strix tool server cancels the previous in-flight task for the same agent on every new request (tool_server.py:94-97). When SDK issues terminal_execute + web_search simultaneously for the same agent, the second cancels the first.

Fix (Phase 1 — safe default). Add to the default RunConfig for every Strix run:

RunConfig(
    model_settings=ModelSettings(
        parallel_tool_calls=False,   # model-side hint: emit one tool call per turn
        ...
    ),
    isolate_parallel_failures=False, # if model emits multiple anyway, don't cascade-cancel
    ...
)

Caveat. parallel_tool_calls is a provider hint (model_settings.py:89-96), not enforced SDK-side. The model may still emit multiple. With isolate_parallel_failures=False, sibling tools survive a single failure; but the tool server still cancels prev-task on same-agent collision.

Fix (Phase 2 — proper). Relax tool_server.py:94-97 to allow concurrent same-agent tool calls. The cancellation logic was Strix's old serialization; we don't need it under the SDK's orchestration. ~10 LOC removal. Re-test multi-agent end-to-end.

Effort: 0.5 day (safe default in Phase 1) + 0.5 day (proper fix + tests in Phase 2).

---

2.2 [BLOCKER] Anthropic prompt cache placement is wrong in plan

The defect. Plan §3.2 said: set ModelSettings(extra_body={"cache_control": {"type": "ephemeral"}}). Verified at extensions/models/litellm_model.py:509-516 — this lands cache_control in the request-level extra_body, not on the system message. Anthropic requires cache_control on the system message itself (per Anthropic API spec). Plan would silently cache nothing.

Fix. Build a thin LitellmModel subclass that injects cache_control into the message list before delegating to parent:

# strix/llm/anthropic_cache_wrapper.py  (~40 LOC)
from agents.extensions.models.litellm_model import LitellmModel

class AnthropicCachingLitellmModel(LitellmModel):
    def _patch_system_message_for_cache(self, input_items: list) -> list:
        if not _is_anthropic(self.model):
            return input_items
        patched = []
        for item in input_items:
            if isinstance(item, dict) and item.get("role") == "system":
                content = item["content"]
                if isinstance(content, str):
                    content = [{"type": "text", "text": content,
                                "cache_control": {"type": "ephemeral"}}]
                patched.append({**item, "content": content})
            else:
                patched.append(item)
        return patched

    async def get_response(self, *, input, **kwargs):
        return await super().get_response(input=self._patch_system_message_for_cache(input), **kwargs)

    async def stream_response(self, *, input, **kwargs):
        async for ev in super().stream_response(input=self._patch_system_message_for_cache(input), **kwargs):
            yield ev

Wire into our MultiProviderMap so any litellm/anthropic/... route uses this wrapper.

Effort: 0.5 day (~40 LOC + tests).

---

2.3 [BLOCKER] DockerSandboxClient subclass requires full method duplication

The reality. Audit #2 verified that _create_container() (sandbox/sandboxes/docker.py:1434-1477) builds create_kwargs locally and does not expose a hook for kwarg injection. Subclass must reimplement the method body. ~100-120 LOC duplication. Plan said "~80 LOC" — bump to ~120 LOC.

Fix. Subclass and copy the parent body verbatim, adding our injections before the final containers.create(**create_kwargs) line:

# strix/runtime/strix_docker_client.py  (~120 LOC)
from agents.sandbox.sandboxes.docker import (
    DockerSandboxClient, _build_docker_volume_mounts,
    _manifest_requires_fuse, _manifest_requires_sys_admin,
    _docker_port_key, parse_repository_tag,
)

class StrixDockerSandboxClient(DockerSandboxClient):
    async def _create_container(self, image, *, manifest=None, exposed_ports=(), session_id=None):
        # --- copy of parent _create_container body (lines 1442-1476) ---
        if not self.image_exists(image):
            repo, tag = parse_repository_tag(image)
            self.docker_client.images.pull(repo, tag=tag or None, all_tags=False)

        environment = None
        if manifest:
            environment = await manifest.environment.resolve()

        create_kwargs = {
            "entrypoint": ["tail"],
            "image": image,
            "detach": True,
            "command": ["-f", "/dev/null"],
            "environment": environment,
        }

        if manifest is not None:
            mounts = _build_docker_volume_mounts(manifest, session_id=session_id)
            if mounts:
                create_kwargs["mounts"] = mounts
            if _manifest_requires_fuse(manifest):
                create_kwargs.setdefault("devices", []).append("/dev/fuse")
                create_kwargs.setdefault("cap_add", []).append("SYS_ADMIN")
                create_kwargs.setdefault("security_opt", []).append("apparmor:unconfined")
            if _manifest_requires_sys_admin(manifest):
                create_kwargs.setdefault("cap_add", []).append("SYS_ADMIN")

        if exposed_ports:
            create_kwargs["ports"] = {
                _docker_port_key(p): ("127.0.0.1", None) for p in exposed_ports
            }

        # --- STRIX INJECTIONS ---
        create_kwargs.setdefault("cap_add", []).extend(["NET_ADMIN", "NET_RAW"])
        create_kwargs.setdefault("extra_hosts", {})["host.docker.internal"] = "host-gateway"

        return self.docker_client.containers.create(**create_kwargs)

Risk. SDK upstream changes to _create_container won't propagate. Pin SDK version; track upstream in CI; consider upstream PR for additional_create_kwargs hook.

Effort: 0.5 day (~120 LOC + integration test).

---

2.4 [BLOCKER] Subagent must exit cleanly via agent_finish — needs tool_use_behavior configuration

The risk. When subagent calls agent_finish and we return a result string from the tool, SDK's loop checks Agent.tool_use_behavior (turn_resolution.py:512-544). If not configured to permit early exit, the loop continues until max_turns. Children would burn budget instead of finishing.

Fix. On every Strix subagent's Agent, configure:

child_agent = Agent(
    name=name,
    instructions=...,
    tools=[..., agent_finish, ...],
    tool_use_behavior={
        "stop_at_tool_names": ["agent_finish"],
    },
    ...
)

This tells the SDK: as soon as agent_finish returns, treat that as final output. Same pattern for root agent + finish_scan:

root_agent = Agent(
    name="strix-root",
    tool_use_behavior={"stop_at_tool_names": ["finish_scan"]},
    ...
)

Effort: Trivial — one config line per agent factory.

---

2.5 [HIGH] Streaming TUI integration needs a planned shape

The reality. Plan §10 phase 5 said "TUI re-pointed at Runner.run_streamed().stream_events()" — true, but Strix's current TUI polls tracer.streaming_content at 2 Hz with per-chunk granularity. SDK stream_events() exposes RawResponsesStreamEvent (raw chunks) and RunItemStreamEvent (semantic items) — sufficient, but we lose Strix's update_streaming_content(agent_id, accumulated_text) API that aggregates incremental text.

Fix. Build a StrixStreamAccumulator that consumes Runner.run_streamed().stream_events() and synthesizes the same shape Strix's tracer used to expose:

async for event in result.stream_events():
    if event.type == "raw_response_event":
        delta = _extract_text_delta(event.data)
        if delta:
            tracer.append_streaming_content(agent_id, delta)
    elif event.type == "run_item_stream_event":
        if event.name == "tool_called":
            tracer.log_tool_start(agent_id, event.item.tool_name)
        elif event.name == "tool_output":
            tracer.log_tool_end(agent_id, event.item.tool_name, event.item.output)

Plus, RunHooks.on_llm_start/on_llm_end/on_tool_start/on_tool_end fire regardless of streaming mode, so child agents launched via Runner.run (not streamed) still feed the tracer through hooks. The TUI subscribes to the same tracer.

Effort: 1.5 days for both stream accumulator + hook bridge + TUI repoint.

---

3. Medium-severity adjustments (Phase 1-2)

#	Issue	Source	Fix	Effort
M1	Cost tracking — SDK has Usage(input/output/cached_tokens) but no cost field. litellm.completion_cost() requires raw litellm response, not SDK's ModelResponse.	usage.py, extensions/models/litellm_model.py:254-293	Inside our AnthropicCachingLitellmModel and a similar light wrapper for OpenAI, capture the litellm response and store cost in ModelResponse.usage via litellm.cost_per_token(...) (which takes tokens, not response). Then RunHooks.on_llm_end reads it.	0.5 day
M2	Vision-less model image stripping — SDK has none, will pass-through and provider rejects.	None	If we end up routing to a non-vision model, build a wrapper Model that strips images. Defer; current models (Claude Sonnet 4.6, GPT-5, Gemini) are all vision-capable.	Defer (0 day)
M3	SQLiteSession uses threading.RLock, not asyncio.Lock. Concurrent async writes from parallel children may interleave.	memory/sqlite_session.py:17-175	Use a separate Session per child (history is per-agent anyway); only share SandboxRunConfig.session. Plan §4.7 already says this — emphasize it in code review.	0 day (already in plan)
M4	Trace processor memory pressure on 300-turn runs.	tracing/processor_interface.py	Custom processor batches every 100 spans + force_flush() periodically.	0.5 day
M5	Streaming events don't expose token deltas — only raw chunks.	stream_events.py	Parse RawResponsesStreamEvent.data chunks for token text manually in our accumulator.	(rolled into 2.5)
M6	trace_include_sensitive_data is binary, no field-level.	run_config.py:193-199	Custom trace processor scrubs PII via existing TelemetrySanitizer. Plan already says this.	0 day (already in plan)
M7	Caido + tool server readiness check needs a place to await — Capability.bind() is sync.	sandbox/capabilities/capability.py:29-31	Spawn a background task in bind() (_healthcheck_task); await it inside RunHooks.on_agent_start. ~30 LOC.	0.5 day
M8	vulnerability_found_callback (TUI popup trigger) — no SDK-native equivalent.	Strix telemetry/tracer.py:89	Wrap create_vulnerability_report tool with an output guardrail that fires the callback on success.	0.5 day
M9	<agent_delegation> XML wrapper today contains structured identity that the system prompt has rules to ignore.	Strix system_prompt.jinja:19-22, agents_graph_actions.py:238-266	Replicate exact XML envelope when inject_messages_filter adds the parent's task message OR when create_agent builds the child's initial input. Keeps system prompt rules intact unchanged.	0.5 day
M10	Whitebox wiki note auto-update on subagent finish (side effect on agent_finish tool).	Strix agents_graph_actions.py:161-202	Implement directly inside our agent_finish function tool body, just like today.	0 day (free port)
M11	_force_stop mid-turn soft-interrupt has no SDK equivalent.	Strix base_agent.py:84	Use result.cancel(mode="after_turn") for cooperative cancel; for mid-turn hard cancel, .cancel(mode="immediate").	0 day (use result.cancel)
M12	85% / N-3 turn warnings as user messages.	Strix base_agent.py:186-211	RunHooks.on_llm_start checks ctx.usage turn count; if at threshold, mutate input_items (passed by reference per lifecycle.py:18-26). Verify mutation visibility in source: hook signature shows input_items is the list; mutations propagate.	0.5 day

---

4. Pre-Phase-1 spike (1 day)

Before writing production code, validate the assumptions in a tiny throwaway script:

1. Two-children messaging smoke test. Build minimal MessageBus + inject_messages_filter + 2 child agents that exchange one message each. Run with LitellmModel("anthropic/claude-sonnet-4-5-20250929") (or whatever Anthropic alias is current). Verify: messages arrive, hooks fire, no deadlock, no message duplication on retry.
2. Anthropic cache wrapper smoke test. Send 3 requests with identical system prompt; check Anthropic response usage cache_creation_input_tokens on call 1 and cache_read_input_tokens on calls 2-3.
3. StrixDockerSandboxClient smoke test. Pull our Kali image, create a session, run nmap -sS scanme.nmap.org via session.exec() to verify NET_RAW works.
4. tool_use_behavior={"stop_at_tool_names": [...]} smoke test. Toy agent with agent_finish-equivalent; verify SDK terminates exactly when expected.
5. Tool server parallel-call smoke test. Issue two POSTs to local tool server with same agent_id simultaneously; observe whether second cancels first under current code.

If any spike fails, fix before Phase 1. If all pass, proceed.

Effort: 1 day.

---

5. Updated migration plan (rev 3 sequencing)

Replaces MIGRATION_EVALUATION.md §10. Same scope, with corrections folded in.

Phase 0 — Spike & corrections (1.5 days)

• Run the 5 spikes above.
• Build AnthropicCachingLitellmModel (~40 LOC). Smoke-tested.
• Build StrixDockerSandboxClient (~120 LOC). Smoke-tested.
• Decide tool server fix: relax serialization (recommended) OR set parallel_tool_calls=False + isolate_parallel_failures=False (safe default).

Phase 1 — Foundation (4 days)

• MultiProvider + MultiProviderMap with StrixModelProvider for our aliases.
• Wire AnthropicCachingLitellmModel into the provider map.
• strix_tool decorator (~30 LOC; just function_tool with default timeout=120, timeout_behavior="error_as_result").
• Custom Session subclass with our memory compressor strategy.
• Custom TracingProcessor with JSONL + scrubadub PII scrub. set_trace_processors([StrixProcessor()]) to disable defaults.
• RunConfig factory that bakes in: tracing_disabled=False, isolate_parallel_failures=False, model_settings.parallel_tool_calls=False (until Phase 6 relaxes), our processors.
• Cost tracking inside the model wrapper (M1).

Phase 2 — Tool ports (8 days)

• Sandbox dispatcher: one helper that POSTs to FastAPI tool server with httpx (Timeout(connect=10, total=150)) + Bearer auth.
• All 30+ tools as @strix_tool. Sync ones use def, SDK auto-threads them.
• Browser as ComputerTool + AsyncComputer subclass (or as a single @strix_tool if ComputerTool semantics don't match).
• Stateful tools key off RunContextWrapper.context["agent_id"] (helper: get_agent_id(ctx)).
• create_vulnerability_report wraps ToolOutputGuardrail to fire the TUI popup callback (M8).
• Verify reentrancy on browser singleton, tmux sessions, IPython kernels.

Phase 3 — Multi-agent orchestration (4 days)

• AgentMessageBus + tests.
• inject_messages_filter + tests including retry simulation (verified safe by audit; no de-dup needed).
• StrixOrchestrationHooks for stat aggregation + tracer wiring.
• Six graph tools (create_agent, send_message_to_agent, wait_for_message, agent_status, view_agent_graph, agent_finish).
• Every child Agent: tool_use_behavior={"stop_at_tool_names": ["agent_finish"]}.
• Root Agent: tool_use_behavior={"stop_at_tool_names": ["finish_scan"]}.
• Identity injection via <agent_delegation> XML in the first user message of the child Runner (M9).
• Wiki auto-update on whitebox agent_finish (M10).

Phase 4 — Sandbox + Caido capability (2 days)

• StrixDockerSandboxClient (already in Phase 0).
• CaidoCapability with process_manifest (env vars), tools() (7 Caido tools), instructions() (proxy-aware system block), bind() (spawn healthcheck task).
• RunHooks.on_agent_start awaits Caido + tool server readiness via the capability's healthcheck task (M7).
• Container reuse keyed by scan_id in our session map.

Phase 5 — Interface + persistence (3 days)

• Streaming accumulator wires Runner.run_streamed().stream_events() → tracer (replaces today's per-chunk update_streaming_content).
• TUI keeps its 2 Hz polling against the tracer; tracer is now event-driven from the accumulator.
• 85% / N-3 turn warnings via RunHooks.on_llm_start mutating input_items (M12).
• Run-directory layout via custom TracingProcessor writing events.jsonl, vulnerabilities/, etc.
• CLI / config / argparse layer unchanged.

Phase 6 — Validation + tool server relaxation (4 days)

• Smoke: every tool runs in sandbox.
• Multi-agent: 2+ parallel children + messaging + cancel.
• Bedrock + Anthropic + OpenAI parity.
• Memory compression at 90K.
• PII redaction.
• Real pentest end-to-end vs Strix baseline diff.
• Now relax tool_server.py:94-97 (remove per-agent task cancellation), set parallel_tool_calls=True and isolate_parallel_failures=True. Re-run multi-agent test. If clean → ship parallelism.

Buffer (2 days)

For unforeseen issues from spike feedback or test failures.

Total: ~28.5 days (vs plan's 25–35). Within budget.

---

6. Final go/no-go

✅ GO.

Why:

• All architectural assumptions validated. No showstoppers found.
• The 5 corrections in §2 are concrete, small, and isolated to specific phases.
• The 12 medium adjustments in §3 are sensible and most are already implicit in the plan.
• The plan's rev-2 effort estimate (25–35 days) holds with corrections (~28.5 days).

Day-1 first commits (in order):

1. strix/llm/anthropic_cache_wrapper.py — AnthropicCachingLitellmModel.
2. strix/runtime/strix_docker_client.py — StrixDockerSandboxClient.
3. strix/orchestration/bus.py — AgentMessageBus.
4. strix/orchestration/filter.py — inject_messages_filter.
5. strix/orchestration/hooks.py — StrixOrchestrationHooks.
6. strix/tools/_decorator.py — strix_tool factory.
7. strix/llm/multi_provider_setup.py — MultiProviderMap wiring + StrixModelProvider.

These seven files (~600 LOC total) form the migration's load-bearing foundation. Everything else is incremental ports onto this foundation.

Branch is already on harness-migration. Ready when you are.