[volen-silo]

Why

Part of #4 and the harness interface asked for in #28; the design is in
this comment.

The host engine assumes one agent. Its command line, environment, skills
directory and "is it installed" check are all written for the claude CLI inside
the solver, so supporting another agent means copying the solver. This puts that
behind a small contract, with Claude Code as the first implementation, so a second
harness is an adapter and a set of tests rather than a change to the engines.

What changes

• skillscope/harness/, with no inspect_ai import so it is testable in the
  plain unit job: a Harness base class (binary, skills directory, command,
  env, require, normalize), a registry, and ClaudeCode, which is the
  command line and environment the engine already used, moved out unchanged.
• no_sandbox.harness_solver(harness, ...) is the existing solver,
  generalised. claude_code_no_sandbox keeps its signature and calls it with
  Claude Code, so every caller behaves as before.
• normalize translates a harness's own events into the stream-json subset
  the engines already read, so stop rules, budgets, scorers and the judge are
  untouched and written once for every harness. The default is the identity,
  which is all Claude Code needs.
• normalizer() (second commit) returns a translator for one run that may keep
  state. It exists because some CLIs do not repeat the answer in their closing
  event (GitHub Copilot CLI's result record carries only an exit code and usage;
  the answer is the last assistant message, printed earlier). The engine asks for
  a fresh one per case, so nothing remembered in one case reaches the next.
• install_skill takes the harness's skills directory, and _stream_until takes
  an optional translator. Both default to today's behaviour.
• docs/harnesses.md explains the contract, the canonical event form, how to add
  a harness, and what is not yet behind the contract.

What does not change

No new flag, no CLI change, no report change. The 505 existing tests pass
unmodified, including the one that drives the real solver against a stub claude.

Verification

• 530 tests pass on Windows with Python 3.12 (505 existing, 25 new); ruff --select F401 is clean; a built wheel contains the new package.
• New tests cover the registry, a golden check that the Claude Code command and
  environment equal what the engine built inline, a contract test that runs
  once per registered harness against recorded output in
  tests/fixtures/harness/ and fails when a harness ships none, and a harness
  whose events look nothing like Claude Code's, driven both through
  _stream_until and through the solver under inspect.
• Each test was checked by breaking the code it covers and seeing it fail: no
  event translation, ignoring the harness's skills directory, changing the
  Claude command, always writing the prompt to stdin, sharing one translator
  across cases, bypassing the translator on the non-streaming path.
• The Claude Code fixture is synthetic (hand-written to the documented
  stream-json shape) and says so in expected.json. The Copilot adapter in the
  follow-up PR ships real recordings.

I have only run the suite on Windows. One existing test,
TestTheHostLegActuallyKillsTheProcess.test_the_work_after_the_decision_never_happens,
is flaky there and fails on a clean checkout of main too (3 of 8 runs): it gives
a child a 1.5 s sleep and asserts the kill took under 1.4 s. Unrelated to this
change, and not touched here.

Not in this PR

Choosing a harness from the command line, and moving the preflight, credential
check, spend flags, CI model pin and judge behind the contract, come with the
second harness (GitHub Copilot CLI), which follows as a separate PR on top of
this one. docs/harnesses.md lists them.

[volen-silo]

@danielholanda this is PR 1 of the plan in #4: the harness contract and the Claude Code adapter, with no CLI or report change. Could you review it when you have time? #34 (Copilot CLI as the second harness) is a draft stacked on it.

[danielholanda]

I can confirm I'm aligned in terms of direction of extending the harnesses supported by skillscope. Just please make sure that folks that are using our reusable workflow only un claude code tests by default, as increasing the default test matrix would generate a huge amount of additional compute that is not feasible for many repos at this point.
@rominf / @johnl-amd would love if you folks could review this PR

[rominf]

Thanks for this. The direction is right, and the contract is very close to what we need: command, env, skills_dir, prompt_on_stdin, normalize / normalizer() into the stream-json subset, a harness package with no inspect_ai import, and a contract test that runs on recorded fixtures for every registered harness. I'd like to keep all of that.

I'm requesting changes for one reason. As written, Harness only works on the host, and the plan on #4 adds sandbox legs for other agents through inspect_swe. That would make every agent exist twice in this repo: once as a Harness for the host, and once as an inspect_swe agent with its own command line, environment, skill staging and event handling. Claude Code already exists twice (claude_code_no_sandbox and inspect_swe.claude_code()), and that is where the sandbox-only workarounds come from: Inspect's cost_limit instead of --max-budget-usd, the room check that can't run because the init event is lost, and the approver-based stop. I'd rather the harness define the agent once for both places, and the sandbox only decide where the process starts.

The changes needed in this PR are small: four host assumptions in the record, plus one paragraph in the docs. Everything else below is direction for follow-ups, so we agree on it before #34 and a Codex adapter are built on top.

Requested changes

1. ClaudeCode.command() puts a host path in the argv. It starts with shutil.which(self.binary), which is an absolute path on the runner and wrong inside a container. Emit self.binary and let the thing that starts the process resolve it (inline comment).
2. ClaudeCode.env() copies the whole host environment. agent.claude_env() starts from dict(os.environ), so PATH, HOME and every API key on the runner would go into a container. env() should return only what the agent needs (CLAUDE_CODE_MAX_RETRIES, CLAUDE_CONFIG_DIR). The host runner can then put it on top of os.environ, so host behaviour doesn't change (inline comment).
3. Harness.require() checks the runner's PATH. That's right on the host and meaningless for a container image. Either document it as the host check, or let the caller say where it's checking (inline comment).
4. RunSpec.config_dir is a host Path. It should be a path where the agent runs, as a str, like workspace already is (inline comment).
5. docs/harnesses.md: say that the harness is used by both legs. Something like: "A harness defines the agent once. On the host the engine starts command(spec) as a subprocess. A sandbox leg starts the same command inside the container rather than delegating to an inspect_swe agent." Please also add the sandbox leg to "What is not yet behind the contract".

The golden test (Claude's command and environment equal what the engine built inline) will need adjusting for 1 and 2. Comparing env() plus os.environ against the old inline environment should keep its intent.

Why not inspect_swe for the sandbox leg

inspect_swe.codex_cli() and friends are inspect_ai.agent.Agents, and they work well for what they're for. They don't fit the measurement skillscope makes:

• The command line is closed. Unknown flags raise TypeError, so --max-budget-usd can't reach the CLI (see meridianlabs-ai/inspect_swe#178, closed without a passthrough).
• The raw stream stays inside the agent. It goes to an internal consumer and comes out as Inspect messages after the run. That's why the init event never reaches us, and why the early stop on that leg depends on an approver, which may drop the tool call that triggered it.
• It rewrites $HOME/.claude/settings.json. _seed_claude_config does this and ignores the env= you pass. That's why verify.require() refuses to run under the local sandbox.
• It would be a second definition of each agent. Its own flags, binary version, skill directory and event parsing, which can drift from the host harness silently.

None of that is needed. Everything inspect_swe uses to run a CLI in a container is public in inspect_ai. sandbox().exec_remote(cmd, ExecRemoteStreamingOptions(...)) streams stdout from the container and can kill() the process; inspect_swe.claude_code() uses exactly that call. sandbox_agent_bridge (in inspect_ai.agent) provides the model proxy.

Where this is today

flowchart TB
  CMD["skillscope routing / behavioral"]
  CMD --> E1["--engine claude-code-no-sandbox"]
  CMD --> E2["--engine claude-code"]

  E1 --> H["Harness (this PR)<br/>command, env, skills_dir, normalize, require"]
  H --> HC["ClaudeCode"]
  H -.-> HP["Copilot (#34)"]
  HC --> R1["harness_solver / _stream_until<br/>asyncio subprocess on the host<br/>streaming, early stop, room check"]
  HP -.-> R1
  R1 --> P1["claude on the host<br/>own credentials, --max-budget-usd"]

  E2 --> SWE["inspect_swe.claude_code()<br/>its own argv, env, skill install, event consumer"]
  SWE --> BR["sandbox_agent_bridge<br/>Inspect's Linux proxy binary in the container"]
  SWE --> P2["claude in the container"]
  SWE -.-> SWE2["planned: inspect_swe.codex_cli()<br/>a second Codex definition"]

  W["only on this leg:<br/>cost_limit instead of --max-budget-usd,<br/>no init event, approver stop"] -.- SWE

Loading

Proposed: one record, two transports, optional proxy

flowchart TB
  CMD["skillscope routing / behavioral<br/>--harness claude-code | copilot | codex<br/>--engine host | sandbox   (later: --proxy)"]
  CMD --> REC["Harness, one per CLI<br/>command, env, skills_dir, normalizer<br/>(later: proxy_env(port))"]
  REC --> RUN["one runner<br/>stage skill, read events, stop rule,<br/>room check, detect_activation"]

  RUN -->|host| HOST["host transport<br/>asyncio subprocess, instant kill<br/>Windows, macOS, Linux"]
  RUN -->|sandbox| CONT["container transport<br/>sandbox().exec_remote(stream=True)<br/>polls every 5 s or more, Linux guest"]

  HOST -->|"default"| H1["CLI's own credentials"]
  HOST -->|"--proxy"| HP["host proxy, in our process"]
  CONT -->|"default"| CP["sandbox_agent_bridge"]
  CONT -->|"no proxy"| C1["key passed into the container"]

  HP --> BR["Inspect bridge: generate_anthropic, generate_responses, ...<br/>eval model, ModelEvents in the log,<br/>message_limit and cost_limit bind"]
  CP --> BR

  INS["inspect_ai: Task, sandbox providers,<br/>.eval logs, scorers, judge"] --- RUN

Loading

We keep almost all of inspect_ai: tasks, samples, sandbox providers (Docker, Compose, local), exec_remote, .eval logs, time_limit, concurrency, scorers, the judge, and the bridge. inspect_swe would be dropped only after the container transport matches --engine claude-code on Claude, and the three workarounds above would go with it. Until then --engine claude-code stays as it is.

The proxy on Windows

The container proxy is only Linux because Inspect ships it as a prebuilt Linux binary. Its source, src/inspect_sandbox_tools/_agent_bridge/proxy.py in inspect_ai (MIT), is about 2,400 lines of standard-library asyncio, binds 127.0.0.1, and serves /v1/messages, /v1/responses and /v1/chat/completions. model_proxy_server(port, call_bridge_model_service_async=...) already accepts a replacement for the function that talks to Inspect, and Inspect's own tests use that hook. On the host, the replacement can call the bridge functions in inspect_ai.agent._bridge.sandbox.service directly:

sequenceDiagram
  autonumber
  participant R as skillscope runner
  participant P as host proxy (proxy.py on 127.0.0.1)
  participant B as Inspect bridge (generate_anthropic)
  participant M as eval model (Inspect model API)
  participant C as claude CLI (host subprocess)

  R->>B: SandboxAgentBridge(state, model)
  R->>P: model_proxy_server(port, call)
  R->>C: start with ANTHROPIC_BASE_URL=127.0.0.1:port
  loop each model turn
    C->>P: POST /v1/messages
    P->>B: call("generate_anthropic", json_data, headers)
    B->>M: generate(), logged as a ModelEvent
    M-->>B: ModelOutput
    B-->>P: Anthropic JSON reply
    P-->>C: streamed reply
    C-->>R: stream-json event on stdout
    R->>R: normalize, check stop rule
  end
  alt skill activated, or limit hit
    R->>C: kill the process group
  end
  R->>P: shut down

Loading

For comparison, this is what sandbox_agent_bridge does in a container:

sequenceDiagram
  autonumber
  participant R as skillscope runner (host)
  participant S as Inspect sandbox service (host)
  participant F as request/response files<br/>/var/tmp/sandbox-services
  participant P as proxy binary (container)
  participant C as claude CLI (container)

  R->>S: sandbox_agent_bridge(state)
  S->>P: exec_remote(model_proxy)
  R->>C: exec_remote(claude, ANTHROPIC_BASE_URL=localhost:13131)
  loop each model turn
    C->>P: POST /v1/messages
    P->>F: write request file
    S->>F: poll (about every 2 s), read request
    S->>S: generate_anthropic, eval model
    S->>F: write response file
    P->>F: poll (every 0.1 s), read response
    P-->>C: streamed reply
    C-->>R: stdout via exec_remote (polled every 5 s or more)
  end

Loading

Doing this on the host has costs:

• It uses private Inspect APIs, SandboxAgentBridge and the generate_* functions, and needs proxy.py vendored. We'd pin inspect_ai and add a round-trip test.
• Our wrapper has to end the sample on limits and refusals, the way the bridge's monitor task does.
• The proxy is an open loopback port while a case runs, so it should check a per-run token.
• Each concurrent case needs its own port.

Long term, I'd rather propose a public host_agent_bridge() upstream. None of this is needed for this PR. proxy_env(port) can be added when the proxy lands.

Suggested order

1. This PR, with the five changes above.
2. A container transport for harness_solver using exec_remote, run against --engine claude-code on Claude and compared case by case.
3. #34 (Copilot) lands host-only, because the container transport from step 2 doesn't exist yet. Once it does, Copilot gets a sandbox leg the same way as Claude: the CLI installed in the image and GITHUB_TOKEN passed in. No Copilot-specific sandbox code.
4. Codex as a Harness, not as inspect_swe.codex_cli().
5. The optional proxy: sandbox_agent_bridge in the container, then the host proxy.

Two things I haven't verified and would check first: whether Copilot CLI accepts a custom model endpoint (if not, it runs without the proxy), and whether claude --max-budget-usd still prices correctly when it's sending to the proxy.

The default stays Claude Code only, so the reusable workflow's compute doesn't grow, per @danielholanda's note.

[rominf]

This resolves the binary on the runner and bakes an absolute host path into the argv, so the same command can't be started inside a container. Could command() emit self.binary and let the caller resolve it? The host runner would keep shutil.which (and the not-found error), and a sandbox runner would leave it to the guest's PATH.

[rominf]

agent.claude_env() starts from dict(os.environ), so the harness's environment is the whole runner environment: PATH, HOME, and every key on the machine. That's fine on the host, but a sandbox leg would carry all of it into the container. Suggest env() returns only what Claude Code needs (CLAUDE_CODE_MAX_RETRIES=0 and CLAUDE_CONFIG_DIR), and the host runner does {**os.environ, **harness.env(spec)}. Host behaviour stays the same, and the golden test can compare that merge with what the engine used to build inline.

[rominf]

This checks the runner's PATH, which is the right question on the host and the wrong one for a container image. Either name it as the host check (docstring: "on the host"), or let the caller say where to look, so a sandbox leg can check the image instead of failing on a runner that doesn't have the CLI installed.

[rominf]

workspace is a str naming a path wherever the agent runs, but config_dir is a host Path. Could it be a str with the same meaning, so a sandbox leg can pass a directory inside the container?

[rominf]

Agreed on the split between where and what. Could this page also say that the harness defines the agent for both answers to where? Something like: "On the host the engine starts command(spec) as a subprocess; a sandbox leg starts the same command inside the container rather than delegating to an inspect_swe agent." Otherwise each new harness invites a second, inspect_swe-based definition for the sandbox. Details and diagrams in the review summary.

[volen-silo]

Thanks, this is a clear review and I agree with all five requested changes. They are in 96d63b9:

1. command() now emits self.binary. A new host_command(spec) resolves it with shutil.which for a run started on the host. On Windows that lookup is what finds an npm-installed .cmd shim, and the existing stub-claude end-to-end test covers it.
2. env() returns only what the harness sets (CLAUDE_CODE_MAX_RETRIES, CLAUDE_CONFIG_DIR). host_env(spec) puts it on top of os.environ, so host behaviour is unchanged. The golden test compares that merge with what the engine built inline, including that the runner's own CLAUDE_CODE_MAX_RETRIES still overrides the default and that the isolation setting still beats the runner's own CLAUDE_CONFIG_DIR.
3. require() is documented as the host check.
4. RunSpec.config_dir is a str, and the engine hands the harness strings, not host Paths.
5. docs/harnesses.md says a harness defines the agent once for both legs, and lists the sandbox leg under what is not yet behind the contract.

The contract tests now run for every registered harness: the command names its executable and does not locate it, env() carries no ambient variables, and on the host the harness's settings go on top of the runner's. I broke each of the five changes in turn and a test failed each time.

On the direction: I agree with one definition per agent and with the order you propose. In particular I withdraw the idea in my RFC of a sandbox leg through inspect_swe. Codex should be a Harness, not inspect_swe.codex_cli().

#34 stays host-only until a container transport exists. I have rebased it onto this branch and it passes: the Copilot adapter needed the same two changes (a bare executable, and settings-only env()), so its diff is now just the Copilot work on top of this one.

One thing to keep in mind for the container transport. Besides command and env, #34 has a few hooks that also start the CLI on the host: preflight, version, complete (the judge) and room. Copilot announces no skills at session start, so room runs copilot skill list --json. A container leg would need those to go through the same transport.

On your two open questions for Copilot: its own help mentions a bring-your-own-key provider configuration (COPILOT_PROVIDERS_CONFIG and the COPILOT_PROVIDER_* variables), so a custom model endpoint looks possible, but I have not tried it. I have not looked at --max-budget-usd through a proxy.

And per @danielholanda's note: the default stays Claude Code only, and nothing in #33 or #34 changes the reusable workflow's matrix.

[rominf]

Thanks, all five changes are in 96d63b9, and they're done the way I'd hoped:

1. command() names self.binary. host_command() resolves it with shutil.which only for a host run, and keeps the bare name if nothing is found, so require() still gives the error. The note about npm's .cmd shims on Windows is a good reason to keep shutil.which there.
2. env() is settings-only: claude_settings() is CLAUDE_CODE_MAX_RETRIES plus CLAUDE_CONFIG_DIR. host_env() puts it on top of os.environ. The golden test pins the old inline behaviour, including the runner's own CLAUDE_CODE_MAX_RETRIES winning and the isolation directory beating the runner's CLAUDE_CONFIG_DIR. claude_env() is still used by the reachability probe, which also runs on the host, so that's fine.
3. require() is documented as the host check.
4. RunSpec.config_dir is a str, and harness_solver converts at the boundary. The new end-to-end test checks that the harness receives strings.
5. docs/harnesses.md now says a harness defines the agent once for both legs, and lists the sandbox leg under what is not yet behind the contract.

The contract tests running for every registered harness are the right guard: the command names its executable, env() carries no ambient variables (including SYSTEMROOT), and the host merge keeps the harness's values. CI is green on Ubuntu and Windows, on Python 3.10 and 3.12.

One thing to note, not a change request: CLAUDE_CODE_MAX_RETRIES is the one runner variable that env() still carries into a container, because claude_settings() reads the override from os.environ. I think that's what we want, since it lets a runner raise retries on either leg, and the docstring says so.

Thanks also for withdrawing the inspect_swe sandbox leg and for rebasing #34. Your point about Copilot's other hooks (preflight, version, complete for the judge, and room, which runs copilot skill list --json) is important for the container transport. Those should go through the same transport as command, so a sandboxed Copilot run asks the image rather than the runner. I'll keep that in mind when we design it. The BYOK variables (COPILOT_PROVIDERS_CONFIG, COPILOT_PROVIDER_*) look like the way to put Copilot behind the proxy later.

Approving.