Skip to content

Add --agent strands-harness, the Strands team's create_harness() agent - #212

Merged
aarora79 merged 1 commit into
mainfrom
feat/strands-harness
Oct 7, 2026
Merged

aarora79 merged 1 commit into
mainfrom
feat/strands-harness

Conversation

@aarora79

@aarora79 aarora79 commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Summary

This PR adds --agent strands-harness, which drives a benchmark task with the Strands team's create_harness() agent from the strands-harness package. The existing --agent strands builds its agent by hand from strands.Agent; the Strands team suggested trying their harness in place of that. The new agent gets its own results folder (<model>/strands-harness/), so the hand-built Strands results stay comparable.

How the agent is built

strands_agent_runner.py gains --harness. With it, _build_harness_agent calls:

create_harness(
    model=_build_model(args),          # our model: keeps the empty-tools fix (#198) and window limit (#197)
    instructions=HARNESS_INSTRUCTIONS, # non-interactive note + the tool map (#202)
    tools=[bash_alias],                # make_shell(name="bash"): the harness shell under a second name
    builtin_tools=["shell", "read", "write", "edit"],
    builtin_plugins=["todos", "environment"],
    session=False,
    memory=False,
    hooks=[hooks],                     # our JSON event stream
    callback_handler=None,
)
Harness default In a benchmark run Why
shell, read, write, edit, context manager (offloads large tool results), caching, skills, todos, environment kept As shipped
web_fetch, web_search off The agent could look up the upstream fix
subagent, programmatic_tool_caller off /swe3 is single-agent
memory, session off Nothing may carry over between runs or tasks

The runner turns off its own 100,000-character tool-output cap in this mode, since the harness offloads large results itself.

Tool names

The swe3 skill names Claude Code's tools (Bash, Edit, Read). Under the harness, minicpm5-2b called bash on every attempt, and vLLM drops a call to an unknown tool, which ends the attempt. Three changes, each measured on the hello-world smoke task:

Change Result
Tool map in the harness instructions only Both attempts ended on bash within 6 turns, 0/6 artifacts
Map also at the top of the first message 44 turns, 5/6 artifacts, ended on one bash call
Plus bash registered as an alias of the harness shell No attempt ends on an unknown tool

Other changes

  • runner_config.py: AGENT_STRANDS_HARNESS, its strands-harness slug, is_strands_harness; is_strands covers both builds, so they share the run, parse and pricing path.
  • run-swe-headless.py: passes --harness, checks that strands_harness is installed, names the agent in log lines.
  • token_accounting.py: strands-harness counts are disjoint, as for strands.
  • run-e2e-benchmark.sh, run-multi-model-benchmark.sh, gen_agent_report.py: accept and label the new agent.
  • pyproject.toml, uv.lock: strands-harness>=0.1.2,<0.2 joins the optional strands group.
  • docs/strands-setup.md: a section on the new agent, what it keeps and turns off, and why the benchmark uses the Python library and not the strands CLI (print mode reports no token counts, and the CLI loads MCP servers from other agents' config files).
  • vend/swe-router/models.json: rebuilt. Its source_commit still named the pre-squash commit of Add Opus 5.5 and Kimi K3 on Bedrock to the omp/swe3 frontier, and count Path 2 runs as metered #210, which fails test_build_vended_models on main today.

Testing

  • New unit tests cover the new agent's config and command, the harness build (model instance, tools, alias, memory and session off), the tool map, and the cap switch. uv run python -m unittest discover -s tests passes in benchmarks/ and self-hosted/vllm/; ruff, mypy and Bandit are clean, and the security-check gate passed.
  • A full 21-task run of minicpm5-2b on mcp-gateway-registry-v2 under --agent strands-harness is in progress on this code; its first 5 tasks each finished with all six artifacts on the first attempt. Its results will follow in a separate PR once judged.

The existing --agent strands builds its agent by hand from strands.Agent.
The Strands team now ships create_harness() in the strands-harness package,
a preconfigured coding agent. This adds it as a separate harness with its
own results folder, so the hand-built runs stay comparable.

strands_agent_runner.py gains --harness. In that mode it calls
create_harness() with benchmark-safe settings: the shell, read, write and edit
tools and the harness's context manager, prompt caching, skills and todos and
environment plugins as shipped; web_fetch, web_search, subagent,
programmatic_tool_caller, memory and session off. The runner passes its own
model instance (so the empty-tools fix and window limit still apply) and its
JSON event hooks, and turns off its own tool-output cap, since the harness
offloads large results itself.

The swe3 skill names Claude Code's tools. Under the harness, minicpm5-2b
called `bash` on every attempt with the tool map only in the instructions.
The map now also leads the first message, and `bash` is registered as an
alias of the harness shell (make_shell(name="bash")).

The agent is wired through runner_config (strands-harness slug, is_strands
covers both builds), run-swe-headless.py, token_accounting, both run scripts
and the report labels. strands-harness joins the optional strands dependency
group. docs/strands-setup.md describes the new agent.

Also rebuild vend/swe-router/models.json, whose source_commit still named the
pre-squash commit of #210, which fails test_build_vended_models on main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant