Repository navigation
Add --agent strands-harness, the Strands team's create_harness() agent - #212
Merged
Merged
Conversation
The existing --agent strands builds its agent by hand from strands.Agent. The Strands team now ships create_harness() in the strands-harness package, a preconfigured coding agent. This adds it as a separate harness with its own results folder, so the hand-built runs stay comparable. strands_agent_runner.py gains --harness. In that mode it calls create_harness() with benchmark-safe settings: the shell, read, write and edit tools and the harness's context manager, prompt caching, skills and todos and environment plugins as shipped; web_fetch, web_search, subagent, programmatic_tool_caller, memory and session off. The runner passes its own model instance (so the empty-tools fix and window limit still apply) and its JSON event hooks, and turns off its own tool-output cap, since the harness offloads large results itself. The swe3 skill names Claude Code's tools. Under the harness, minicpm5-2b called `bash` on every attempt with the tool map only in the instructions. The map now also leads the first message, and `bash` is registered as an alias of the harness shell (make_shell(name="bash")). The agent is wired through runner_config (strands-harness slug, is_strands covers both builds), run-swe-headless.py, token_accounting, both run scripts and the report labels. strands-harness joins the optional strands dependency group. docs/strands-setup.md describes the new agent. Also rebuild vend/swe-router/models.json, whose source_commit still named the pre-squash commit of #210, which fails test_build_vended_models on main.
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds
--agent strands-harness, which drives a benchmark task with the Strands team'screate_harness()agent from thestrands-harnesspackage. The existing--agent strandsbuilds its agent by hand fromstrands.Agent; the Strands team suggested trying their harness in place of that. The new agent gets its own results folder (<model>/strands-harness/), so the hand-built Strands results stay comparable.How the agent is built
strands_agent_runner.pygains--harness. With it,_build_harness_agentcalls:shell,read,write,edit, context manager (offloads large tool results), caching, skills,todos,environmentweb_fetch,web_searchsubagent,programmatic_tool_caller/swe3is single-agentmemory,sessionThe runner turns off its own 100,000-character tool-output cap in this mode, since the harness offloads large results itself.
Tool names
The
swe3skill names Claude Code's tools (Bash,Edit,Read). Under the harness,minicpm5-2bcalledbashon every attempt, and vLLM drops a call to an unknown tool, which ends the attempt. Three changes, each measured on thehello-worldsmoke task:instructionsonlybashwithin 6 turns, 0/6 artifactsbashcallbashregistered as an alias of the harness shellOther changes
runner_config.py:AGENT_STRANDS_HARNESS, itsstrands-harnessslug,is_strands_harness;is_strandscovers both builds, so they share the run, parse and pricing path.run-swe-headless.py: passes--harness, checks thatstrands_harnessis installed, names the agent in log lines.token_accounting.py:strands-harnesscounts are disjoint, as forstrands.run-e2e-benchmark.sh,run-multi-model-benchmark.sh,gen_agent_report.py: accept and label the new agent.pyproject.toml,uv.lock:strands-harness>=0.1.2,<0.2joins the optionalstrandsgroup.docs/strands-setup.md: a section on the new agent, what it keeps and turns off, and why the benchmark uses the Python library and not thestrandsCLI (print mode reports no token counts, and the CLI loads MCP servers from other agents' config files).vend/swe-router/models.json: rebuilt. Itssource_commitstill named the pre-squash commit of Add Opus 5.5 and Kimi K3 on Bedrock to the omp/swe3 frontier, and count Path 2 runs as metered #210, which failstest_build_vended_modelsonmaintoday.Testing
uv run python -m unittest discover -s testspasses inbenchmarks/andself-hosted/vllm/; ruff, mypy and Bandit are clean, and thesecurity-checkgate passed.minicpm5-2bonmcp-gateway-registry-v2under--agent strands-harnessis in progress on this code; its first 5 tasks each finished with all six artifacts on the first attempt. Its results will follow in a separate PR once judged.