Repository navigation
harness: add Strands Agents as a third harness option (--agent strands) - #194
Merged
Merged
Conversation
Add strands_agent_runner.py, which drives one SWE benchmark task using the Strands Agents Python SDK. The runner uses BedrockModel for provider=bedrock and OpenAIModel for provider=endpoint (vLLM or the LiteLLM proxy), the built-in shell and file_editor tools, and the AgentSkills plugin pointed at the swe3 skill directory. It emits JSON-line events (model_call, tool_call, result) on stdout for the harness to consume. - run-e2e-benchmark.sh: accept --agent strands, auto-install the optional strands uv dependency group if missing, pass --group strands to uv run - run-swe-headless.py: route agent=strands to the new runner, parse its JSON event stream, extract final token counts from the result event; handle Bedrock vs OpenAI-endpoint double-counting differences - runner_config.py: expose context_window and auto_compact_fraction for Strands SummarizingConversationManager - token_accounting.py: correct OpenAI-compatible endpoint double-counting - bedrock_pricing.py: extend to Claude 5 family, Haiku 4.5, cross-region profile rates (us./ global. prefixes) - run-multi-model-benchmark.sh, run-benchmark-batch.sh: pass --agent strands - benchmarks/pyproject.toml: add strands optional dependency group - docs/strands-setup.md: install, credential, first-run, and accounting docs - AGENTS.md, SKILL.md: document strands as a supported harness - tests: extend bedrock_pricing, run_swe_headless, add test_strands_agent_runner - benchmark data: initial smoke runs for claude-haiku-4-5 and claude-opus-4-8
Main gained the private-repo pre-flight, the env-sourced gateway key and the onboard-benchmark skill (aarora79#193) while this branch was open. Textual conflicts, all three in .claude/skills/benchmark/SKILL.md, where this branch and aarora79#193 both rewrote the agent documentation: - The agent input. Took aarora79#193's table form and added a `strands` row, rather than keeping the older prose list. The row records what the orchestrator actually enforces: strands runs on either provider, and unlike every other agent it has no CLI binary. Dropped the stale claim that pi cannot run --provider bedrock, which aarora79#193 corrected. - The CLI pre-flight. Took aarora79#193's structure and kept this branch's strands fact as a following paragraph: there is no binary to find on PATH, the SDK runs in-process, and the orchestrator installs the optional dependency group with `uv sync --group strands`. - The orchestrator command line. Took both additions, strands in the agent list and aarora79#193's --skill flag. One semantic conflict git could not see. `_build_strands_env` sets the Strands API key from `config.api_key` directly, which bypasses the `api_key_env` support aarora79#193 added and would send the "local" placeholder to a gateway that needs a real credential. Routed through `resolved_api_key()`, matching the other six call sites. Verified: with api_key_env set, the strands env now carries the token from the environment rather than the placeholder. Also added strands to the harness-selection table in the onboard-benchmark skill, which aarora79#193 introduced and this branch had no way to know about, so the skill does not silently omit an agent.
shekharprateek
pushed a commit
to shekharprateek/agentic-coding-harness-benchmarks
that referenced
this pull request
Oct 2, 2026
21 tasks, all scored, mean 81.21, mean cost $2.46/task. First benchmark of the Strands Agents harness (aarora79#194) on a full dataset.
4 tasks done
aarora79
pushed a commit
that referenced
this pull request
Oct 4, 2026
…set (#195) * benchmark(opus-5-5): Opus 5.5 on Bedrock via strands/swe3, v2 dataset 21 tasks, all scored, mean 81.21, mean cost $2.46/task. First benchmark of the Strands Agents harness (#194) on a full dataset. * benchmark(opus-5-5): Opus 5.5 on Bedrock via omp/swe3, v2 dataset 21 tasks, all scored, mean 81.16, mean cost $4.44/task. Direct harness comparison with the strands run (81.21 at $2.46/task): same quality, strands costs 45% less from prompt caching. --------- Co-authored-by: Prateek Sinha <prsinp@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Strands Agents as a third harness alongside
claude/pi/ompandcodex. Pass--agent strandstorun-e2e-benchmark.shto drive a benchmark task with the Strands Agents Python SDK.What's new
Core runner (
benchmarks/scripts/strands_agent_runner.py)AgentwithBedrockModel(provider=bedrock) orOpenAIModel(provider=endpoint)shellandfile_editortools plus theAgentSkillsplugin pointed at.claude/skills/swe3/SummarizingConversationManagerkeeps context bounded;context_windowandauto_compact_fractionare configurablemodel_call,tool_call,result) for the harness to consumestrands-stream.jsonlalongside the other task artifactsHarness integration
run-e2e-benchmark.sh: newstrandsagent choice; auto-installs the optionalstrandsuv dependency group if missing; passes--group strandstouv runrun-swe-headless.py: routesagent=strandsto the new runner, parses its event stream, extracts token counts from theresultevent; corrects OpenAI-endpoint double-counting (prompt_tokens includes cache reads)runner_config.py,runner.example.yaml: exposecontext_window/auto_compact_fractiontoken_accounting.py,bedrock_pricing.py: Strands reports no cost; pricing extended to Claude 5 family, Haiku 4.5, and cross-region profile rates (us./ global. prefixes)run-multi-model-benchmark.sh,run-benchmark-batch.sh: thread through--agent strandsDocs and tests
docs/strands-setup.md: install, credentials, first-run, token and cost accountingAGENTS.md,SKILL.md: strands listed as a supported harnesstest_strands_agent_runner.py; extendedtest_run_swe_headless.pyandtest_bedrock_pricing.pyInitial benchmark data
claude-haiku-4-5/strands/swe3/Hello-World— smoke run, 6 artifacts producedclaude-opus-4-8/strands/swe3/Hello-World— smoke run, 6 artifacts + eval scoresclaude-opus-4-8/strands/swe3/mcp-gateway-registry— two tasks;remove-faisshitEventLoopException: maximum recursion depth exceededat 320 cycles (context-window tuning needed)Known issue
The
remove-faisstask ran away (320 agent cycles, 62M tokens mostly cache-read). Thecontext_windowconfig flag exists to address this — the default needs tuning for longer tasks.