Skip to content

harness: add Strands Agents as a third harness option (--agent strands) - #194

Merged
aarora79 merged 3 commits into
aarora79:mainfrom
shekharprateek:feat/strands-harness
Sep 29, 2026
Merged

aarora79 merged 3 commits into
aarora79:mainfrom
shekharprateek:feat/strands-harness

Conversation

@shekharprateek

Copy link
Copy Markdown

Summary

Adds Strands Agents as a third harness alongside claude/pi/omp and codex. Pass --agent strands to run-e2e-benchmark.sh to drive a benchmark task with the Strands Agents Python SDK.

What's new

Core runner (benchmarks/scripts/strands_agent_runner.py)

  • Builds a Strands Agent with BedrockModel (provider=bedrock) or OpenAIModel (provider=endpoint)
  • Uses the SDK's built-in shell and file_editor tools plus the AgentSkills plugin pointed at .claude/skills/swe3/
  • SummarizingConversationManager keeps context bounded; context_window and auto_compact_fraction are configurable
  • Emits JSON-line events on stdout (model_call, tool_call, result) for the harness to consume
  • Writes strands-stream.jsonl alongside the other task artifacts

Harness integration

  • run-e2e-benchmark.sh: new strands agent choice; auto-installs the optional strands uv dependency group if missing; passes --group strands to uv run
  • run-swe-headless.py: routes agent=strands to the new runner, parses its event stream, extracts token counts from the result event; corrects OpenAI-endpoint double-counting (prompt_tokens includes cache reads)
  • runner_config.py, runner.example.yaml: expose context_window / auto_compact_fraction
  • token_accounting.py, bedrock_pricing.py: Strands reports no cost; pricing extended to Claude 5 family, Haiku 4.5, and cross-region profile rates (us./ global. prefixes)
  • run-multi-model-benchmark.sh, run-benchmark-batch.sh: thread through --agent strands

Docs and tests

  • docs/strands-setup.md: install, credentials, first-run, token and cost accounting
  • AGENTS.md, SKILL.md: strands listed as a supported harness
  • New: test_strands_agent_runner.py; extended test_run_swe_headless.py and test_bedrock_pricing.py

Initial benchmark data

  • claude-haiku-4-5/strands/swe3/Hello-World — smoke run, 6 artifacts produced
  • claude-opus-4-8/strands/swe3/Hello-World — smoke run, 6 artifacts + eval scores
  • claude-opus-4-8/strands/swe3/mcp-gateway-registry — two tasks; remove-faiss hit EventLoopException: maximum recursion depth exceeded at 320 cycles (context-window tuning needed)

Known issue

The remove-faiss task ran away (320 agent cycles, 62M tokens mostly cache-read). The context_window config flag exists to address this — the default needs tuning for longer tasks.

shekharprateek and others added 3 commits September 29, 2026 14:09
Add strands_agent_runner.py, which drives one SWE benchmark task using the
Strands Agents Python SDK. The runner uses BedrockModel for provider=bedrock
and OpenAIModel for provider=endpoint (vLLM or the LiteLLM proxy), the
built-in shell and file_editor tools, and the AgentSkills plugin pointed at
the swe3 skill directory. It emits JSON-line events (model_call, tool_call,
result) on stdout for the harness to consume.

- run-e2e-benchmark.sh: accept --agent strands, auto-install the optional
  strands uv dependency group if missing, pass --group strands to uv run
- run-swe-headless.py: route agent=strands to the new runner, parse its
  JSON event stream, extract final token counts from the result event;
  handle Bedrock vs OpenAI-endpoint double-counting differences
- runner_config.py: expose context_window and auto_compact_fraction for
  Strands SummarizingConversationManager
- token_accounting.py: correct OpenAI-compatible endpoint double-counting
- bedrock_pricing.py: extend to Claude 5 family, Haiku 4.5, cross-region
  profile rates (us./ global. prefixes)
- run-multi-model-benchmark.sh, run-benchmark-batch.sh: pass --agent strands
- benchmarks/pyproject.toml: add strands optional dependency group
- docs/strands-setup.md: install, credential, first-run, and accounting docs
- AGENTS.md, SKILL.md: document strands as a supported harness
- tests: extend bedrock_pricing, run_swe_headless, add test_strands_agent_runner
- benchmark data: initial smoke runs for claude-haiku-4-5 and claude-opus-4-8
Main gained the private-repo pre-flight, the env-sourced gateway key and the
onboard-benchmark skill (aarora79#193) while this branch was open.

Textual conflicts, all three in .claude/skills/benchmark/SKILL.md, where this
branch and aarora79#193 both rewrote the agent documentation:

- The agent input. Took aarora79#193's table form and added a `strands` row, rather than
  keeping the older prose list. The row records what the orchestrator actually
  enforces: strands runs on either provider, and unlike every other agent it has
  no CLI binary. Dropped the stale claim that pi cannot run --provider bedrock,
  which aarora79#193 corrected.
- The CLI pre-flight. Took aarora79#193's structure and kept this branch's strands fact
  as a following paragraph: there is no binary to find on PATH, the SDK runs
  in-process, and the orchestrator installs the optional dependency group with
  `uv sync --group strands`.
- The orchestrator command line. Took both additions, strands in the agent list
  and aarora79#193's --skill flag.

One semantic conflict git could not see. `_build_strands_env` sets the Strands
API key from `config.api_key` directly, which bypasses the `api_key_env` support
aarora79#193 added and would send the "local" placeholder to a gateway that needs a real
credential. Routed through `resolved_api_key()`, matching the other six call
sites. Verified: with api_key_env set, the strands env now carries the token from
the environment rather than the placeholder.

Also added strands to the harness-selection table in the onboard-benchmark
skill, which aarora79#193 introduced and this branch had no way to know about, so the
skill does not silently omit an agent.
@aarora79
aarora79 merged commit 04d86ba into aarora79:main Sep 29, 2026
7 checks passed
shekharprateek pushed a commit to shekharprateek/agentic-coding-harness-benchmarks that referenced this pull request Oct 2, 2026
21 tasks, all scored, mean 81.21, mean cost $2.46/task.
First benchmark of the Strands Agents harness (aarora79#194) on a full dataset.
aarora79 pushed a commit that referenced this pull request Oct 4, 2026
…set (#195)

* benchmark(opus-5-5): Opus 5.5 on Bedrock via strands/swe3, v2 dataset

21 tasks, all scored, mean 81.21, mean cost $2.46/task.
First benchmark of the Strands Agents harness (#194) on a full dataset.

* benchmark(opus-5-5): Opus 5.5 on Bedrock via omp/swe3, v2 dataset

21 tasks, all scored, mean 81.16, mean cost $4.44/task.
Direct harness comparison with the strands run (81.21 at $2.46/task):
same quality, strands costs 45% less from prompt caching.

---------

Co-authored-by: Prateek Sinha <prsinp@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants