Skip to content

benchmark(sonnet-5-5): Sonnet 5.5 on Bedrock via strands/swe3, v2 dataset - #203

Merged
aarora79 merged 1 commit into
aarora79:mainfrom
shekharprateek:benchmark/sonnet-5-5-strands-swe3-v2
Oct 5, 2026
Merged

aarora79 merged 1 commit into
aarora79:mainfrom
shekharprateek:benchmark/sonnet-5-5-strands-swe3-v2

Conversation

@shekharprateek

Copy link
Copy Markdown

Summary

  • Benchmark Claude Sonnet 5.5 (global.anthropic.claude-sonnet-5-5) on Amazon Bedrock using the Strands Agents harness (--agent strands), swe3 single-agent skill, on the v2 dataset (21 tasks across 4 complexity tiers).
  • Mean score: 76.73 across all 21 tasks, no failures, $0.96 mean cost per task.
  • All 21 tasks produced 6/6 artifacts; top scores on consistent-csrf-across-toggle-endpoints (86.0) and honor-cloud-provider-override-in-ui (86.0); weakest on macos-setup-python-version-precheck (51.6).
Complexity Tasks Score range Mean
Trivial 5 51.6 - 85.2 75.1
Low 6 67.2 - 86.0 76.9
Medium 5 68.2 - 86.0 76.3
High 5 76.2 - 85.6 81.2

Test plan

  • All 21 tasks completed with 6/6 artifacts
  • All 21 tasks scored by codex judge (0 failures)
  • run-summary.json written with per-task scores, turns, and cost
  • Pre-flight checks passed (repo reachability, AWS credentials, codex judge proven)

…aset

Score 76.73 mean across 21 tasks (no failures), $0.96 mean cost per task.
Global cross-region inference profile on Amazon Bedrock, Strands Agents
harness, swe3 single-agent skill, mcp-gateway-registry-v2 dataset.
@aarora79
aarora79 merged commit 14a41fe into aarora79:main Oct 5, 2026
5 checks passed
aarora79 added a commit that referenced this pull request Oct 5, 2026
model_to_slug only stripped two-letter region prefixes (us., eu.), so the
Sonnet 5.5 run from #203, which used the global. cross-region profile, landed
in swe-benchmark-data/global.anthropic.claude-sonnet-5-5/. The prefix pattern
now also matches apac., us-gov. and global.; a replay of every stored model id
shows that folder is the only one whose slug changes.

Move that folder to claude-sonnet-5-5/ with git mv and set its model_slug
fields to match. Scores, metrics and the raw model ids are unchanged.

Regenerate the Strands results page and charts, which now cover three models:
claude-opus-5-5 (81.21), claude-sonnet-5-5 (76.73) and minicpm5-2b (45.28).

Closes #205
aarora79 added a commit that referenced this pull request Oct 5, 2026
* Add the Strands harness results page and list Strands in the README

Generate docs/harness-strands-swe3.md with gen_agent_report.py and the three
per-harness charts. The page covers both Strands runs on the v2 dataset:
claude-opus-5-5 on Amazon Bedrock (81.21, 21/21) and minicpm5-2b on vLLM
(45.28, 21/21). A hand-written section notes that the two rows used different
runner versions and records the two hello-world sanity runs.

The README now lists strands among the supported harnesses and links its
setup and results pages, and the results index links the new page.
gen_agent_report.py gains a label and a setup-doc link for strands.

* Strip global. from model slugs and add Sonnet 5.5 to the Strands page

model_to_slug only stripped two-letter region prefixes (us., eu.), so the
Sonnet 5.5 run from #203, which used the global. cross-region profile, landed
in swe-benchmark-data/global.anthropic.claude-sonnet-5-5/. The prefix pattern
now also matches apac., us-gov. and global.; a replay of every stored model id
shows that folder is the only one whose slug changes.

Move that folder to claude-sonnet-5-5/ with git mv and set its model_slug
fields to match. Scores, metrics and the raw model ids are unchanged.

Regenerate the Strands results page and charts, which now cover three models:
claude-opus-5-5 (81.21), claude-sonnet-5-5 (76.73) and minicpm5-2b (45.28).

Closes #205

* Add Haiku 4.5 to the Strands results page

Regenerate the Strands page and charts after merging #206, which adds
claude-haiku-4-5 under strands/swe3 on the v2 dataset (51.20 over 20 of 21
tasks). The notes record its one failure, a RecursionError after 973 turns on
honor-cloud-provider-override-in-ui.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants