Repository navigation
benchmark(sonnet-5-5): Sonnet 5.5 on Bedrock via strands/swe3, v2 dataset - #203
Merged
aarora79 merged 1 commit intoOct 5, 2026
Conversation
…aset Score 76.73 mean across 21 tasks (no failures), $0.96 mean cost per task. Global cross-region inference profile on Amazon Bedrock, Strands Agents harness, swe3 single-agent skill, mcp-gateway-registry-v2 dataset.
aarora79
added a commit
that referenced
this pull request
Oct 5, 2026
model_to_slug only stripped two-letter region prefixes (us., eu.), so the Sonnet 5.5 run from #203, which used the global. cross-region profile, landed in swe-benchmark-data/global.anthropic.claude-sonnet-5-5/. The prefix pattern now also matches apac., us-gov. and global.; a replay of every stored model id shows that folder is the only one whose slug changes. Move that folder to claude-sonnet-5-5/ with git mv and set its model_slug fields to match. Scores, metrics and the raw model ids are unchanged. Regenerate the Strands results page and charts, which now cover three models: claude-opus-5-5 (81.21), claude-sonnet-5-5 (76.73) and minicpm5-2b (45.28). Closes #205
aarora79
added a commit
that referenced
this pull request
Oct 5, 2026
* Add the Strands harness results page and list Strands in the README Generate docs/harness-strands-swe3.md with gen_agent_report.py and the three per-harness charts. The page covers both Strands runs on the v2 dataset: claude-opus-5-5 on Amazon Bedrock (81.21, 21/21) and minicpm5-2b on vLLM (45.28, 21/21). A hand-written section notes that the two rows used different runner versions and records the two hello-world sanity runs. The README now lists strands among the supported harnesses and links its setup and results pages, and the results index links the new page. gen_agent_report.py gains a label and a setup-doc link for strands. * Strip global. from model slugs and add Sonnet 5.5 to the Strands page model_to_slug only stripped two-letter region prefixes (us., eu.), so the Sonnet 5.5 run from #203, which used the global. cross-region profile, landed in swe-benchmark-data/global.anthropic.claude-sonnet-5-5/. The prefix pattern now also matches apac., us-gov. and global.; a replay of every stored model id shows that folder is the only one whose slug changes. Move that folder to claude-sonnet-5-5/ with git mv and set its model_slug fields to match. Scores, metrics and the raw model ids are unchanged. Regenerate the Strands results page and charts, which now cover three models: claude-opus-5-5 (81.21), claude-sonnet-5-5 (76.73) and minicpm5-2b (45.28). Closes #205 * Add Haiku 4.5 to the Strands results page Regenerate the Strands page and charts after merging #206, which adds claude-haiku-4-5 under strands/swe3 on the v2 dataset (51.20 over 20 of 21 tasks). The notes record its one failure, a RecursionError after 973 turns on honor-cloud-provider-override-in-ui.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
global.anthropic.claude-sonnet-5-5) on Amazon Bedrock using the Strands Agents harness (--agent strands), swe3 single-agent skill, on the v2 dataset (21 tasks across 4 complexity tiers).consistent-csrf-across-toggle-endpoints(86.0) andhonor-cloud-provider-override-in-ui(86.0); weakest onmacos-setup-python-version-precheck(51.6).Test plan
run-summary.jsonwritten with per-task scores, turns, and cost