Repository navigation
Add the Strands results page and fix global. model slugs - #207
Merged
Merged
Conversation
Generate docs/harness-strands-swe3.md with gen_agent_report.py and the three per-harness charts. The page covers both Strands runs on the v2 dataset: claude-opus-5-5 on Amazon Bedrock (81.21, 21/21) and minicpm5-2b on vLLM (45.28, 21/21). A hand-written section notes that the two rows used different runner versions and records the two hello-world sanity runs. The README now lists strands among the supported harnesses and links its setup and results pages, and the results index links the new page. gen_agent_report.py gains a label and a setup-doc link for strands.
model_to_slug only stripped two-letter region prefixes (us., eu.), so the Sonnet 5.5 run from #203, which used the global. cross-region profile, landed in swe-benchmark-data/global.anthropic.claude-sonnet-5-5/. The prefix pattern now also matches apac., us-gov. and global.; a replay of every stored model id shows that folder is the only one whose slug changes. Move that folder to claude-sonnet-5-5/ with git mv and set its model_slug fields to match. Scores, metrics and the raw model ids are unchanged. Regenerate the Strands results page and charts, which now cover three models: claude-opus-5-5 (81.21), claude-sonnet-5-5 (76.73) and minicpm5-2b (45.28). Closes #205
Regenerate the Strands page and charts after merging #206, which adds claude-haiku-4-5 under strands/swe3 on the v2 dataset (51.20 over 20 of 21 tasks). The notes record its one failure, a RecursionError after 973 turns on honor-cloud-provider-override-in-ui.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds the Strands harness results page, lists Strands in the README, and fixes the folder-name rule that misfiled the Sonnet 5.5 Strands run.
PR #204 merged before its last commit landed, so
docs/harness-strands-swe3.mdand the README changes never reachedmain. This PR carries that commit and brings the page up to date with the Sonnet 5.5 run from #203 and the Haiku 4.5 run from #206, merged in frommain.Strands results page
docs/harness-strands-swe3.mdis generated withgen_agent_report.py --harness strands --skill swe3 --repo mcp-gateway-registry-v2, with its cost-quality, radar and cost-accuracy charts. It covers the four Strands runs on the v2 dataset:global.profile (#203)A hand-written section below the generated table records which runner version each row used, and Haiku 4.5's one failure:
honor-cloud-provider-override-in-uicrashed withRecursionErrorafter 973 turns. It also records the twohello-worldsanity runs, which stay out of the table because they use a different dataset.The README now lists
strandsamong the supported harnesses and links its setup and results pages. The results index links the new page, andgen_agent_report.pygains a label and setup-doc link forstrands.Folder-name fix (#205)
model_to_slugstripped only two-letter region prefixes (us.,eu.). The Sonnet 5.5 run usedglobal.anthropic.claude-sonnet-5-5, so its results landed inswe-benchmark-data/global.anthropic.claude-sonnet-5-5/and the charts labelled it with the full profile id.apac.,us-gov.andglobal.. A replay of all 48 stored model ids and folder names through the old and new rule shows this folder is the only one whose slug changes.git mvmoves the folder toclaude-sonnet-5-5/. The 22model_slugfields in its files now readclaude-sonnet-5-5. Scores, metrics and the rawmodelids are unchanged.Testing
uv run python -m unittest discover -s testspasses inbenchmarks/, with new tests for each longer prefix.security-checkgate passed.preflight_check.py --model global.anthropic.claude-sonnet-5-5now targetsclaude-sonnet-5-5/.