Skip to content

Add the Strands results page and fix global. model slugs - #207

Merged
aarora79 merged 4 commits into
mainfrom
docs/strands-results-page
Oct 5, 2026
Merged

aarora79 merged 4 commits into
mainfrom
docs/strands-results-page

Conversation

@aarora79

@aarora79 aarora79 commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR adds the Strands harness results page, lists Strands in the README, and fixes the folder-name rule that misfiled the Sonnet 5.5 Strands run.

PR #204 merged before its last commit landed, so docs/harness-strands-swe3.md and the README changes never reached main. This PR carries that commit and brings the page up to date with the Sonnet 5.5 run from #203 and the Haiku 4.5 run from #206, merged in from main.

Strands results page

docs/harness-strands-swe3.md is generated with gen_agent_report.py --harness strands --skill swe3 --repo mcp-gateway-registry-v2, with its cost-quality, radar and cost-accuracy charts. It covers the four Strands runs on the v2 dataset:

Model Score Completed Cost per task Path
claude-opus-5-5 81.21 21/21 $2.46 (metered) Amazon Bedrock (#195)
claude-sonnet-5-5 76.73 21/21 $0.96 (metered) Amazon Bedrock, global. profile (#203)
claude-haiku-4-5 51.20 20/21 $0.70 (metered) Amazon Bedrock (#206)
minicpm5-2b 45.28 21/21 $0.33 (hardware-derived) vLLM on g6e.4xlarge (#204)

A hand-written section below the generated table records which runner version each row used, and Haiku 4.5's one failure: honor-cloud-provider-override-in-ui crashed with RecursionError after 973 turns. It also records the two hello-world sanity runs, which stay out of the table because they use a different dataset.

The README now lists strands among the supported harnesses and links its setup and results pages. The results index links the new page, and gen_agent_report.py gains a label and setup-doc link for strands.

Folder-name fix (#205)

model_to_slug stripped only two-letter region prefixes (us., eu.). The Sonnet 5.5 run used global.anthropic.claude-sonnet-5-5, so its results landed in swe-benchmark-data/global.anthropic.claude-sonnet-5-5/ and the charts labelled it with the full profile id.

  • The prefix pattern now also matches apac., us-gov. and global.. A replay of all 48 stored model ids and folder names through the old and new rule shows this folder is the only one whose slug changes.
  • git mv moves the folder to claude-sonnet-5-5/. The 22 model_slug fields in its files now read claude-sonnet-5-5. Scores, metrics and the raw model ids are unchanged.

Testing

  • uv run python -m unittest discover -s tests passes in benchmarks/, with new tests for each longer prefix.
  • Ruff, mypy and Bandit are clean, and the security-check gate passed.
  • preflight_check.py --model global.anthropic.claude-sonnet-5-5 now targets claude-sonnet-5-5/.

Generate docs/harness-strands-swe3.md with gen_agent_report.py and the three
per-harness charts. The page covers both Strands runs on the v2 dataset:
claude-opus-5-5 on Amazon Bedrock (81.21, 21/21) and minicpm5-2b on vLLM
(45.28, 21/21). A hand-written section notes that the two rows used different
runner versions and records the two hello-world sanity runs.

The README now lists strands among the supported harnesses and links its
setup and results pages, and the results index links the new page.
gen_agent_report.py gains a label and a setup-doc link for strands.
model_to_slug only stripped two-letter region prefixes (us., eu.), so the
Sonnet 5.5 run from #203, which used the global. cross-region profile, landed
in swe-benchmark-data/global.anthropic.claude-sonnet-5-5/. The prefix pattern
now also matches apac., us-gov. and global.; a replay of every stored model id
shows that folder is the only one whose slug changes.

Move that folder to claude-sonnet-5-5/ with git mv and set its model_slug
fields to match. Scores, metrics and the raw model ids are unchanged.

Regenerate the Strands results page and charts, which now cover three models:
claude-opus-5-5 (81.21), claude-sonnet-5-5 (76.73) and minicpm5-2b (45.28).

Closes #205
Regenerate the Strands page and charts after merging #206, which adds
claude-haiku-4-5 under strands/swe3 on the v2 dataset (51.20 over 20 of 21
tasks). The notes record its one failure, a RecursionError after 973 turns on
honor-cloud-provider-override-in-ui.
@aarora79
aarora79 merged commit 82b1ada into main Oct 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant