Skip to content

Benchmark minicpm5-2b under the Strands harness and fix Strands on vLLM - #204

Merged
aarora79 merged 1 commit into
mainfrom
bench/minicpm5-2b-strands-swe3
Oct 5, 2026
Merged

aarora79 merged 1 commit into
mainfrom
bench/minicpm5-2b-strands-swe3

Conversation

@aarora79

@aarora79 aarora79 commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR adds a benchmark of minicpm5-2b under the Strands harness (--agent strands, /swe3) on the 21-task mcp-gateway-registry-v2 dataset, served by vLLM at a 131,072-token window on one L40S. It also changes the Strands runner in five places. Without the first three, Strands could not finish long tasks on a vLLM endpoint. The last two make its score comparable with omp's.

Result

All 21 tasks produced six artifacts. The judge mean is 45.28 (median 44.0, range 20.0 to 63.6). omp on the same model averaged 42.59 over 19 tasks, with 2 failures. Strands used 160.6M tokens against omp's 222.3M.

Tier n strands omp
trivial 5 48.0 43.8
low 5 50.1 51.0
medium 6 46.8 40.4 (n=5)
high 5 35.8 33.5 (n=4)

Source: benchmarks/swe-benchmark-data/minicpm5-2b/strands/swe3/mcp-gateway-registry-v2/run-summary.json. The model guide (self-hosted/vllm/models/minicpm5-2b.md) records the result.

Changes

Strands runner (benchmarks/scripts/strands_agent_runner.py). Only --agent strands runs this file.

Issue Change What it fixed
#197 Pass context_window - max_tokens as context_window_limit on the endpoint path Strands assumed a 200K window, and vLLM counts max_tokens against the window. Run 3 hit 160 overflows in 925 calls
#198 OpenAIModel subclass drops "tools": [] vLLM rejects an empty tools array with HTTP 400, so every summary failed. Upstream: strands-agents/harness-sdk#4854
#199 AfterToolCallEvent hook truncates tool results over 100,000 characters One view of two 280 KB and 220 KB files overflowed the window, and the summarizer cannot recover with 10 or fewer messages. Upstream: strands-agents/harness-sdk#981, #3723
#201 Put SKILL.md in the first message, as omp and kiro get it, in place of the AgentSkills plugin The model skipped loading the skill on 5 of 12 attempts
#202 Map the skill's Claude Code tool names (Read, Edit, Write, Bash, ...) to file_editor and shell in the system prompt vLLM drops calls to unknown tools, and 4 of 6 attempts ended that way

Harness (benchmarks/scripts/run-swe-headless.py, #200). The Strands parser leaves cache keys out when the endpoint reported none, so cache reads fall back to vLLM's Prometheus counter, as they already do for omp. When that happens for a disjoint agent, input_tokens becomes the fresh prompt tokens, so totals count the prompt once. Replaying all 840 stored metrics.json files through the old and new code changed no record for claude, codex, kiro, omp or pi, on either the Amazon Bedrock or the vLLM path.

Pre-flight (benchmarks/scripts/preflight_check.py). Any non-empty task folder now counts as existing, so --clear removes a failed attempt's metrics.json and stream file before the next run appends to them. Every target folder must resolve inside swe-benchmark-data/. A task id or --model containing .. now exits 1 instead of pointing the delete elsewhere. Behavior change: a model with failed-task folders from an earlier run now stops at the pre-flight unless you pass --yes.

Docs. docs/strands-setup.md lists the five departures from Strands as it ships, with when to remove each.

Testing

  • uv run python -m unittest discover -s tests passes in benchmarks/ and self-hosted/vllm/. The new tests cover each change, including one that checks omp's token shape does not move.
  • Ruff, mypy and Bandit are clean, and the security-check gate passed after adding the path guard.
  • Smoke tests on hello-world with Strands on vLLM after each change. The last scored 6/6 with only shell and file_editor calls and recorded 1,028,896 cache-read tokens.
  • The full run in this PR used the final code.

Strands results page

docs/harness-strands-swe3.md is the per-harness results page for Strands, generated with gen_agent_report.py --harness strands --skill swe3 --repo mcp-gateway-registry-v2, with its cost-quality, radar and cost-accuracy charts. It holds both Strands runs on the v2 dataset: claude-opus-5-5 on Amazon Bedrock (81.21, 21/21, from #195) and minicpm5-2b on vLLM (45.28, 21/21, this PR). A hand-written section below the generated table notes that the Opus 5.5 run used the runner from before this PR. The README now lists strands among the supported harnesses, and the README and the results index link the new page.

Run the 21-task mcp-gateway-registry-v2 dataset with minicpm5-2b on vLLM
under --agent strands with /swe3. All 21 tasks produce six artifacts and the
judge mean is 45.28, against 42.59 over 19 tasks for omp on the same model.

Strands as it ships could not finish long tasks on a vLLM endpoint. The
runner now:

- passes the served window minus max_tokens as context_window_limit, so
  proactive summarization runs before an overflow (#197)
- drops an empty tools array from OpenAIModel requests, which vLLM rejects
  with HTTP 400 and which broke every summary (#198)
- truncates tool results over 100,000 characters, so one large file view
  cannot end the task (#199)
- puts SKILL.md in the first message, as omp and kiro get it, in place of
  the on-demand AgentSkills plugin (#201)
- maps the skill's Claude Code tool names to file_editor and shell in the
  system prompt (#202)

The harness leaves Strands cache keys out when the endpoint reports none, so
cache reads fall back to vLLM's Prometheus counter as they do for omp, and
keeps a disjoint agent's token fields disjoint when it does (#200).

The pre-flight now treats any non-empty task folder as existing, so a failed
attempt's metrics.json and stream file no longer leak into the next run, and
refuses any target folder that resolves outside swe-benchmark-data.

Closes #196, #197, #198, #199, #200, #201, #202
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant