Repository navigation
Benchmark minicpm5-2b under the Strands harness and fix Strands on vLLM - #204
Merged
Merged
Conversation
Run the 21-task mcp-gateway-registry-v2 dataset with minicpm5-2b on vLLM under --agent strands with /swe3. All 21 tasks produce six artifacts and the judge mean is 45.28, against 42.59 over 19 tasks for omp on the same model. Strands as it ships could not finish long tasks on a vLLM endpoint. The runner now: - passes the served window minus max_tokens as context_window_limit, so proactive summarization runs before an overflow (#197) - drops an empty tools array from OpenAIModel requests, which vLLM rejects with HTTP 400 and which broke every summary (#198) - truncates tool results over 100,000 characters, so one large file view cannot end the task (#199) - puts SKILL.md in the first message, as omp and kiro get it, in place of the on-demand AgentSkills plugin (#201) - maps the skill's Claude Code tool names to file_editor and shell in the system prompt (#202) The harness leaves Strands cache keys out when the endpoint reports none, so cache reads fall back to vLLM's Prometheus counter as they do for omp, and keeps a disjoint agent's token fields disjoint when it does (#200). The pre-flight now treats any non-empty task folder as existing, so a failed attempt's metrics.json and stream file no longer leak into the next run, and refuses any target folder that resolves outside swe-benchmark-data. Closes #196, #197, #198, #199, #200, #201, #202
This was referenced Oct 5, 2026
Closed
Closed
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a benchmark of
minicpm5-2bunder the Strands harness (--agent strands,/swe3) on the 21-taskmcp-gateway-registry-v2dataset, served by vLLM at a 131,072-token window on one L40S. It also changes the Strands runner in five places. Without the first three, Strands could not finish long tasks on a vLLM endpoint. The last two make its score comparable with omp's.Result
All 21 tasks produced six artifacts. The judge mean is 45.28 (median 44.0, range 20.0 to 63.6). omp on the same model averaged 42.59 over 19 tasks, with 2 failures. Strands used 160.6M tokens against omp's 222.3M.
Source:
benchmarks/swe-benchmark-data/minicpm5-2b/strands/swe3/mcp-gateway-registry-v2/run-summary.json. The model guide (self-hosted/vllm/models/minicpm5-2b.md) records the result.Changes
Strands runner (
benchmarks/scripts/strands_agent_runner.py). Only--agent strandsruns this file.context_window - max_tokensascontext_window_limiton the endpoint pathmax_tokensagainst the window. Run 3 hit 160 overflows in 925 callsOpenAIModelsubclass drops"tools": []AfterToolCallEventhook truncates tool results over 100,000 charactersSKILL.mdin the first message, as omp and kiro get it, in place of theAgentSkillspluginRead,Edit,Write,Bash, ...) tofile_editorandshellin the system promptHarness (
benchmarks/scripts/run-swe-headless.py, #200). The Strands parser leaves cache keys out when the endpoint reported none, so cache reads fall back to vLLM's Prometheus counter, as they already do for omp. When that happens for a disjoint agent,input_tokensbecomes the fresh prompt tokens, so totals count the prompt once. Replaying all 840 storedmetrics.jsonfiles through the old and new code changed no record for claude, codex, kiro, omp or pi, on either the Amazon Bedrock or the vLLM path.Pre-flight (
benchmarks/scripts/preflight_check.py). Any non-empty task folder now counts as existing, so--clearremoves a failed attempt'smetrics.jsonand stream file before the next run appends to them. Every target folder must resolve insideswe-benchmark-data/. A task id or--modelcontaining..now exits 1 instead of pointing the delete elsewhere. Behavior change: a model with failed-task folders from an earlier run now stops at the pre-flight unless you pass--yes.Docs.
docs/strands-setup.mdlists the five departures from Strands as it ships, with when to remove each.Testing
uv run python -m unittest discover -s testspasses inbenchmarks/andself-hosted/vllm/. The new tests cover each change, including one that checks omp's token shape does not move.security-checkgate passed after adding the path guard.hello-worldwith Strands on vLLM after each change. The last scored 6/6 with onlyshellandfile_editorcalls and recorded 1,028,896 cache-read tokens.Strands results page
docs/harness-strands-swe3.mdis the per-harness results page for Strands, generated withgen_agent_report.py --harness strands --skill swe3 --repo mcp-gateway-registry-v2, with its cost-quality, radar and cost-accuracy charts. It holds both Strands runs on the v2 dataset:claude-opus-5-5on Amazon Bedrock (81.21, 21/21, from #195) andminicpm5-2bon vLLM (45.28, 21/21, this PR). A hand-written section below the generated table notes that the Opus 5.5 run used the runner from before this PR. The README now listsstrandsamong the supported harnesses, and the README and the results index link the new page.