Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion benchmarks/scripts/strands_agent_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,16 @@
)
EXIT_OK = 0
EXIT_AGENT_ERROR = 1
# Strands' event loop calls itself after every round of tool calls, adding 3
# frames per turn, so Python's default limit of 1,000 frames ends a task after
# about 320 turns (issue #208). Size the limit for MAX_TOOL_TURNS, plus headroom
# for the frames below the loop; a task that runs past it still stops with a
# RecursionError. Our longest finished task took 222 turns; one looping
# attempt reached 316, just under the old ceiling.
MAX_TOOL_TURNS = 1000
FRAMES_PER_TOOL_TURN = 3
RECURSION_HEADROOM = 500
RECURSION_LIMIT = MAX_TOOL_TURNS * FRAMES_PER_TOOL_TURN + RECURSION_HEADROOM

SYSTEM_PROMPT = (
"You are a software engineering agent running non-interactively: no human "
Expand Down Expand Up @@ -435,9 +445,17 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
return args


def _raise_recursion_limit(limit: int = RECURSION_LIMIT) -> None:
"""Raise Python's recursion limit to ``limit``, never lowering it."""
if sys.getrecursionlimit() < limit:
sys.setrecursionlimit(limit)


def main() -> None:
"""Parse arguments and run the task."""
sys.exit(_run(_parse_args()))
args = _parse_args()
_raise_recursion_limit()
sys.exit(_run(args))


if __name__ == "__main__":
Expand Down
24 changes: 24 additions & 0 deletions benchmarks/tests/test_strands_agent_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,30 @@ def test_request_without_tools_omits_tools_key(self) -> None:
self.assertNotIn("tools", model.format_request([]))


class RaiseRecursionLimitTest(unittest.TestCase):
"""Strands recurses per tool turn, so the runner raises the limit (#208)."""

def setUp(self) -> None:
self._original = sys.getrecursionlimit()

def tearDown(self) -> None:
sys.setrecursionlimit(self._original)

def test_low_limit_is_raised(self) -> None:
sys.setrecursionlimit(1000)
runner._raise_recursion_limit(5000)
self.assertEqual(sys.getrecursionlimit(), 5000)

def test_default_limit_covers_the_turn_budget(self) -> None:
needed = runner.MAX_TOOL_TURNS * runner.FRAMES_PER_TOOL_TURN
self.assertGreater(runner.RECURSION_LIMIT, needed)

def test_higher_limit_is_never_lowered(self) -> None:
sys.setrecursionlimit(20000)
runner._raise_recursion_limit(5000)
self.assertEqual(sys.getrecursionlimit(), 20000)


class SystemPromptToolMapTest(unittest.TestCase):
def test_every_tool_the_skill_names_is_mapped(self) -> None:
# swe3/SKILL.md names these Claude Code tools (issue #202).
Expand Down
2 changes: 1 addition & 1 deletion docs/harness-strands-swe3.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Opus 5.5 scored 81.21 under Strands and 81.16 under omp on the same 21 tasks, an

MiniCPM5-2B did. Before #204, Strands lost task attempts on this model to rejected summaries, single large tool results, skipped skill loads and Claude Code tool names. The [model guide](../self-hosted/vllm/models/minicpm5-2b.md#measured-results-strands-swe3-mcp-gateway-registry-v2) compares the run with omp's on the same model: 45.28 over 21 tasks against 42.59 over 19.

Haiku 4.5 scored 51.20 over 20 of 21 tasks under Strands, against 56.18 over 21 under omp, at $0.70 a task against $0.76. Its one failure, `honor-cloud-provider-override-in-ui`, ran 973 turns and then crashed with `RecursionError: maximum recursion depth exceeded`, close to Python's default limit of 1,000 frames, with no artifacts written (`benchmarks/swe-benchmark-data/claude-haiku-4-5/strands/swe3/mcp-gateway-registry-v2/honor-cloud-provider-override-in-ui/metrics.json`). None of the changes in #204 address this.
Haiku 4.5 scored 51.20 over 20 of 21 tasks under Strands, against 56.18 over 21 under omp, at $0.70 a task against $0.76. Its one failure, `honor-cloud-provider-override-in-ui`, ran 973 turns and then crashed with `RecursionError: maximum recursion depth exceeded`, close to Python's default limit of 1,000 frames, with no artifacts written (`benchmarks/swe-benchmark-data/claude-haiku-4-5/strands/swe3/mcp-gateway-registry-v2/honor-cloud-provider-override-in-ui/metrics.json`). None of the changes in #204 address this. [#208](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/208) raises the recursion limit so the runner allows up to 1,000 tool turns.

Sonnet 5.5 has no omp run to compare against. Its results first landed in a folder named after the full profile id, `global.anthropic.claude-sonnet-5-5`, because the folder-name rule did not strip the `global.` prefix; [#205](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/205) fixed the rule and moved the folder to `claude-sonnet-5-5`.

Expand Down
3 changes: 2 additions & 1 deletion docs/strands-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Strands reports no dollar cost. The harness prices the tokens with [bedrock_pric

## Where the runner departs from Strands as it ships

The runner changes Strands in five places. Without the first three, Strands could not finish long tasks on a vLLM endpoint. The last two make the Strands score comparable with omp's. All five came from the `minicpm5-2b` run on vLLM ([#196](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/196)).
The runner changes Strands in six places. Without the first three, Strands could not finish long tasks on a vLLM endpoint. The next two make the Strands score comparable with omp's. The first five came from the `minicpm5-2b` run on vLLM ([#196](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/196)); the sixth came from a Claude Haiku 4.5 task that crashed with `RecursionError`.

| Change | Why | Remove it when |
|---|---|---|
Expand All @@ -70,6 +70,7 @@ The runner changes Strands in five places. Without the first three, Strands coul
| An `AfterToolCallEvent` hook cuts any tool result over `MAX_TOOL_RESULT_CHARS` (100,000 characters, about 25K tokens) and tells the model to read a smaller part ([#199](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/199)) | `file_editor view` returns whole files up to 1 MB. Two views of 280 KB and 220 KB files overflowed a 128K window in one turn, and the summarizer cannot help while the conversation has 10 or fewer messages | Strands caps tool output itself, or [strands-agents/harness-sdk#3723](https://github.com/strands-agents/harness-sdk/pull/3723) ships and makes the overflow recoverable |
| The runner puts the full `SKILL.md` ahead of the task prompt, worded as omp gets it, and does not load the `AgentSkills` plugin ([#201](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/201)) | With the plugin, the model sees only the skill's name and has to call the `skills` tool to read it. `minicpm5-2b` skipped that call on 5 of 12 attempts, and three of those wrote the wrong files or none. Text in the first message also survives summarization, and a tool result does not | Never, while the benchmark compares Strands with omp and kiro, which get the skill the same way |
| The system prompt maps the tool names the skill uses (`Read`, `Edit`, `Write`, `Bash`, `Grep`, `Glob`, `Task`) to the two tools the agent has, `file_editor` and `shell` ([#202](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/202)) | `swe3/SKILL.md` was written for Claude Code and names its tools 29 times. vLLM drops a call to a tool that is not in the request, so the call reaches Strands as text and the attempt ends. With the skill in the first message, 4 of the first 6 attempts ended this way | Never, while the skill names Claude Code tools |
| `main()` raises Python's recursion limit to `RECURSION_LIMIT`, 3,500 frames, sized for `MAX_TOOL_TURNS` = 1,000 tool turns ([#208](https://github.com/aarora79/agentic-coding-harness-benchmarks/issues/208)) | Strands' event loop calls itself after every round of tool calls and adds 3 frames per turn, so the default limit of 1,000 frames ends a task after about 320 turns. A task that runs past 1,000 turns still stops with `RecursionError` | Strands replaces the recursion with a loop |

The cap changes what a model can do in a turn. A model that reads a 280 KB file sees the first 100,000 characters and a note, where unmodified Strands would have ended the task. Claude Code's `Read` tool stops at about the same size, so the cap brings Strands closer to the other harnesses. Each `tool_call` event in `strands-stream.jsonl` records `truncated_from_chars`, so a reader can count how often the cap fired.

Expand Down
Loading