Skip to content

Add a loop guard to the Strands runner - #213

Merged
aarora79 merged 1 commit into
mainfrom
feat/strands-loop-guard
Oct 7, 2026
Merged

aarora79 merged 1 commit into
mainfrom
feat/strands-loop-guard

Conversation

@aarora79

@aarora79 aarora79 commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Summary

This PR adds a loop guard to the Strands runner. It covers --agent strands and --agent strands-harness. It builds on #212, whose commit is included here; merge #212 first.

Why

A small model can repeat one tool call hundreds of times. In the 21-task --agent strands-harness run of minicpm5-2b, two tasks looped:

Task Repeated call Repeats Outcome
lifecycle-workflow-webhooks git diff HEAD --stat 674 844 turns, 53M tokens, 6/6 artifacts
logout-id-token-hint-out-of-browser-url the same grep 542 1,148 turns, 72M tokens, crashed at the recursion limit with 0 artifacts

Those two tasks used 125M of the run's 209M tokens.

How it works

_LoopGuard is a Strands hook provider that counts every tool call by tool name and exact input over the whole attempt. It counts totals, not back-to-back repeats, because the loops cycled through several commands.

  • Past LOOP_WARN_REPEATS (20), BeforeToolCallEvent.cancel_tool cancels the repeated call. The model gets back a message telling it the result will not change and to finish or try something else.
  • At LOOP_STOP_REPEATS (40), AfterToolsEvent.end_turn ends the attempt like a normal finish. The harness then scores the artifacts already written and does not retry.
  • Each action is a loop_guard event in strands-stream.jsonl, and the result event records loop_guard_stopped.

Thresholds

Across 42 task attempts in both Strands runs of minicpm5-2b, no healthy task repeated an identical call more than 16 times. The three loops reached 29, 542 and 674. Warning at 20 and stopping at 40 leaves every measured healthy task untouched.

Testing

  • Unit tests: calls up to the warning count run; a call past it is cancelled; different inputs count separately; the stop count ends the turn; no stop leaves the turn open; the warning threshold clears the measured healthy maximum.
  • A live check through the real Strands event loop, with a fake model that repeats one call forever: the tool ran 20 times; calls 21 to 39 were cancelled with the warning; at call 40 the agent stopped with stop_reason=end_turn and the text "Stopped by the loop guard: the same tool call was repeated 40 times."
  • A hello-world smoke run under --agent strands-harness finished 6/6 in 20 turns, and the guard did not fire.
  • uv run python -m unittest discover -s tests passes in benchmarks/, apart from the known intermittent failure in test_run_throughput_harness (Flaky test: test_untracked_file_written_during_the_level_is_quarantined fails about 1 in 15 runs #211). Ruff and Bandit are clean, and the security-check gate passed.

A small model can repeat one tool call hundreds of times. Under
--agent strands-harness, minicpm5-2b ran `git diff HEAD --stat` 674 times in
one task and a `grep` 542 times in another; the second hit the recursion limit
after 1,148 turns with no artifacts. Those two tasks used 125M of the run's
209M tokens.

_LoopGuard counts every tool call by tool name and exact input over the whole
attempt, since a loop can cycle through a few commands. Past
LOOP_WARN_REPEATS (20) it cancels the repeated call through
BeforeToolCallEvent.cancel_tool and tells the model to move on. At
LOOP_STOP_REPEATS (40) it sets AfterToolsEvent.end_turn, which ends the attempt
like a normal finish, so the harness scores the artifacts already written and
does not retry. Each action is a loop_guard event in the stream, and the result
event records loop_guard_stopped.

The thresholds come from 42 task attempts across both Strands runs of
minicpm5-2b: no healthy task repeated an identical call more than 16 times; the
three loops reached 29, 542 and 674. The guard covers --agent strands and
--agent strands-harness.
@aarora79
aarora79 force-pushed the feat/strands-loop-guard branch from b56a629 to 70ad507 Compare October 7, 2026 14:05
@aarora79
aarora79 merged commit 616e6da into main Oct 7, 2026
5 checks passed
@aarora79
aarora79 deleted the feat/strands-loop-guard branch October 7, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant