Repository navigation
fix(multiagent): clear pending interrupts when a graph or swarm invocation fails - #5633
Merged
mkmeral merged 2 commits intoOct 28, 2026
Merged
Conversation
…ation fails A node failing alongside an interrupted sibling (or while resuming one) left _interrupt_state activated with no completed_nodes recorded, so every later call raised TypeError or KeyError. A FAILED invocation now drops its pending interrupts, including those of interrupted node executors, so the next call starts fresh. Fixes strands-agents#5340
_execute_graph empties state.interrupted_nodes at the start of a resume, so a failure during a resume left the node agent activated. Iterate every node executor instead, matching Swarm.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Contributor
|
Assessment: Comment (approve-leaning) Clean, well-scoped fix. The private Review notes
Nicely documented reasoning throughout the PR. |
mkmeral
approved these changes
Oct 28, 2026
opieter-aws
approved these changes
Oct 28, 2026
gautamsirdeshmukh
pushed a commit
to gautamsirdeshmukh/harness-sdk
that referenced
this pull request
Nov 2, 2026
…ation fails (strands-agents#5633) Co-authored-by: strandly-the-agent <strandly-the-agent@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by: @mkmeral — investigate #5340 and fix it with the fail-fast approach (drop pending interrupts when an invocation fails), kept private, Swarm included.
Human Overview
When parallel node executions in graph fail and interrupt at the same batch. the interrupt is keeping the graph from working. Continuing is meaningless because the graph already failed, so it forces unnecessary back and forth.
This PR adds a private method to clear interrupts. The agents can recover from dangling tool use, so everything continues properly in the next invocation.
Description
When one node in a parallel
Graphbatch interrupts and a sibling in the same batch raises,_activate_interrupthas already set_interrupt_state.activated, but the exception skips the batch-exit bookkeeping in_execute_graph(writingcontext["completed_nodes"]/deactivate()). The graph is then locked for good: a new task raisesTypeError … must resume from interrupt, the interrupt responses raiseKeyError: 'completed_nodes', and a session reload carries the broken state forward.Swarmhas the same lockout when a node fails while resuming an interrupt (it returnsFAILEDwithout raising, so the top-level handler never ran either).A FAILED invocation now drops its pending interrupts so the next call starts fresh:
MultiAgentBase._clear_interrupts()(private) deactivates the orchestrator's_interrupt_state;Graphoverrides it to also clear every node executor (nestedMultiAgentBaserecursively, anything else via_interrupt_state.deactivate()),Swarmto clear every node agent. It runs instream_async'sfinallywhen the status is FAILED, beforeAfterMultiAgentInvocationEvent, so session managers persist the cleared state. Interrupted agents' messages are left alone — the agent already heals a danglingtoolUseon its next invocation.Deliberately out of scope: a public "abandon this interrupt" API (
Agenthas the same gap; needs its own API review — happy to file the follow-up), turning the fail-fast raise into aFAILEDresult, andasyncio.CancelledErrorescaping a batch (#4419).Related Issues
Fixes #5340
Documentation PR
N/A — no public API change.
Type of Change
Bug fix
Testing
Five regression tests (
tests/strands/multiagent/test_graph.py,test_swarm.py), each fails onmainwithout the src change: hook interrupt + failing sibling, agent-level interrupt + failing sibling, failure on the resume path (node agent's own_interrupt_statemust be cleared), nested-MultiAgentBaserecursion, and Swarm interrupt → failed resume → new task. The parallel-batch tests order the failure after the interrupt with anasyncio.Event, not a sleep.Offline repro scripts (stub
Model, no network) for the four scenarios — issue repro, tool-level interrupt landing first, nestedGraphnode, Swarm — all recover on this branch and lock out onmain.hatch run preparenot run as such (no hatch in the sandbox); the equivalent ruff/mypy/pytest steps above were run from the project venv.Additional Details
Independent review ledger (two rounds + adversarial pass)
Round 1 — 🔴 fixed in
f7ce961.Graph._clear_interruptsiteratedstate.interrupted_nodes, which_execute_graphempties at the start of a resume, so a failure during a resume (exception, ormax_node_executionsexhausted →should_continuereturns FAILED without raising) left the node agent activated and the next string task still raisedTypeError. Now iteratesself.nodes.values()(Swarm's shape). New testtest_graph_node_failure_on_resume_clears_interrupted_agent_statefails on the first commit, passes on the second.Round 2 — approved, no new findings. Verified with a real
Agent+ stub model that the no-raise resume-limit path is cleared; confirmed iterating all nodes doesn't over-clear (nodes are fixed afterbuild(), every node task is gathered before thefinally, agents already drop answered invocation-scoped responses at the end of their own invocation).⚪ nits — accepted:
asyncio.sleepordering in tests →asyncio.Event; one-line comment on why the clear sits beforeAfterMultiAgentInvocationEvent;#5340docstring on the nested test.⚪ nits — not taken, with reasons:
_interrupt_stateonMultiAgentBaseand call it directly instead ofgetattr. Keptgetattr:Graphrecurses into nestedMultiAgentBaseexecutors, which can be third-party subclasses without that attribute. AnAttributeErrorthere would run inside afinallyand mask the original exception, which is worse than skipping the clear.state.interrupted_nodes/ nodeexecution_statuson clear. Not needed: nothing reads them whileactivatedis False, and the next invocation builds a freshGraphState. Matches what a plain failure leaves behind today.strands-tsvery likely has the same lockout (its status resolution ranks FAILED over INTERRUPTED in a mixed batch). Flagging rather than bundling — a[PORT]follow-up is the right vehicle.Adversarial pass (a first run timed out at 15 min and was re-run with a 10-min budget; it completed the high-priority probes). Probed and fine: original repro, tool-level interrupt + sibling raise, nested
Graph, Swarm,max_node_executionsexhausted on first resume iteration, interrupt responses submitted after a clear (loudValueError, no lockout), legitimate interrupt with no failure still resumes normally and is never cleared (status is INTERRUPTED, not FAILED). Pre-existing, out of scope: a consumer thatbreaks out ofstream_asyncright after the interrupt event and then resumes hitsKeyError: 'completed_nodes'— identical onmain, same family as this bug (completed_nodesis only written on the clean interruptreturn), worth its own issue. Not probed for lack of time: nestedSwarmas a Graph node,cancel_nodefrom a hook on resume,node_timeouton an interrupted node's sibling,reset_on_revisitcycles, andFileSessionManagerrestore into a fresh builder after the failure (the before-the-hook placement is what should make that work; verified by code reading only).Checklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.