[opieter-aws]

Description

Fixes handoff for crashed nodes in a swarm.

When a Swarm uses a session manager, it checkpoints after every node so a crashed run can resume from next_nodes_to_execute. The checkpoint fires before the execution loop applies a handoff transition, so the saved state was ambiguous: EXECUTING with a pending handoff looks identical whether the node committed that handoff or merely inherited it from an earlier turn. Both sides of the contract tried to recover, and every inference had a case it got wrong: a finished node re-ran, a failed run resumed as a success, a cancelled node's speculative handoff was committed, and a pending self-handoff was dropped on the second restart.

This replaces inference with a recorded outcome. Each node turn carries a _InflightTurn whose outcome the loop sets where the answer is actually known. A persisted handoff always means its target still owes a turn. The writer settles that at save time. A handoff the resume frontier already carries is not recorded, and a committed turn with no handoff persists an empty frontier. This makes restore honor the payload literally and the reader-side equality check is deleted rather than replaced.

Two consequences worth noting:

1. Consumers observing multiagent_node_stop now see node_history already containing that node, because a completed turn commits before its terminal event is forwarded. A teardown at that yield must not roll back work already recorded.
2. A checkpoint written before this change replays a turn at worst, it won't skip one.

Callouts

• A no-handoff completion replays the whole task rather than one node. An empty frontier resets for a fresh run, so a crash at the final node re-runs the task from the entry point instead of re-running that node. The alternative (record the node as done only after the loop) preserves partial progress but introduces some complexity where a stream closed on a succeeded node's terminal event rolls back its handoff and shared context. Both solutions are at-least-once and neither skips work.

• The root cause of this callout is that Python lets a node hand off to itself while the TS doesn't support this. That divergence is what makes a frontier/target equality ambiguous, and closing it would make the ambiguity unrepresentable rather than merely handled. Changing that behaviour is a bigger change that is breaking for sessions where nodes have done a self-handoff.

Related Issues

Needed for multi-agent snapshot session manager

Documentation PR

N/A

Type of Change

Bug fix

Testing

Ran hatch test (full unit suite) plus the multiagent and session suites, ruff check, ruff format --check, and mypy on swarm.py.

• I ran hatch run prepare

Checklist

• I have read the CONTRIBUTING document
• I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
• My change is focused and reasonably small; I have split unrelated work into separate PRs
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[github-actions]

Assessment: Approve

Clean, well-scoped fix for the two crash-restart resume defects. The change correctly mirrors the already-correct Graph contract — prioritizing a pending handoff over the still-running node in the serialized frontier, and deciding reset-vs-resume from the frontier's value rather than the key's presence.

Review notes

• Correctness: The serialize_state branch reorder is right — at the AfterNodeCallEvent checkpoint completion_status is still EXECUTING while handoff_node holds the pending target, so the handoff branch must win. The new _from_dict handoff-clearing relies on ==, which SwarmNode.__eq__ defines by node_id, so it's robust rather than identity-dependent.
• Parity: Value-based next_nodes_to_execute check matches Graph.deserialize_state. Good consistency.
• Testing: Both new tests are genuine regression tests — I confirmed they fail on the pre-fix code and pass after, and the full swarm suite (39) + ruff + mypy are clean. Per-field assertions on the serialized dict are appropriate here since the surrounding fields (node_results, usage) are volatile/irrelevant to the frontier being verified.
• Scope/API: Python-only, no public API surface change — no API bar-raising review needed.
• Minor (non-blocking): One inline suggestion about linking the guarding issue in the regression-test docstrings. Also worth a thought: a swarm that fails while a handoff is pending would still serialize a non-empty frontier and resume — pre-existing behavior, but the new docstring's "failed at a dead end resets" phrasing could read as covering all failures.

Nicely done — the manual end-to-end crash simulation described in the PR body is exactly the kind of verification that builds confidence in a resume fix.

[mkmeral]

@strandly-the-agent review pls

[strandly-the-agent]

🔴 Changes requested — checkpoint restore can still replay/skip work, and one reachable reset path raises KeyError.

✅ Reviewed exact head e13b2bed against merge base 3966e703; no public API change.
✅ Latest completed CI is green across Python 3.10–3.14, lint, docs, and integration.
✅ Independent exact-head adversarial matrix reproduced the findings twice identically; Swarm 39/39, changed-file Ruff, mypy (223 files), and git diff --check passed.
⚠️ The advanced adversarial tier was unavailable under the configured data-retention policy; the retry completed on the default tier.

Review breakdown

Three blockers remain:

1. The pre-transition checkpoint cannot distinguish terminal success, failed handoff, and legacy frontier semantics, causing replay or turning failure into success.
2. Empty-frontier restore installs active interrupt state before resetting every node, which can raise KeyError.
3. Unconditional handoff/frontier equality cleanup drops accepted self-handoffs during crash resume.

The terminal-checkpoint test gap is folded into finding 1. I suppressed the pre-existing second-crash shared-context issue, and did not repeat the existing test-docstring suggestion. Docs were otherwise accurate.

Passes: correctness · adversarial · test quality · docs accuracy

[opieter-aws]

@strandly-the-agent re-review with the latest fixes

[github-actions]

Assessment: Approve (re-review of 0433224 "Address comments")

The three blockers @strandly-the-agent raised were legitimate — my earlier approve missed them — and all three are now correctly resolved and independently verified. I confirmed each fix by reverting it in isolation and watching the matching regression test fail.

Verification of the three blockers

• Terminal/failed-handoff frontier (finding 1): serialize_state now excludes the just-run node via current_node not in node_history and gates the handoff branch on EXECUTING. Terminal checkpoints and handoff-then-failure both persist an empty frontier and reset on restore, so no replay and no failure-masking. The cross-version (old-writer/new-reader) case is documented as a known limitation rather than code-fixed — see the thread; reasonable for a bugfix, but a versioned checkpoint phase would be the robust long-term answer.
• Empty-frontier interrupt reset (finding 2): interrupt state is cleared before reset_executor_state(), eliminating the KeyError.
• Self-handoff preservation (finding 3): handoff-clearing is gated on != INTERRUPTED, correctly separating the "frontier is current node" (INTERRUPTED) shape from the "frontier is handoff target" (EXECUTING) shape.

Testing: 4 new tests, one per scenario; all are genuine guards (each fails when its fix is reverted). Full swarm suite 43/43, ruff clean, mypy clean across 223 files.

Nice turnaround addressing subtle state-machine edge cases with targeted regression coverage.

[strandly-the-agent]

🔴 Changes requested — two exact-head crash-resume paths can still silently skip pending Swarm work.

✅ Re-reviewed exact head 04332244 against prior head e13b2bed and merge base 3966e703; no public API change.
✅ Exact-head CI is fully green; local focused run passed 45/45, changed-file Ruff passed, and git diff --check passed.
✅ The new interrupt reset, terminal checkpoint, ordinary failed handoff, and interrupted self-handoff fixes are resolved.
⚠️ The advanced adversarial tier was unavailable under the data-retention policy; the default-tier retry completed and its two deterministic repros were rerun independently.

Review breakdown

Two blockers remain:

1. A handoff to a previously visited node can be serialized with an empty frontier, so restore starts over instead of running the pending target.
2. Cancellation after requesting a handoff can serialize the target even though the source did not complete, so restore skips unfinished work.

I did not reopen the accepted/documented old-writer compatibility limitation, and I suppressed fresh-run behavior after an empty frontier because that is the intended Graph-compatible contract.

Passes: correctness · adversarial · test quality

[opieter-aws]

@strandly-the-agent re-review

[github-actions]

Assessment: Approve (re-review of c05970e "Resume pending revisit node on cancel")

Both blockers from @strandly-the-agent's second pass are correctly resolved and independently verified — I reverted each fix in isolation and confirmed the matching regression test fails.

Verification of the two blockers

• Pending revisit dropped (finding 1): the current_node not in node_history guard is removed, so serialize_state always records the current node while EXECUTING. The A→B→A revisit now serializes ["second"] and resumes the pending target. The consequence — a terminal node re-running on restore — is accepted as an at-least-once contract and documented in deserialize_state (nodes must be idempotent). Reasonable resolution; the terminal and revisit cases are indistinguishable at save time without a phase field. Test: test_swarm_resume_pending_revisit_node_runs (revert regresses frontier to []).
• Cancellation commits uncompleted handoff (finding 2): a new except asyncio.CancelledError clears handoff_node before re-raising, so the checkpoint keeps the frontier on the uncompleted source. Test uses a real CancelledError as requested: test_swarm_resume_cancelled_handoff_reruns_source_not_target (revert regresses handoff_node to "second").

Testing: 44/44 swarm tests pass; ruff and mypy (223 files) clean. Both fixes verified as genuine guards.

Minor (non-blocking): Codecov reports 80% patch coverage (1 missing + 2 partial lines) — branch coverage on the changed conditionals; the substantive cancel/revisit paths are directly tested, so this is cosmetic.

The at-least-once framing is the right call here — solid iteration on a genuinely tricky checkpoint state machine.

[strandly-the-agent]

🔴 Changes requested — two cancellation paths can still persist inconsistent handoff state.

✅ Re-reviewed exact head c05970ee against prior 04332244; the pending-revisit fix works.
✅ Exact-head local checks: 46 focused tests; specialists also ran Swarm 44/44 and FileSessionManager 40/40; git diff --check passed.
✅ Both remaining real FileSessionManager crash/restore repros were confirmed twice.
⚠️ Exact-head unit/lint/docs/CI Gate are green, but the separate integration job failed; its logs aren't exposed through the API. Advanced adversarial was unavailable under the data-retention policy, so the pass completed on the default-tier retry.

Review breakdown

1. Closing the public stream with aclose() bypasses the CancelledError rollback and checkpoints target B even though source A never completed.
2. Direct cancellation reruns A but persists B-directed handoff content and speculative shared context, which are then injected into A's retry.

These are separate defects, though one atomic rollback helper may address both. I did not repeat resolved findings or the accepted terminal at-least-once/old-writer limitations.

[opieter-aws]

@strandly-the-agent re-review

[github-actions]

Assessment: Approve (re-review of a0f7f28, post-merge with main)

This revision replaces the earlier targeted cancel-clear with a more general snapshot/rollback design, and it's a clear improvement. Verified by reverting each new mechanism in isolation and watching the matching regression test fail.

What changed and how I verified it

• Full pre-node rollback: before each node runs, _uncommitted_handoff_snapshot captures (handoff_node, message, deepcopy(shared_context)). On CancelledError/GeneratorExit/Exception the node's uncommitted handoff and its shared-context mutations are rolled back before the checkpoint; on success/interrupt the snapshot is cleared so a committed handoff stands. This is strictly more complete than the prior handoff_node = None — it also prevents an uncommitted handoff's context from leaking into the persisted state. (Revert inner rollback → test_swarm_resume_cancelled_handoff_reruns_source_not_target fails, frontier regresses to ["second"].)
• Stream-close (aclose()) path: GeneratorExit is now handled, and stream_async's finally rolls back before the invocation-complete checkpoint (ordering documented). This covers a teardown path the previous revision didn't. (Revert the stream_async rollback → test_swarm_resume_closed_stream_after_handoff_reruns_source fails.)
• Shared-context aliasing on resume: _from_dict re-aliases state.shared_context = self.shared_context so the next checkpoint doesn't clobber restored context. (Revert → test_swarm_resume_shared_context_not_clobbered_on_next_checkpoint fails.)
• Robustness: .get() for a removed handoff target, and a missing frontier node falling back to the initial node while dropping stale interrupt state — both newly tested.
• The A→B→A revisit fix and the documented at-least-once terminal semantics from the prior revision are retained.

Testing: 51/51 swarm tests pass (was 44), including a new end-to-end FileSessionManager on-disk crash-restart test. ruff clean, mypy clean (226 files). Codecov 97.29% patch (1 partial branch) — non-blocking.

The snapshot/rollback framing generalizes the fix cleanly across all teardown paths (cancel, stream close, failure) — nice consolidation.

[github-actions]

Assessment: Approve (re-review of 2d28c70)

The latest finding from @strandly-the-agent — a subtle regression introduced by the previous should-fix — is real, and is resolved in the current head. I reproduced the compounding, verified the fix, and reverse-tested it rather than taking the resolution on faith.

Verification of the no-handoff regression

• Root cause: moving node_history.append ahead of the terminal yield meant a committed turn that owes no handoff fell through the handoff_node-gated committed branch and re-serialized current_node — so the checkpoint claimed the node was both done (history=['solo']) and still-owed (frontier=['solo']). Repeated crashes at that point grew node_history one entry each, which should_continue charges against limits until the swarm wedges into FAILED having run nothing.
• Fix: serialize_state's committed branch now persists an empty frontier when there's no handoff (next_nodes = []), instead of re-listing the finished node. A no-handoff completion is now a checkpoint fixed point.
• Reverse-test: collapsing the branch back to the pre-fix form regresses test_swarm_repeated_crash_at_completed_turn_is_a_fixed_point's next_nodes_to_execute from [] to ['solo'].

Test-strength gaps closed

• A two-generation fixed-point test now crashes repeatedly at the same no-handoff completion (the shape whose absence let the regression through).
• The write-side guard is now exercised through a real node self-handoff + teardown path, not only a hand-built _turn.
• The stop-event test now asserts on the serialized snapshot and rebuilds/resumes from it, so a payload-only bug can't pass.

State of the PR: 63/63 swarm tests pass (was 61), ruff and mypy clean, Codecov all modified lines covered.

Settling the frontier/handoff at save time keeps paying off here — the no-handoff case is now just another branch in that one writer, and the fixed-point test locks the resume contract across crash generations. No remaining blocking concerns from my side.

[github-actions]

Assessment: Approve (re-review of 085b4cf)

This follow-on — "release the turn record when a handoff advances the frontier" — closes the last teardown window in the frontier/handoff state machine. I reproduced the gap, verified the fix, and reverse-tested it.

The window it closes

After a handoff transition advances the frontier (current_node = target, handoff_node = None) but before the target starts its turn, self._turn still held the committed record from the node that just finished. A teardown in that window made serialize_state see turn_committed = True with handoff_node = None → the committed branch fell to its else and persisted an empty frontier, silently dropping the handoff target. The one-line self._turn = None after the transition makes such a teardown fall to the elif ... current_node branch instead, correctly recording ['second'].

• Reachability: exercised via a BeforeNodeCallEvent hook that raises before second starts — a real between-nodes crash point, not a synthetic state.
• Reverse-test: dropping the self._turn = None line regresses test_swarm_teardown_between_nodes_keeps_the_handoff_target_as_frontier's next_nodes_to_execute from ['second'] to [] (and resume then never runs second).

State of the PR: 64/64 swarm tests pass (was 63), ruff and mypy clean, tree clean.

The _turn lifecycle is now consistent end to end — it's cleared exactly when the finished turn stops speaking for the frontier, mirroring the empty-frontier handling on the no-handoff path. No remaining blocking concerns from my side.

[opieter-aws]

@strandly-the-agent review latest

[strandly-the-agent]

Warning

Superseded and partly retracted — see #pullrequestreview-4823897865 for my actual position.

This review was submitted through a tooling error on my side and overstates the severity. The finding below is factually reproducible — the terminal AfterNodeCallEvent write does land status="executing" with frontier=[], and restoring exactly that write does reset and discard the finished run — but framing it as trading "a compounding-history bug for a worse one" is wrong: the baseline it compares against (87ff5ada) is the revision that wedged the swarm into FAILED with zero executions. Re-running the task is bounded and within the documented at-least-once contract, so this is a non-blocking trade-off, not changes-requested. My corrected review approves 085b4cf3 and carries the item as 🟡 with that framing.

---

Original text, kept for the record (severity overstated — read the correction above first)

🔴 Changes requested — the no-handoff empty-frontier fix trades a compounding-history bug for a "discard the whole completed run" bug.

✅ Re-reviewed exact head 085b4cf3 against prior head 87ff5ada and merge base 59bf1ed6.
✅ Commit 2d28c70f genuinely fixes the compounding-history blocker I raised last round — reverse-tested against 87ff5ada.
✅ Commit 085b4cf3 genuinely fixes the teardown-between-nodes handoff-drop — reverse-tested by removing the self._turn = None line, frontier regresses ['second'] → [].
🔴 New regression: a committed terminal turn now serializes next_nodes_to_execute = [], and deserialize_state treats any empty frontier as "nothing to resume" — routing to _reset_for_fresh_run(), which discards node_history, node_results, accumulated_usage/metrics and shared_context wholesale even though the run actually succeeded.

Verified correct (both commits do what they claim):

• 2d28c70f (swarm.py:1049-1057): a committed turn with no handoff now serializes an empty frontier instead of re-listing the just-finished node. Reverse-tested: collapsing it back to the pre-fix form makes a solo/max_iterations=2 repeated-crash-at-the-same-point repro compound node_history across generations again.
• 085b4cf3 (swarm.py:891): self._turn = None right after the handoff transition. Confirmed a teardown between "first"'s handoff and "second" starting persists next_nodes_to_execute=['second'] with 085b4cf3, vs [] with only 2d28c70f applied (silently dropping the target).

The window: the new empty-frontier branch is reached whenever a turn commits with no handoff while completion_status is still EXECUTING — before the loop's else: completion_status = Status.COMPLETED. SessionManager wires AfterNodeCallEvent → sync_multi_agent, and that hook fires from inside that window for every node including the terminal one, so FileSessionManager writes it to disk. A real on-disk repro (A→B, B terminal, no crash injected):

write #0 (after A's AfterNodeCallEvent): status=executing node_history=['first']          next_nodes=['second']
write #1 (after B's AfterNodeCallEvent): status=executing node_history=['first','second'] next_nodes=[]
write #2 (after AfterMultiAgentInvocationEvent): status=completed node_history=['first','second'] next_nodes=[]

Restoring write #1 gives completion_status=PENDING, node_history=[], results={} — the entire swarm reruns from the initial node. The identical repro against 87ff5ada writes next_nodes=['second'] at that point, so restore there resumes at "second" while preserving history/results for "first".

Suggested direction: flipping completion_status to COMPLETED where the turn commits, instead of after the AfterNodeCallEvent hook, would make every checkpoint written in this window report status="completed" — funnelling into the documented reset-on-resume contract instead of colliding with it under a misleading "executing" label.

[strandly-the-agent]

✅ Approving 085b4cf3 — both new commits are genuine fixes, verified by my own repros. One non-blocking trade-off to decide, inline.

First, a correction of my own. The review I posted 20 minutes ago (#pullrequestreview-4823866300) went out through a tooling error on my side, and it overstates the severity: it frames the terminal-window behavior as trading "a compounding-history bug for a worse one". My measurements don't support "worse" — the baseline it compares against (87ff5ada) is the revision that wedged the swarm into FAILED. Treat this review as my position; that one is superseded.

✅ 2d28c70f (empty frontier for a committed turn with no handoff) — verified a fixed point: three successive crashes at the same no-handoff completion each persist history=['solo'] frontier=[], and a clean resume is completed / execution_count=1. At 87ff5ada the same script compounds to ['solo','solo','solo'] and then returns failed with zero node calls. My blocker is closed.
✅ 085b4cf3 (self._turn = None after the handoff transition) — verified a real guard against skipped work: tearing down at the multiagent_handoff event persists frontier=['second'] at head, but frontier=[] at 2d28c70f, where the resume then drops second entirely (history=['first'], only first re-runs).
✅ Exact head: 106 focused tests pass (Swarm 64 + file-session + multi-agent hooks), changed-file ruff check/format --check clean, mypy on swarm.py clean, git diff --check clean.
⚠️ CI rollup is red only on label-size; every test/lint/docs/integration job and CI Gate are green — looks like the diff outgrew its size label rather than a functional failure.

The one non-blocking item (detail inline)

The terminal node's AfterNodeCallEvent checkpoint — a real SessionManager write — now lands status="executing" with frontier=[]. Restoring exactly that write resets and discards the finished run, so a crash in the gap before the invocation-final write costs a full task re-run instead of a one-node replay. It is bounded and within your at-least-once contract (no skipped work, no corruption, no wedge), and it converges with what a completed checkpoint already does — so which side of the trade you want is your call, not a merge blocker.

The _turn lifecycle reads consistently end to end now: it is created when a turn starts, committed where the state that makes it true is durable, rolled back on every teardown path, and released the moment the finished turn stops speaking for the frontier. Nice iteration on a genuinely subtle state machine.

[JackYPCOnline]

From my reading of the description, I thought swarm resume is broken, after reading the code, I found it fixes non-happy path case, dies in a specific window so can we explicitly update the description?

[JackYPCOnline]

Given TS avoids this entirely by construction, do we have a tracking issue for this? I would rather have this as temorary bridge.

[github-actions]

Assessment: Approve (re-review of 542f396)

Pure rename refactor — _TurnCheckpoint → _InflightTurn — plus a docstring clarification. No behavior change.

• Rename is complete and consistent: class, the self._turn annotation, the construction site, and the test import/usage all updated; a grep finds zero stale _TurnCheckpoint references in src/tests.
• 64/64 swarm tests pass, ruff and mypy clean.
• The new name is an improvement: _InflightTurn describes the object (a turn in progress) and no longer collides conceptually with the serialize checkpoint, which had made the old name ambiguous.

No blocking concerns on the code.

Two open maintainer questions worth resolving before merge (@JackYPCOnline)

Neither is a code defect, but both seem worth closing out:

1. Description scope — the opening reads as if swarm resume were broadly broken, when the fix actually targets two specific crash-window (non-happy-path) defects. Tightening that framing in the description would match what the code does.
2. Convergence tracking — since the TS SDK avoids this class of bug by construction (deriving the resume point from saved results rather than persisting a frontier), a tracking issue to treat Python's frontier-persistence as a temporary bridge would capture the intended direction.

[opieter-aws]

Given TS avoids this entirely by construction, do we have a tracking issue for this? I would rather have this as temorary bridge.

The underlying cause is that Python stores the resume frontier as a separate field from node_history, so serialize_state has to work out what’s still owed from status + handoff_node + a turn marker.

TS doesn’t have this problem since the handoff decision is stored inside the node’s own result, so the result is the record that the turn committed.

Fixing that in Python would mean changing the persisted resume format and reworking how _execute_swarm records turns. That's a big Swarm rework with a session-compatibility story, not a quick follow-up. So I’d rather not frame this PR as a temporary fix

[JackYPCOnline]

Thanks for the reply and I see your point, i am incline to approve. Suggesting to have Murat / Patrick to do a bar raise before merging.

[mkmeral]

my 2c around the last node edge case: we don't make any idempotency promises around swarm runs, the whole point is handing the task around, so the agent might re-run multiple times. Also given the current case is crash, I am okay merging this. I'd like to solve this edge case if we can, but I wouldn't want that to block this PR. so all good from me