Repository navigation
fix(multiagent): preserve shared context and cumulative accounting across serialize/deserialize - #3396
Conversation
…ross serialize/deserialize
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
Assessment: Comment (approve-leaning — no blocking issues) Well-targeted fix for two real round-trip defects. I confirmed the swarm resume path ( Review notes
Nice job pinning down the object-identity subtlety and driving the full restore→resume→run path end to end. |
|
@strandly-the-agent review pls |
strandly-the-agent
left a comment
There was a problem hiding this comment.
🔴 Changes requested: persisted execution time still drops the current invocation at crash checkpoints, and Graph returns prior-only elapsed time after resume.
✅ Reviewed exact head ac77e2501b701fbfedbe9de3ab55504d3e53f542 · 112 focused tests passed · changed-file Ruff/format + diff check passed · exact-head CI green.
🔴 Two execution-time lifecycle blockers are inline; both were reproduced deterministically.
Review passes and verification
- Correctness/adversarial: repeated the FileSessionManager process-crash repro twice; both checkpoints stored
execution_time=0after a 250ms node, and both resumed past a 150ms total timeout. A separate mocked-clock probe restored 555ms, ran 600ms, returned 555ms, then left state at 1155ms. - Surviving probes: restored usage/metrics/execution count accumulated correctly; Swarm restored shared context retained identity and reached resumed model input.
- Tests:
hatch test -q tests/strands/multiagent/test_graph.py tests/strands/multiagent/test_swarm.py→ 112 passed in 2.54s. - Static checks: Ruff check/format passed on all four changed files;
git diff --checkpassed. The full-tree local format check found 11 unrelated files outside this diff; GitHub Python Lint and CI Gate are green at this SHA. - Suppressed: resolved threads about full-dict equality, shallow copies, and evergreen docstrings; weaker test/doc observations subsumed by the reproduced defects.
- Routing: correctness, adversarial, test-quality, and docs passes ran; API/LLM-context were not applicable because no public/model-facing surface changed.
|
@strandly-the-agent re-review latest fixes |
|
Assessment: Approve (re-review of The latest commit resolves both of @strandly-the-agent's execution-time lifecycle blockers. The fix centralizes the logic in two Verification
Nice turnaround on the lifecycle subtlety — extracting the interval math into shared helpers made both the checkpoint and result paths correct in one place. |
strandly-the-agent
left a comment
There was a problem hiding this comment.
🔴 Changes requested: both prior execution-time blockers are verified fixed at ef85b239, but tracing setup failures leave the new active timer running and corrupt later checkpoints.
✅ Exact head ef85b239f21c019ba16fdba1fb6720f25365bc85 · 115 focused tests passed · changed-file Ruff/format + diff check passed · exact-head CI green.
✅ Prior blockers resolved: the real FileSessionManager crash/restart repro now checkpoints 251ms and correctly rejects B under a 150ms total timeout (repeated twice); restored/nested timing now commits and propagates exactly once.
🔴 One new-delta lifecycle blocker is inline.
Review passes and verification
- Correctness/adversarial: exercised process restart, restored and nested Graph timing, repeated serialization, final-hook errors, execution errors, cancellation, early generator close, back-to-back invocations, and Swarm parity. No double-counting or marker leak occurred after entering the protected span.
- New failure repro:
start_multiagent_span()and span context entry failures each leaked the marker for Graph and Swarm; repeated twice identically. At faket=1000, serialization att=1000.5reported 500ms after the invocation had already raised. The prior head remained at 0. - Tests:
hatch test -q tests/strands/multiagent/test_graph.py tests/strands/multiagent/test_swarm.py→ 115 passed in 2.74s. - Static/CI: Ruff check and format passed on all five changed files;
git diff --checkpassed; GitHub Python 3.10–3.14 matrices, integration, lint, docs, and CI Gate are green. - Routing: correctness, adversarial, and test-quality passes ran. API/LLM-context/docs were not applicable. The advanced adversarial tier was unavailable under data-retention mode; its default-tier retry completed.
- Suppressed: resolved full-dict equality, shallow-copy, and evergreen-docstring threads; the weaker Swarm lifecycle test gap is subsumed by the reproduced tracing failure and requested regression test.
|
@strandly-the-agent re-review |
|
Assessment: Approve (re-review of The only change since my last pass (
No new issues. Both previously-blocking findings remain resolved; PR stays approvable pending a maintainer formally clearing @strandly-the-agent's "Changes requested" status. |
|
Assessment: Approve (re-review of All three of @strandly-the-agent's execution-time lifecycle blockers are now resolved and each is locked down by a targeted regression test. The latest change fixes the third blocker (tracing-setup failure leaking the timer) by moving the Verification
Solid handling of a genuinely subtle lifecycle seam — placing the timer start inside the span context keeps the start/commit symmetric and closes the failure-path leak. Note: @strandly-the-agent's review is still in "Changes requested" state at the review level; only that reviewer or a maintainer can formally dismiss it before merge, though all its threads are now resolved. |
strandly-the-agent
left a comment
There was a problem hiding this comment.
✅ Approved: the timer-marker leak is fixed for both Graph and Swarm; no findings remain at aca610f4b29118db4c9e20ef3a65729292ac469c.
✅ 121 focused tests passed · Ruff/format/diff-check clean · exact-head CI green · tracing failure matrix and real crash/restart repros passed repeatedly.
Re-review evidence
- Prior blocker: verified resolved. Graph and Swarm now publish
_invocation_start_timeonly after successful span entry; span creation anduse_span.__enter__failures both leave the marker unset and serialization stable. - Cold adversarial checks: Graph/Swarm × both tracing failure phases passed in two local runs (8/8 cases). FileSessionManager crash/restart passed twice: a 250ms node persisted 251ms, and resume correctly rejected B under a 150ms total timeout.
- Lifecycle checks: restored/nested totals, repeated serialization, cancellation, early close, node/final-hook errors, and back-to-back invocations showed no leaks or double-counting. Main
#3390resume-frontier interactions also passed. - Merge provenance:
aca610f4matches the synthetic merge of fix commit6c9ca250with main7b0717c4; no conflict-resolution hunk altered the timer fix. - Tests/static:
hatch test -q tests/strands/multiagent/test_graph.py tests/strands/multiagent/test_swarm.py→ 121 passed in 2.91s. Ruff check/format andgit diff --checkpassed; worktree stayed clean. - CI: Python 3.10–3.14 Linux/Windows/macOS, integration, lint, docs, dependency review, and CI Gate all succeeded at this SHA.
- Routing: correctness, adversarial, and test-quality passes ran; API/LLM-context/docs were not applicable. The advanced adversarial tier was unavailable under its data-retention restriction; the default-tier retry completed.
Description
SwarmandGraphpersist their run state viaserialize_state()/deserialize_state()so a session can be resumed after a restart. Two round-trip defects meant that resuming silently discarded state.Swarm shared context was erased on the next save.
serialize_statereads shared context from theSwarmState-owned object while the restore path wrote it to theSwarm-owned object. During a live run these are the same object (stream_asyncbinds them when it builds a freshSwarmState), but the resume path never rebuilds that state — so afterdeserialize_state, the restored context lived onself.shared_contextwhileserialize_statekept reading an emptyself.state.shared_context, and the next save persisted{}. The fix re-establishes the object identity a live run relies on, so the serialize path and the node-input builder read the same context after resume.Cumulative accounting reset to zero on both orchestrators.
serialize_statenever persistedexecution_time/accumulated_usage/accumulated_metrics(plusexecution_counton Graph), and the restore path never rehydrated them. After a resume these reset to their defaults, which corrupts the total-execution timeout budget (should_continueadds accumulatedexecution_timeto fresh wall-clock) and under-reports the token/metric totals in the final result. These fields are now serialized and restored, so a resumed run continues accounting from where it left off rather than starting over.Swarm restores three fields; Graph restores four — Graph tracks
execution_countexplicitly, whereas Swarm derives it fromnode_history, which already round-trips.Related Issues
Requirement for #3283
Documentation PR
N/A
Type of Change
Bug fix
Testing
Added a
serialize → deserialize → serializeround-trip regression test for each orchestrator, asserting the second serialize equals the first for shared context (Swarm) and the cumulative fields (both). Confirmed both tests fail on the pre-fix source and pass after.Beyond the round-trip invariant, I drove the full restore → resume → run path end to end: restored a mid-run payload into a fresh instance, called
invoke_async, and confirmed the resumed run accumulates on top of the restored totals (e.g. restored 33 tokens + 30 from the resumed node = 63; Graphexecution_count3 → 4) and that the Swarm node receives the restored shared context during execution. Ran the full multiagent unit suite and ruff.Note:
hatch run preparecould not run locally — the dev env fails to build an unrelated optional dependency (cedar-policy-mcp-schema-generator, needs rustc 1.89). The full gate will run in CI.hatch run prepareChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.