[nicolas-m-romero]

Description

SnapshotSessionManager (added in #3283) is single-agent only — attaching it to a Graph or Swarm raises NotImplementedError, leaving orchestrators on the message-log managers (FileSessionManager / S3SessionManager), which have no checkpointing and cannot resume from a snapshot after a crash or restart. Applications that run both single agents and orchestrators are forced to mix two session systems with two on-disk formats, and the legacy managers cannot be deprecated while they remain the only option for orchestrators.

This brings the Python SDK to parity with the TypeScript SessionManager, which already persists Graph and Swarm through initMultiAgent with a node/invocation save strategy. Python uses the spec-defined multiAgent storage scope for a consistent cross-SDK on-disk format. A long-running Graph that crashes on node 40 of 60 now resumes from the last completed node instead of re-running from the start, and a single Storage backend can hold both an app's agents and its orchestrators.

Two decisions worth surfacing for review. State is restored lazily on the orchestrator's first invocation (guarded so it runs once), rather than eagerly at construction — matching TS and deferring the storage read until it is actually needed. Orchestrator state is wrapped in the versioned Snapshot envelope rather than persisted as a bare dict, so the format can evolve, and the load path rejects a snapshot whose orchestrator id does not match the target.

One open question for maintainers: the issue notes overlap with durable execution / checkpointing (#2777, #1230) and asks which layer should own orchestrator resume. This implements it at the session-management layer to match TS; if that ownership belongs elsewhere, it is worth settling before merge.

Related Issues

Resolves #3662

Documentation PR

Updated the session management guide with Python Graph configuration and save-strategy behavior.

Type of Change

New feature

Testing

• I ran hatch run prepare

Additionally exercised end to end: a Graph and a Swarm run with the manager attached persist their state to the multiAgent scope, a fresh orchestrator restores from that snapshot on its first invocation, and a completed orchestrator re-runs from the start rather than resuming a terminal frontier.

Checklist

• I have read the CONTRIBUTING document
• I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
• My change is focused and reasonably small; I have split unrelated work into separate PRs
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

Public API Changes

SnapshotSessionManager gains a multi_agent_save_latest_on option and no longer rejects orchestrators:

# Before — attaching to an orchestrator raised
Swarm(nodes=[...], session_manager=SnapshotSessionManager("s", storage=storage))
# NotImplementedError: ... does not support multi-agent (Graph/Swarm) persistence

# After — orchestrators are persisted and restored
from strands.session import SnapshotSessionManager
from strands.storage import LocalFileStorage

session = SnapshotSessionManager("s", storage=LocalFileStorage(), multi_agent_save_latest_on="node")
swarm = Swarm(nodes=[...], session_manager=session, id="my-swarm")
swarm("task")   # snapshot written after each node and at invocation end
# ... process restarts ...
swarm("task")   # state restored lazily on the first invocation

multi_agent_save_latest_on: Literal["node", "invocation"] = "node" chooses whether the orchestrator snapshot is saved after every node (a mid-run crash resumes at the last completed node) or only at invocation boundaries (lower I/O for a large graph on remote storage). Orchestrators are persisted latest-only under a distinct scope, session/<session_id>/scopes/multiAgent/<orchestrator_id>/snapshots/, so single-agent snapshots (scopes/agent/...) are unchanged and the two never collide. The Snapshot scope literal additionally accepts "multiAgent". Storage must be supplied to the constructor when the manager backs an orchestrator, since an orchestrator has no agent-level storage to resolve from.

[codecov]

Codecov Report

❌ Patch coverage is 96.61017% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...py/src/strands/session/snapshot_session_manager.py	95.12%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[nicolas-m-romero]

@strandly-the-agent review

[nicolas-m-romero]

@strandly-the-agent review

[strandly-the-agent]

Changes requested — the crash-between-nodes resume genuinely works (verified end to end), but two reachable defects in the restore path don't: an interrupted Graph/Swarm can never resume in a new process, and a failed restore silently overwrites the good checkpoint. The headline resume path also has no test, and the docs page still says this feature doesn't exist.

• 🔴 Lazy restore lands after the orchestrator has consumed the interrupt response → interrupted runs can't resume after a restart.
• 🔴 Orchestrator id is marked restored before the restore runs → a transient read error or corrupt/newer snapshot means the retry silently re-runs from scratch and overwrites the checkpoint.
• 🟡 Storage-required check is construction-order dependent; a per-node save failure aborts the run and masks the node's error; multiagent/snapshot.py reads public by accident; resume-through-the-manager is untested (3 delete-restore mutations survive 121 tests).
• 🟡 Docs: site/.../session-management.mdx:92,215 still say SnapshotSessionManager does not support Graph/Swarm; the PR template asks for docs in-PR.
• Public surface changed (new kwarg, widened Scope, new module, removed NotImplementedError) and there's no needs-api-review label — it needs one. Not an API-meeting case.

Verified (head 4408ed0)

✅ Tests	pytest tests/strands/session tests/strands/multiagent tests/strands/agent/test_snapshot.py --ignore=…/a2a → 450 passed; the two touched files → 121 passed
✅ Lint	ruff + mypy clean on changed files
✅ Repros	both 🔴s reproduced with real Graph/Swarm + mocked models (no Bedrock), independently re-run; fix for 🔴#2 applied locally → 121 pass, C1 scenario flips to resuming ['b','c'] with the snapshot intact
✅ Works	linear Graph crash-after-node resume (runs only the remaining nodes); Swarm crash after 1st/2nd node resume; completed-then-reinvoked Graph and Swarm start clean with the new task
✅ Cleared	register_hooks override leaks none of the base's raising sync_multi_agent/initialize_multi_agent stubs (registry dump); delete_session covers multiAgent keys; no dangling refs to _reject_multi_agent
⚠️ Passes	adversarial pass produced scripts/output but timed out before reporting (results folded in where corroborated); API/DevX pass ran on the default model tier
📎 Artifacts	repro scripts + captured output, mutation-test results, and 9 verified proposed tests are uploaded to the Strandly portal for this PR

Issue alignment vs #3662 — partially aligned

Every letter item is implemented at TS parity (key layout byte-identical; lazy restore + restored-ids guard; node/invocation cadence registering AfterNodeCallEvent only under node; versioned envelope + id check). The gap is proof and prose.

In-contract: types/_snapshot.py:20,31 · multiagent/snapshot.py:22-59 · snapshot_session_manager.py:71-83, :159-168, :240+:275-279, :318-368, removal of _reject_multi_agent.
Supporting: the new tests, test_snapshot.py:304, docstrings :162, :218-221, :253-254.
Out-of-contract: multiagent/snapshot.py as a non-underscore module with public-looking functions (see inline). Nothing else — the issue's follow-ups (migration, deprecation, #3633 fix, rewind) are correctly absent.

Acceptance bar	Status
(a) crash/restart stopping between nodes	❌ no test drives restore through the manager (behaviour verified working by probe)
(b) completed-then-reinvoked	⚠️ :205 exists but the re-invoke assertion result.status is not None can't fail; bypasses the manager
(c) serialize→deserialize→serialize invariant	⚠️ not in this PR; covered on main (test_swarm.py:1288, test_graph.py:3211); envelope round-trip verified by probe
(d) four landmines	2/3/4 fixed on main and correctly relied on; #3633 still open, its precondition is this PR's default cadence, and the diff doesn't mention it

Questions

Blocking

1. Ownership. #3662 is agent-filed with zero maintainer comments, and you raise the layer question yourself. Closest signal on record is zastrowm on #1230 endorsing a snapshotting session manager as the TS design. Is a maintainer ack on #3662 the real gate here?
2. Storage contract. Agents get a LocalFileStorage fallback; orchestrators hard-fail — but only if no Agent resolved storage first. Is the intent "storage required for orchestrators", or "never guess a backend"? (If the latter, is the agent fallback the thing that's wrong?)
3. #3633. The default "node" cadence is precondition #2 of the open fan-in double-execute, and we reproduced it through this manager (checkpoint frontier ['a','c'] with sibling a still PENDING → c ran twice on resume). This PR adds no new exposure — the repository managers already persist per node with no opt-out — so "caveat + ship" is defensible. Land #3633's fix first, or document the caveat in the MultiAgentSaveLatestStrategy docstring and link it from the PR?
4. Resolves vs Part of. Latest-only is in-contract, but Resolves auto-closes the issue with its rewind use case undelivered. Keep Resolves and re-file rewind onto #2777, or downgrade?

Non-blocking
5. Is cross-SDK reading of snapshot payloads a goal, or only path compatibility? data uses orchestrator_id vs TS orchestratorId, so the byte-identical key holds payloads the other SDK can't read. That answer also settles the camelCase "multiAgent" literal vs AGENTS.md:52, and deserves a team/DECISIONS.md line either way.
6. sync_multi_agent/initialize_multi_agent still raise "does not support multi-agent persistence" on a manager that now does. Intentional, or worth overriding for API honesty?

Reading order

1. src/strands/types/_snapshot.py:20,31 — the Scope/VALID_SCOPES widening; smallest hunk, highest criticality (persisted cross-SDK format).
2. src/strands/multiagent/snapshot.py — the envelope contract: take_snapshot :22, load_snapshot :40 with its three rejection checks.
3. src/strands/session/snapshot_session_manager.py:318-368 — the whole feature: _init_multi_agent wiring → lazy restore + guard (:336-342) → save handlers → restore/save internals. All inline findings live here.
4. Same file, supporting surface: MultiAgentSaveLatestStrategy :71-83, key builder :159-168, class docstring :218-221, ctor kwarg :240/:275-279.
5. tests/strands/session/test_snapshot_session_manager.py — write path :89-125, envelope + restore :128-223, wiring :226-265.
6. tests/strands/agent/test_snapshot.py:304 — the only change outside the feature.

Appendix — non-blocking (19) + pre-existing

Robustness

1. 🟡 snapshot_session_manager.py:363-368 — sibling nodes in a parallel Graph batch each save the whole snapshot to one key with no ordering/CAS; on S3 (put_object via to_thread) snapshot_latest can go backwards (simulated: [['p1','p2'], ['p1'], …]). LocalFileStorage is safe (tmp + os.replace). Fix: asyncio.Lock per orchestrator across serialize+write.
2. ⚪ :159-166 — _multi_agent_latest_key lacks the non-empty/./.. guard the ctor applies to session_id; id="" collapses the path segment. Build the key once in _init_multi_agent so a bad id fails at construction, not mid-run.
3. ⚪ :339 — restored-ids keyed on orchestrator.id; two unnamed orchestrators (default_graph) on one manager share a key and the second never restores. Warn when an id is wired twice.
4. ⚪ multiagent/snapshot.py:40 — a Graph and Swarm sharing an id: KeyError: 'context' on restore; load_snapshot checks scope and id but not state["type"].

Tests (mutation-tested: 18 applied, 8 survived; 9 verified replacement tests are in the uploaded artifacts)
5. 🟡 :332-333, :344-350 — neither cadence is proven, only hook registration; both persisted-after-run tests use one-node orchestrators. A write-count test (node→3, invocation→1) kills three surviving mutations; pattern exists at :481-501.
6. 🟡 :226-250 — registration tests assert event types only; swapping save/restore handlers keeps everything green. Assert (event_type, callback) pairs.
7. 🟡 multiagent/snapshot.py:51 — deleting snapshot.validate() breaks no test; schema-version rejection unproven.
8. ⚪ :223 — assert result.status is not None is vacuous (Status is non-optional). Pin COMPLETED, one fresh turn of history, and the new task.
9. ⚪ multiagent/snapshot.py:36 — app_data untested and stored by reference, while Agent.take_snapshot deep-copies (agent/agent.py:2024). A behavioural decision, not just a test gap.
10. ⚪ No tests/strands/multiagent/test_snapshot.py mirroring the new module (strands-py/AGENTS.md:267).
11. ⚪ test_snapshot.py:304 — set(VALID_SCOPES) == set(get_args(Scope)) is self-consistency, not value-pinning; == {"agent", "multiAgent"} costs nothing.
12. ⚪ No integ coverage for orchestrators (tests_integ/test_snapshot_session.py); "node" cadence against real S3Storage is the case a unit test can't reach.

API / docs
13. 🟡 session/__init__.py:19-28 exports SaveLatestStrategy but not MultiAgentSaveLatestStrategy, which the ctor docstring points users at (:254). Package docstring :5-9 still says "persists a single agent".
14. ⚪ Ctor Raises: (:260-263) omits the new multi_agent_save_latest_on ValueError (strands-py/AGENTS.md:118); _restore_multi_agent's bool return is unused.
15. ⚪ multiagent/snapshot.py:5 — "mirrors TS"/"byte-identical" is over-broad given orchestrator_id vs orchestratorId; narrow the claim to the key layout.
16. ⚪ types/_snapshot.py:20,31 — first camelCase literal value in a public Python type vs AGENTS.md:52; decide explicitly and leave a one-line comment citing the rule.
17. ⚪ :71-79 — a caveat naming #3633 and the fan-in condition under which "node" can double-execute on resume.
18. ⚪ Resolves #3662 over-claims the rewind use case (Q4).
19. ⚪ Docs layout block session-management.mdx:558-571 lacks multiAgent/, unlike the TS block at :593-607.

Pre-existing in graph.py resume (reproduce identically with FileSessionManager; not this PR's problem, will file as issues)

• A resumed Graph reports FAILED even when every node succeeds: _from_dict restores failed_nodes (graph.py:1476-1478) and never clears an entry that later succeeds. Lands on this PR's headline flow — a clean crash-restart resume returns FAILED.
• A completed Graph with a never-taken conditional edge serializes a non-empty frontier (_compute_ready_nodes_for_resume has no COMPLETED short-circuit, graph.py:1327-1329), so the next process resumes that branch with the old task.
• A Graph whose topology changed between deploys runs zero nodes and reports COMPLETED: _from_dict filters unknown frontier ids (graph.py:1509) but keeps _resume_from_session = True; Swarm guards this (swarm.py:1129-1134).
• Fan-in double-execute through this manager's default cadence → already open as #3633; confirmation belongs there.

[github-actions]

Assessment: Comment (mostly ready — a few loose ends)

This revision cleanly resolves the two headline defects from the prior review: the module is now private (_snapshot.py), and moving _interrupt_state.resume(task) after BeforeMultiAgentInvocationEvent in both graph.py and swarm.py correctly lets the lazy restore rehydrate interrupt state before the response is applied — nicely covered by the new interrupted-Graph/Swarm tests. The remaining items are packaging/docs correctness plus one concurrency edge; none block the core resume path.

Still open after this revision

• Docs contradict the feature: site/src/content/docs/user-guide/concepts/agents/session-management.mdx lines 92 and 215 still state "SnapshotSessionManager does not support Graph or Swarm." The PR description says "No documentation changes required," but these are now factually wrong and should be updated in this PR.
• Public type not exported: MultiAgentSaveLatestStrategy is missing from session/__init__.py __all__, even though the ctor docstring directs users to :data:MultiAgentSaveLatestStrategy`` and the parallel SaveLatestStrategy is exported. Users can't import it for annotations and the doc cross-ref won't resolve. The package docstring also still says it "persists a single agent."
• Concurrency (inline): default "node" cadence can reorder snapshot_latest writes across parallel graph nodes on a non-atomic backend.
• Docstring: ctor Raises: omits the new multi_agent_save_latest_on ValueError.
• Testing (inline): test_completed_orchestrator_reinvokes_from_scratch uses a vacuous is not None assertion.
• Process: public surface changed (new kwarg, widened Scope, new module) but there's no api/needs-review label — worth adding so the API change gets the required sign-off.

The mark-restored-before-restore ordering (🔴 in the open thread) I'm leaving to that discussion — the TS-parity rationale and follow-up plan are reasonable. Overall this is a well-structured, low-complexity port that matches the TS layout closely.

[github-actions]

Assessment: Comment (re-review of latest revision)

Good iteration — most open threads are now resolved.

Resolved this round

• Scope naming: reverted to multiAgent with a 1-line comment tying it to the shared TS on-disk format (@liramon2's call). Restores cross-SDK read compatibility, re-aligns with [FEATURE] Support snapshot session management for multi-agent (Graph/Swarm) #3662, and makes the PR description accurate again.
• Test quality: test_completed_orchestrator_reinvokes_from_scratch now asserts result.status == Status.COMPLETED and an empty resume frontier instead of the earlier vacuous is not None.
• Concurrency ("node" cadence write race): accepted as intentional TS-parity behavior, to be tracked as a separate cross-SDK issue (please link it here once filed).

Still open (minor)

• Ctor docstring: the Raises: block still lists only save_latest_on; it should also mention that an invalid multi_agent_save_latest_on raises ValueError (the check exists at snapshot_session_manager.py:266).
• Docs (out-of-PR): user-guide/concepts/agents/session-management.mdx still says SnapshotSessionManager "does not support Graph or Swarm" in two places — now contradicted by this feature. Worth a follow-up docs update.

Core feature and the interrupt-ordering fix look solid; only a docstring line and a docs follow-up remain.

[nicolas-m-romero]

@strandly-the-agent Could you give this PR another review.

[strandly-the-agent]

Approving at 53e1862. Of my six findings, one 🔴 and two 🟡 are fixed and verified; the other 🔴 and two 🟡 were deliberately kept for TS parity with tracking issues promised. Nothing new rises above ⚪. Three things to land before merge, none of which need another review from me:

• Link the two tracking issues you promised (restore-guard ordering; parallel-node write race) in their threads — neither exists yet as far as I can find, and the restore-ordering one ships a known silent-rerun-on-retry path.
• Ctor Raises: (snapshot_session_manager.py:251-254) still omits the multi_agent_save_latest_on ValueError.
• Public surface changed and there's still no needs-api-review label.

Status of previous findings

#	Finding	Status
🔴 1	Lazy restore after interrupt response consumed	Fixed (933a5fe): _interrupt_state.resume(task) now runs after BeforeMultiAgentInvocationEvent in graph.py:649-650 and swarm.py:424-425; new cross-process interrupt tests for Graph and Swarm drive restore through the manager. Re-verified: crash-after-n1 on a 3-node chain resumes in a fresh manager+graph with n1 not re-executed.
🔴 2	Id marked restored before restore runs (:176-179)	Open by choice — kept for TS parity, TS-first fix + port promised. Still reproducible; I'm not re-blocking on a decision you've explained, but please link the tracking issue.
🟡 3	Storage check order-dependent	Withdrawn for parity — accepted.
🟡 4	Per-node save failure aborts run	Withdrawn for parity — accepted.
🟡 5	multiagent/snapshot.py public by accident	Fixed (af67b17): _snapshot.py.
🟡 6	Restore through the manager untested	Fixed (933a5fe): the interrupt tests construct a fresh manager and restore via _on_before_multi_agent_invocation. A non-interrupt crash-between-nodes test would still be a nice addition but isn't needed.
🟡 docs	Site docs say unsupported	Still true (session-management.mdx:92,215), deferred to a follow-up by agreement — fine, but please open it so it isn't lost.

Verified (head 53e1862)

• pytest tests/strands/session tests/strands/multiagent tests/strands/agent/test_snapshot.py tests/strands/test_interrupt.py --ignore=…/a2a → 492 passed; ruff clean on changed files (mypy errored internally in my sandbox — environmental).
• Scope back to multiAgent with the one-line comment; MultiAgentSaveLatestStrategy exported; package docstring fixed; test_completed_orchestrator_reinvokes_from_scratch now asserts COMPLETED + new task.
• The reorder in graph.py/swarm.py is the only core change; all existing interrupt/resume tests pass.

Appendix — non-blocking

1. ⚪ graph.py:649-650 / swarm.py:424-425 — side effect of the reorder: if resume(task) raises TypeError (plain-string task while an interrupt is active), BeforeMultiAgentInvocationEvent has fired but AfterMultiAgentInvocationEvent never does (previously neither fired). Verified. Only matters to hooks that pair before/after; if you care, validate the task before dispatching the event.
2. ⚪ Pre-existing graph.py resume bugs from my first review (resumed run reports FAILED after a successful retry; completed Graph with a never-taken conditional edge resumes the old task; topology change runs zero nodes) are still unfiled — the first one shows up in the resume path above (run2 status: FAILED with every node completed). Happy to file them if you'd like.

[github-actions]

Assessment: Comment (re-review at 4b38b74)

The docs commit resolves the last of my non-parity items — session-management.mdx now documents multi-agent persistence and the node/invocation cadence instead of saying the feature doesn't exist. Combined with earlier revisions, the happy-path feature (crash-between-nodes resume, completed-then-reinvoked, versioned envelope + id check) is in good shape and the interrupt-ordering fix is correctly in place.

Remaining open — two deferred parity items (for maintainer merge decision)

Both are being deferred to separate cross-SDK issues on TS-parity grounds, which is a defensible stance — but they are real defects, not cosmetics, so they should be tracked+linked before merge:

• Restore guard marks-before-load (:328 before :329): a failed/transient restore is still recorded as "restored," so the retry runs from scratch and the next save overwrites the good checkpoint. This is local checkpoint data-loss (not a cross-SDK on-disk-format concern), and the fix is a ~2-line swap (mark restored only after a successful load). Worth weighing whether to just fix in-PR rather than defer.
• "node" cadence write race: concurrent node saves can regress snapshot_latest on a non-atomic backend (S3); safe on LocalFileStorage. Reasonable to document as a known limitation + track.

Minor: one inline note on the constructor Raises: docstring (omits multi_agent_save_latest_on). Nice work turning this around across revisions.

[github-actions]

Documentation Preview Ready

Your documentation preview has been successfully deployed!

Changed pages:

• user-guide/concepts/agents/session-management

Updated at: 2026-09-18T18:31:52.989Z