[BUG] Multi-agent restore failures skip retries and overwrite checkpoints #4396

Checks

• I have updated to the latest minor and patch version of Strands (verified against current main)
• I have checked the documentation and this is not expected behavior
• I have searched ./issues and there are no duplicates of my issue

SDK Language

TypeScript and Python

Strands Version

Current main at 08f5bdd

Language Runtime Version

Python 3.14.7 for the dynamic reproduction; the affected control flow is runtime-independent

Operating System

macOS for the dynamic reproduction; the affected control flow is operating-system-independent

Installation Method

git clone

Steps to Reproduce

1. Persist a mid-run Graph or Swarm snapshot after node A completes, with node B still pending.
2. Create a new orchestrator and session manager for the same session and orchestrator ID.
3. Use snapshot storage whose first read throws a transient error and whose second read would return the persisted snapshot.
4. Invoke once and observe the storage error.
5. Retry with the same orchestrator and session manager after storage recovers.
6. Observe that the storage read count remains 1: the retry never attempts to load the checkpoint, node A can run again, and a subsequent automatic save can replace the valid checkpoint with state from the restarted run.

A minimal fail-once Python storage reproduced this exactly: after the first read raised, the retry made no second read (read_calls_after_retry=1) while the Graph ID remained in _multi_agent_restored_ids.

Expected Behavior

A failed restore attempt remains retryable. After storage recovers, the next invocation reads and applies the checkpoint, resumes at node B, and preserves completed node A. A successful read that finds no snapshot may still count as a completed restore attempt.

Actual Behavior

Both SDKs mark the Graph or Swarm as restored before its snapshot has been read and applied. If the storage read, deserialization, validation, or state load throws, the marker remains set. Retrying with the same orchestrator and manager skips restoration, starts from fresh in-memory state, and may overwrite the valid mid-run checkpoint on the next automatic save. Completed or non-idempotent nodes can run twice without warning.

Additional Context

The TypeScript SessionManager._onBeforeMultiAgentInvocation commits the guard before the awaited read and state load:

if (this._multiAgentRestoredIds.has(event.orchestrator.id)) return
this._multiAgentRestoredIds.add(event.orchestrator.id)
const snapshot = await this._snapshotStorage.loadSnapshot({ location })
// snapshot is applied after the marker is committed

The Python SnapshotSessionManager._on_before_multi_agent_invocation has the same transition:

if orchestrator.id in self._multi_agent_restored_ids:
    return
self._multi_agent_restored_ids.add(orchestrator.id)
await self._restore_multi_agent(orchestrator)

The bug is the premature guard transition, not the underlying storage failure. Current tests cover successful and malformed snapshot restoration, but not retrying the same manager after a failed restore.

Possible Solution

Commit the restored marker only after the read and snapshot application return successfully, or roll back the marker when restoration throws. If concurrent first invocations are supported, track an in-flight restore separately rather than treating “started” as “completed.”

Add a regression test in each SDK with fail-once storage and a valid mid-run snapshot. Assert that the second invocation performs a second read, resumes only pending nodes, and does not overwrite the checkpoint before restoration succeeds.

Related Issues

• [FEATURE] Support snapshot session management for multi-agent (Graph/Swarm) #3662
• feat(session-py): support snapshot session management for Graph and Swarm #4240
• Original review finding