[BUG] Python parallel node saves can regress the latest multi-agent snapshot #4397

Checks

• I have updated to the lastest minor and patch version of Strands
• I have checked the documentation and this is not expected behavior
• I have searched existing issues and there are no duplicates of my issue

SDK Language

Python

Strands Version

main at 08f5bdd3ad105af661f76c98b9adb0af024ae7f7

Language Runtime Version

Any supported Python version; this is an implementation-level asynchronous write race

Operating System

Any; not operating-system-specific

Installation Method

git clone

Steps to Reproduce

1. Create a Python Graph with two parallel entry nodes and attach SnapshotSessionManager using the default multi_agent_save_latest_on="node" strategy.
2. Use a deterministic asynchronous Storage backend for snapshot_latest.json that:
   • blocks node A's first write after capturing state S_A;
   • lets node B complete and persist newer state S_AB;
   • releases A so its older write finishes last; and
   • blocks the invocation-end write before it commits.
3. Read or restore snapshot_latest.json while the invocation-end write is blocked.
4. Observe that the stored snapshot is S_A, so B is no longer marked complete and may execute again after restart.

This ordering can occur naturally with remote storage when parallel node saves have different latency and the process stops before the invocation-end save repairs the checkpoint.

Expected Behavior

Once a snapshot includes a completed node, a later-finishing save must not replace it with an older execution frontier. After B completes, snapshot_latest.json must retain state at least as new as S_AB.

Actual Behavior

Python node-save callbacks can overlap, and every save unconditionally overwrites the same key. Storage completion order can therefore differ from snapshot capture order: A's older S_A write can replace B's newer S_AB write. Restoring during that failure window loses B's completed state and can repeat non-idempotent work.

Additional Context

The race is present in the current Python code path:

1. Graph._execute_nodes_parallel runs node streams in separate asyncio tasks.
2. Each task updates shared Graph state and independently awaits AfterNodeCallEvent callbacks in Graph._execute_node.
3. SnapshotSessionManager._save_multi_agent_latest captures state and asynchronously overwrites one shared key without per-orchestrator serialization or a monotonic write condition.

LocalFileStorage is unlikely to expose the race because its replacement path does not yield mid-write. An asynchronous backend such as S3 can expose it.

This specific race does not occur in the TypeScript Graph path, either at the 100429263 revision originally cited by this issue or on current main. Parallel node producers use await queue.send(...) for AfterNodeCallEvent; one queue consumer awaits the hook callback and calls ack() before processing the next queued event. Its Graph-driven snapshot saves therefore cannot overlap. TypeScript Swarm also executes nodes sequentially.

The TypeScript SessionManager.saveSnapshot method itself has no lock or monotonic-write guard, so independently issuing concurrent manual saves to the same location could still race. That lower-level possibility is not the automatic parallel-Graph execution path described by this issue. The original PR discussion concluded that both SDKs were affected after comparing the save methods, but tracing event dispatch shows that conclusion was incorrect.

Possible Solution

Serialize the complete capture + write operation per (session_id, orchestrator_id), using the same mechanism for node-end and invocation-end saves. Locking only after capture would still permit stale snapshots to queue.

Add the deterministic delayed-write regression described above and assert that the final readable snapshot contains both completed nodes. This fix is Python-specific; no TypeScript port is needed unless its event-dispatch model changes.

Related Issues

• [FEATURE] Support snapshot session management for multi-agent (Graph/Swarm) #3662
• feat(session-py): support snapshot session management for Graph and Swarm #4240
• Original review discussion: feat(session-py): support snapshot session management for Graph and Swarm #4240 (comment)