[SuperMarioYL]

Description

When a graph node is dispatched (asyncio.create_task) but the process crashes before the node's generator body sets Status.EXECUTING (e.g., Lambda timeout, container restart, os._exit), the node is left PENDING. On session resume, _compute_pending_sources seeds only from EXECUTING nodes — so a dispatched-but-not-started node with no completed ancestor is indistinguishable from a bypassed dead branch. Its fan-in children are marked ready off the completed sibling alone, and the fan-in is double-executed on resume.

This adds a DISPATCHED status to the Status enum, set at the dispatch site immediately before asyncio.create_task. _compute_pending_sources now seeds from both EXECUTING and DISPATCHED, closing the window between dispatch and generator-body start. This completes the partial fix from #3390, which added the EXECUTING seed but left the dispatched-but-not-started gap.

Changes:

• base.py: Add DISPATCHED = "dispatched" to Status enum (additive, non-breaking)
• graph.py: Set DISPATCHED at the dispatch site in _execute_nodes_parallel; update _compute_pending_sources to seed from EXECUTING and DISPATCHED
• test_graph.py: Add TestResumeDispatchedSibling with two tests verifying the fan-in is not double-executed when a sibling was dispatched but not started

Related Issues

Fixes #3633

Documentation PR

No documentation changes needed — this is an internal status enum addition with no new public API surface.

Type of Change

Bug fix

Testing

• Added TestResumeDispatchedSibling with two tests:

  • test_dispatched_sibling_excluded_from_resume_frontier: focused unit test that injects DISPATCHED state directly (right.execution_status = DISPATCHED) and asserts the fan-in join is excluded from next_nodes_to_execute while the dispatched sibling right is included. It pins the resume-frontier computation; on raw master it raises AttributeError on Status.DISPATCHED.
  • test_resume_runs_fan_in_once_for_dispatched_sibling: real-dispatch end-to-end reproduction — a fast-completing left entry point and a right entry point held in the dispatched window (dispatched via the real _execute_nodes_parallel path, blocked before _execute_node sets EXECUTING). The snapshot is captured from left's AfterNodeCallEvent, deserialized into a fresh graph, and resumed; join must run exactly once with both parents' outputs present.

• Both tests are red on master via distinct, valid signals: the end-to-end test fails with an assertion failure (join lands in next_nodes_to_execute → double-execution on resume, because right stays PENDING and is treated as a bypassed dead branch), while the unit test raises AttributeError on Status.DISPATCHED. Both are green on the branch.

• All 83 existing test_graph.py tests pass

• All 181 broader multiagent tests pass (excluding a2a which has a pre-existing missing-dependency issue)

• ruff check and ruff format --check pass

• mypy passes with no issues

• I ran hatch run prepare

Checklist

• I have read the CONTRIBUTING document
• I have reviewed and understand every line of code in this PR, including any generated by AI tools, and I can explain why it works
• My change is focused and reasonably small; I have split unrelated work into separate PRs
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[github-actions]

Assessment: Approve

Clean, minimal, and well-targeted fix. The DISPATCHED status precisely closes the dispatch→EXECUTING window in _compute_pending_sources, correctly completing the partial fix from #3390. I verified the resume logic: a DISPATCHED entry node is re-included in the frontier while its fan-in child is held back via pending_sources, and no other execution_status comparisons need updating. All 83 test_graph.py tests pass locally.

Review Notes

• Correctness: Logic is sound. Dead branches are never dispatched, so DISPATCHED seeding cannot resurrect a bypassed branch. Fail-fast/timeout paths that leave a node DISPATCHED correctly cause a re-run on resume, which is the desired semantics.
• Testing: Excellent — two tests that are red-on-master via distinct signals (AttributeError for the unit test, assertion failure for the end-to-end double-execution repro) and green on the branch. The real-dispatch reproduction with the blocking_right_dispatch wrapper is a thoughtful way to widen the one-tick race window deterministically.
• API surface: One consideration raised inline — Status is publicly exported, so this is an additive public-API change rather than purely internal (see inline comment).

Nicely written docstrings that explain the why behind the change — makes the resume-frontier reasoning easy to follow.

[SuperMarioYL]

Hi @liramon2 — same situation here as #3723: named checks green since the 10-15 rebase, but the integration-test jobs and Trigger Strands Review have been in waiting for 4 days. This one adds the DISPATCHED status so fan-in nodes cannot double-execute a retried dispatch (issue #3633). Happy to address any feedback.

[liramon2]

@SuperMarioYL Thanks for the contribution! I left some comments on this PR. The implementation looks sound, but the comments can be shortened.

[SuperMarioYL]

Thanks @liramon2! All four spots trimmed in edb9c80: the _execute_nodes_parallel comment and the _compute_pending_sources bullet in graph.py are down to the essential in-flight note, the TestResumeDispatchedSibling class and both test docstrings are one-liners now, and the blocking-wrapper comment keeps just the hold-until-capture intent. Both tests still pass.