fix(orchestration): ground whole-plan verify_plan() against DAG-wide tool trace - #6299
Merged
Merged
Conversation
bug-ops
enabled auto-merge (squash)
July 14, 2026 20:54
bug-ops
force-pushed
the
fix/6287-whole-plan-verify-grounding
branch
from
July 14, 2026 20:54
1747a95 to
a68c7cc
Compare
…tool trace PlanVerifier::verify_plan() (whole-plan completeness verification, run once after all DAG tasks complete) had no tool-call grounding. Only the per-task verify() path gained deterministic grounding against the real tool-execution trace in #6278/PR #6286, which explicitly scoped whole-plan out with a TODO(critic) marker. A hallucinated aggregated-output claim could pass whole-plan verification ungrounded. verify_plan() now grounds against the DAG-wide union of every completed task's real tool_trace, rebuilt from transcripts at whole-plan-verify time by reimplementing the resolution logic build_tool_trace_for_task uses for the per-task path (independent of whether per-task Verify ran for a given task -- deliberate defense-in-depth against a future dispatch mode that skips it). Trace availability is all-or-nothing at the DAG level: the aggregate is Some(union) only if every completed task's trace resolves; any one unavailable trace degrades the whole aggregate to None and grounding fails open, exactly reproducing prior ungrounded behavior. Whole-plan grounding is strictly weaker than per-task grounding at catching a single task's own hallucination -- it is additive defense-in-depth, not a replacement. The trace-path resolution loop is offloaded to spawn_blocking to avoid synchronous transcript reads blocking the async finalization path. Also fixes a pre-existing, previously-untested bug in execute_partial_replan_dag (whole-plan replan execution, unrelated to grounding) surfaced by the new end-to-end test: gap tasks from replan_from_plan carry IDs continuing the parent graph's numbering, but the standalone partial TaskGraph built to execute them requires 0-based positional IDs, so every whole-plan replan attempt against a non-empty graph was silently rejected by dag::validate and fail-opened to no replan. Gap-task IDs are now remapped to local 0-based IDs for the partial scheduler run and back to the original global IDs on the way out. specs/009-orchestration/spec.md updated with the new grounding contract, Key Invariant, and AC-13..AC-16. Closes #6287
bug-ops
force-pushed
the
fix/6287-whole-plan-verify-grounding
branch
from
July 14, 2026 20:55
a68c7cc to
337446b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PlanVerifier::verify_plan()(whole-plan completeness verification, run once after all DAG tasks complete) had no tool-call grounding. Only the per-taskverify()path gained deterministic grounding against the real tool-execution trace in #6278/PR #6286, which explicitly scoped whole-plan out with aTODO(critic)marker. A hallucinated aggregated-output claim (e.g. "ran the full test suite across all tasks") could pass whole-plan verification ungrounded.verify_plan()now grounds against the DAG-wide union of every completed task's realtool_trace, rebuilt from transcripts at whole-plan-verify time (independent of whether per-taskVerifyran — deliberate defense-in-depth against a future dispatch mode that skips it).Some(union)only if every completed task's trace resolves; any one unavailable trace (e.g. aRunInlinetask) degrades the whole aggregate toNoneand grounding fails open, exactly reproducing prior ungrounded behavior (now with a debug log noting the degradation).spawn_blockingto avoid synchronous transcript reads blocking the async finalization path.specs/009-orchestration/spec.mdupdated with the new grounding contract, Key Invariant, and AC-13..AC-16.Also fixes (unrelated, surfaced by the new end-to-end test)
execute_partial_replan_dag(pre-existing whole-plan replan execution code) built a standaloneTaskGraphfromreplan_from_plan's gap tasks and passed it toDagScheduler::new, which enforces 0-based positional task IDs. Butreplan_from_planassigns gap-task IDs continuing the parent graph's numbering — always>= 1in practice, since whole-plan verify only ever runs after tasks complete. Every whole-plan replan attempt was therefore silently rejected bydag::validateand fail-opened to no replan; the pipeline appears to have never actually worked prior to this fix. Gap-task IDs are now remapped to local 0-based IDs for the partial scheduler run and back to the original global IDs on the way out.Process
Built via
/rust-agents:team-develop(new-feature chain): architect → critic (design approved with changes, addressed) → developer → parallel validation (tester, perf, security, impl-critic — all approved, no blockers) → code review (changes requested: one wiring-gap test, one doc-accuracy fix) → developer fix pass (also surfaced and fixed theexecute_partial_replan_dagbug, confirmed with the user to keep bundled since narrowing the test would defeat its own purpose) → re-review (approved).Closes #6287
Test plan
cargo +nightly fmt --checkcargo clippy --profile ci --workspace --all-targets --features "desktop,ide,server,chat,pdf,scheduler,testing" -- -D warningscargo nextest run --config-file .github/nextest.toml --workspace --features "desktop,ide,server,chat,pdf,scheduler" --lib --bins— 13710 passed, 0 failedRUSTFLAGS="-D warnings" RUSTDOCFLAGS="--deny rustdoc::broken_intra_doc_links" cargo doc --no-deps --workspace --features "desktop,ide,server,chat,pdf,scheduler"gitleaks protect --staged --no-banner --redactqwen2.5:7b) confirming the newclaimed_executions/trace-section prompt schema round-trips without errorrun_whole_plan_verifydirectly (hallucinated-claim → replan triggered; honest-claim → no replan), proving the full union → verify_plan → grounding → should_replan → replan_from_plan → execute_partial_replan_dag pipeline is genuinely wiredSome(vec![])must never collapse toNone)/plan-CLI-level live testing (multi-task plan → whole-plan verify → grounding fires) — documented as Scenario G in.local/testing/playbooks/orchestration-verifier-grounding.md, left for a future CI live-testing cycle per project convention (CI sessions are separate from implementation sessions)