Description
Live-tested the newly merged deterministic ensemble-merge PlanVerifier (spec 073, #6232, commit
e55727a9) end-to-end via /plan <goal> → /plan confirm → /status (CI-1380, closing the
Scenario 12 gap left open by CI-1377). The ensemble mechanism itself works correctly (see coverage
update below), but the test surfaced a separate, pre-existing UX gap in zeph-core/src/agent/plan.rs:
when per-task or whole-plan verification determines the output is incomplete/low-confidence, and no
replan gap-tasks end up injected, the CLI user receives no indication whatsoever that anything
was flagged.
Both run_whole_plan_verify (plan.rs:491-578) and the per-task ensemble/verify branch
(scheduler_loop.rs:540-656) only ever call tracing::debug!/tracing::warn! on an incomplete
verdict or a failed/empty replan — neither path has a single self.channel.send(...) call for this
case. finalize_plan_execution (plan.rs:666-720) only branches user-visible messages on
GraphStatus (Completed/Failed/Paused/Canceled), which reflects task execution outcome, not
the verifier's completeness judgment — a plan whose only task technically "completed" (produced some
output) is reported to the user as an unqualified success even when verification confidently judged
that output wrong.
Reproduction Steps
- Enable
[orchestration] verify_completeness = true and a valid 3-member [orchestration.ensemble]
config (enabled = true, verify = true).
/plan <goal that a subagent will answer poorly> then /plan confirm.- In this session's repro, the assigned subagent (
rust-architect persona) attempted an unrelated
write tool call instead of directly answering, got blocked by the confirmation gate, and returned
"The write tool requires confirmation..." as its final task output.
- Debug log confirms both verifiers caught it:
ensemble per-task verification result task_id=0 complete=false confidence=0.283 agreement_ratio=1.0 and later whole-plan verification result complete=false confidence=0.0 gaps=1 threshold=0.7.
- Observe the CLI transcript: after "Confirmed. Executing plan (1 tasks)..." and the sub-agent's
non-answer, the only further output is from the separately-issued /status command. No
"plan verification found gaps" / "low confidence" / "replan attempted and failed" message is ever
shown. /status's "Ensemble verify:" section shows aggregate EMA scores per member but does not
flag that a specific task/plan was judged incomplete.
Expected Behavior
When verification (per-task or whole-plan) judges output incomplete/low-confidence and does not
result in a successful replan, the user should get some visible signal — even a low-key one line —
that the plan's output may not fully satisfy the goal, rather than silence that reads identically to
a plan verification never having run at all.
Actual Behavior
Silent. The only observable trace is in debug-level logs.
Environment
Logs / Evidence
crates/zeph-core/src/agent/plan.rs:544-577 (run_whole_plan_verify) — no channel.send on
incomplete verdictcrates/zeph-core/src/agent/scheduler_loop.rs:610-656 — same for the per-task pathcrates/zeph-core/src/agent/plan.rs:666-720 (finalize_plan_execution) — branches only on
GraphStatus, not verifier completeness- Live repro this cycle (CI-1380):
.local/testing/config/ensemble-live-test.toml,
.local/testing/journal/ci-1380.md
Coverage note (positive result, not a defect)
This same session confirmed the ensemble merge/quorum mechanism itself works correctly end-to-end for
the first time live: all 3 configured members (aliased to local Ollama qwen2.5:7b, since cloud
providers were quota-exhausted this session) were independently queried
(orchestration.ensemble.verify_member spans per member), merged with unanimous
agreement_ratio=1.0, and /status rendered the documented "Ensemble verify:" block with correct
per-member EMA scores. This closes the CI-1377 "Scenario 12" live-verification gap for #6247.
Description
Live-tested the newly merged deterministic ensemble-merge
PlanVerifier(spec 073, #6232, commite55727a9) end-to-end via/plan <goal>→/plan confirm→/status(CI-1380, closing theScenario 12 gap left open by CI-1377). The ensemble mechanism itself works correctly (see coverage
update below), but the test surfaced a separate, pre-existing UX gap in
zeph-core/src/agent/plan.rs:when per-task or whole-plan verification determines the output is incomplete/low-confidence, and no
replan gap-tasks end up injected, the CLI user receives no indication whatsoever that anything
was flagged.
Both
run_whole_plan_verify(plan.rs:491-578) and the per-task ensemble/verify branch(
scheduler_loop.rs:540-656) only ever calltracing::debug!/tracing::warn!on an incompleteverdict or a failed/empty replan — neither path has a single
self.channel.send(...)call for thiscase.
finalize_plan_execution(plan.rs:666-720) only branches user-visible messages onGraphStatus(Completed/Failed/Paused/Canceled), which reflects task execution outcome, notthe verifier's completeness judgment — a plan whose only task technically "completed" (produced some
output) is reported to the user as an unqualified success even when verification confidently judged
that output wrong.
Reproduction Steps
[orchestration] verify_completeness = trueand a valid 3-member[orchestration.ensemble]config (
enabled = true,verify = true)./plan <goal that a subagent will answer poorly>then/plan confirm.rust-architectpersona) attempted an unrelatedwritetool call instead of directly answering, got blocked by the confirmation gate, and returned"The
writetool requires confirmation..." as its final task output.ensemble per-task verification result task_id=0 complete=false confidence=0.283 agreement_ratio=1.0and laterwhole-plan verification result complete=false confidence=0.0 gaps=1 threshold=0.7.non-answer, the only further output is from the separately-issued
/statuscommand. No"plan verification found gaps" / "low confidence" / "replan attempted and failed" message is ever
shown.
/status's "Ensemble verify:" section shows aggregate EMA scores per member but does notflag that a specific task/plan was judged incomplete.
Expected Behavior
When verification (per-task or whole-plan) judges output incomplete/low-confidence and does not
result in a successful replan, the user should get some visible signal — even a low-key one line —
that the plan's output may not fully satisfy the goal, rather than silence that reads identically to
a plan verification never having run at all.
Actual Behavior
Silent. The only observable trace is in debug-level logs.
Environment
1fe1d0e2(main, 2026-07-14)zeph-core(agent/plan.rs,agent/scheduler_loop.rs)Logs / Evidence
crates/zeph-core/src/agent/plan.rs:544-577(run_whole_plan_verify) — nochannel.sendonincomplete verdict
crates/zeph-core/src/agent/scheduler_loop.rs:610-656— same for the per-task pathcrates/zeph-core/src/agent/plan.rs:666-720(finalize_plan_execution) — branches only onGraphStatus, not verifier completeness.local/testing/config/ensemble-live-test.toml,.local/testing/journal/ci-1380.mdCoverage note (positive result, not a defect)
This same session confirmed the ensemble merge/quorum mechanism itself works correctly end-to-end for
the first time live: all 3 configured members (aliased to local Ollama
qwen2.5:7b, since cloudproviders were quota-exhausted this session) were independently queried
(
orchestration.ensemble.verify_memberspans per member), merged with unanimousagreement_ratio=1.0, and/statusrendered the documented "Ensemble verify:" block with correctper-member EMA scores. This closes the CI-1377 "Scenario 12" live-verification gap for #6247.