fix(agent): replace hard agent deadline with pi-session-aware silence watchdog - #842
Conversation
… watchdog
The daemon killed healthy Pi agents as "silent 30m" because it used a
fixed per-context deadline while pi in --mode json buffers stdout when
piped, so a healthy turn showed zero stdout bytes. Forensics on run
01M0WRW741G2YPQFXYE5TG6DZD proved the fixer's session JSONL advanced
until 23s before SIGTERM. And one review-step budget spanned the fix
turn and the rereview turn, so run 01M0WF7T6DEKM614JT3DM4ZT2F's new
reviewer inherited ~20m of spent budget and died ~10 minutes into its
own turn.
Fold all liveness evidence into one monotonic last-activity clock per
invocation (internal/pipeline/liveness.go): stdout bytes from the shared
native-command pipe, native process lifecycle (a start re-arms the
clock, giving every new process a fresh full budget), and bound
pi-session JSONL advancement. The watchdog cancels with the whole
process tree only after the full configured budget passes with no
activity from any source, so the real timeout is preserved unchanged for
a genuinely frozen agent while a healthy long turn is never cut off.
The pi session binding (internal/agent/pi_liveness.go) is narrow and
fail-closed: resume binds only the one existing _<uuid>.jsonl in the
launched cwd's session dir (pi's encoding, resolved physical like pi's
process.cwd()); a fresh durable session binds only when exactly one new
file appears after launch and unbinds if the late stdout header names a
different id; --no-session turns, relocated session storage, ambiguity,
and unresolvable dirs all keep the conservative stdout/lifecycle-only
behavior.
Review and Test now pass their configured budgets per agent turn through
the seam (review_agent_timeout, test_agent_timeout) instead of
pre-installing one hard context, so every turn gets its own fresh
silence budget. Timeout diagnostics name per-kind last-activity ages
("pi session events 21.995s ago, process lifecycle 24s ago") with no
content or paths, and bind/unbind transitions land in the step log as
liveness notes. Managed-server adapters (opencode, rovodev) report no
activity and behave exactly as the legacy fixed deadline.
Docs: global-config.md owns the changed agent_timeout /
review_agent_timeout / test_agent_timeout semantics (silence budget, not
wall-clock cap; review is per-turn, no longer per-round).
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains. Reviews (2): Last reviewed commit: "fix(agent): order liveness evidence dete..." | Re-trigger Greptile |
|
Speaking as Kun's firstmate: Fork CI on HEAD VISION.md per-rule (from PR body + docs/config patches; full code review of liveness/pi binding continues as CI runs):
Not auto-merging even if CI goes green. This changes default product semantics of The expected |
evidence() sorted per-kind activity with an unstable sort keyed only on the observation timestamp, so two kinds stamped within one clock tick (the Windows CI timer granularity) rendered in Go map iteration order: the "most recent first" diagnostic contract was nondeterministic at exact ties, which is what failed TestLivenessEvidence_NamesEveryKindSeen on the windows-core leg with both ages rendering as 0s. Fold a per-invocation record sequence into the sort: most recent first, exact clock ties break in record order. Records are mutex-serialized, so the later record is the genuinely more recent activity and the ordering is a deterministic total order on every platform. Names, ages, and the no-content/no-paths diagnostic contract are unchanged, as are the watchdog budget, pi session binding, process-tree termination, per-invocation clock reset, and conservative fallback. Regression: TestLivenessEvidence_OrdersByRecencyAndBreaksClockTiesInRecordOrder constructs an exact tie through the recordAt seam (a coarse timer can no longer make the case vacuous) alongside a real ordering difference, and TestLivenessEvidence_NamesEveryKindSeen is now stable on every platform.
|
Speaking as Kun's firstmate: New HEAD Still not auto-merging even if they go green: default product semantics of |
Defect (incident-proven)
The daemon's agent timeout was a fixed 30-minute deadline whose diagnostic read "agent silent for 30m0s", but Pi in
--mode jsonbuffers stdout when piped, so a healthy turn shows zero stdout bytes. Two failure classes, both proven by session forensics (see the linked diagnosis below):01M0WRW741G2YPQFXYE5TG6DZD, fixer pid 55993: its pi session JSONL kept advancing until 23 seconds before SIGTERM, yet the wrapper killed it as "silent 30m".01M0WF7T6DEKM614JT3DM4ZT2F, reviewer pid 2574: the review step installed one budget at the fix turn's start, so the rereview launched 20 minutes later received only ~10 minutes before being killed.Fix
Agent timeouts are now silence budgets, not hard wall-clock caps. One monotonic last-activity clock per invocation (
internal/pipeline/liveness.go), fed by three evidence kinds (agent.ActivityKind):startNativeAgentCommandchoke point, covering every native adapter with no per-adapter logic change;internal/agent/pi_liveness.go).The watchdog terminates the whole process tree only after the full configured budget passes with no activity from any source, so a genuinely frozen agent is still killed on the same budget (
shellenvprocess-group kill path unchanged — verified in the journey below).The pi session binding is deliberately narrow and fail-closed:
--no-sessionturns persist nothing and stay stdout/lifecycle-only;_<uuid>.jsonlin the launched cwd's session dir (pi's--<cwd>--encoding, resolved physical like pi'sprocess.cwd();PI_CODING_AGENT_DIRhonored);--session-dir/PI_CODING_AGENT_SESSION_DIR), multiple matches, or an unresolvable dir all mean no binding — the invocation keeps the conservative stdout behavior instead of crediting unrelated activity. The watcher never reads session content and never credits broad filesystem changes.Review and Test now pass their configured budgets per agent turn through the shared seam (
RunAgentBudget/RunAgentSessionBudget) instead of pre-installing one hard context, so the fix turn and the rereview turn each get a fresh fullreview_agent_timeout. Nested seams (RunAgentoutside, the executortimeoutAgentbackstop inside) share one clock via a context-carried owner; a caller-supplied hard deadline is still honored unchanged.Diagnostics: timeout errors now name per-kind last-activity ages — e.g.
agent fix timed out after 20s (review agent silent for 20s: last activity: pi session events 21.995s ago, process lifecycle 24s ago)— with no prompt content, session content, or paths. Bind/unbind transitions also land in the step log aspi liveness:notes.Unchanged: managed-server adapters (opencode, rovodev) report no activity and behave exactly as the legacy fixed deadline; non-pi native adapters change only through the shared seam/pipe owner.
Test evidence
New committed coverage (all in
go test -race):internal/pipeline/liveness_test.go— watchdog re-arm/fire/parent-cancel, evidence rendering, nested-seam single owner, existing-deadline precedence.internal/agent/pi_liveness_test.go— session-dir encoding (verified against pi'sdist/core/session-manager.js), agent-dir resolution, resume binding, fresh-session unique-new-file binding, ambiguity → credit nothing, header-mismatch unbind, relocated storage → no binding, watcher shutdown, stdout/lifecycle pipe activity.internal/pipeline/agent_run_pi_test.go— process-level: a realpiAgentdriving a fake pi subprocess (tempPI_CODING_AGENT_DIR, no real~/.picontact): (a) quiet-stdout invocation with an advancing session survives >2× the budget and returns successfully; (b) frozen session is killed after the full budget with session evidence in the error; (c)--no-sessionquiet invocation is killed at budget with lifecycle-only evidence.TestReviewStep_EachAgentTurnGetsItsOwnFreshBudget— the rereview turn measurably gets its own full budget after a long fix turn (failure class 2 regression).Full-suite results on this machine:
go test -race ./...green except three pre-existing, environment-only failures that fail identically on the base commit (TestCIStep_CommitAndPush_CommitsLocallyWithoutPushing— local git hook injects a ticket prefix into commit subjects;TestCIWorkflow_WindowsHangSurfacesAsGoTimeoutNotJobCancellation+internal/e2edaemonTestReapAll_*— the documented macOS fork-pre-exec git flake / process-reaping environment).make lintgreen (skill unchanged, vet clean).gofmtclean.Tagged e2e (via
scripts/e2e.sh, fake claude/codex journeys through the modified seam):TestUserJourney,TestAxiAgentJourney,TestReviewPipelineOwnedPRCriterionDoesNotPark,TestAxiRunReattachesAfterManagedFix— all pass (240s).Built-binary journey (real daemon in temp
NM_HOME/HOME, fakepion PATH,agent_timeout/review_agent_timeout: 20s, no real~/.no-mistakesor~/.pitouched):review.logshowspi liveness: watching session 019ff2f3-…. Previously this was SIGTERM at exactly 30m/20s.agent fix timed out after 20s (review agent silent for 20s: last activity: pi session events 21.995s ago, process lifecycle 24s ago); zero leftover processes.User-visible semantics changes (owned in
docs/.../reference/global-config.md)agent_timeout,review_agent_timeout,test_agent_timeoutare now silence budgets (kill after this long with no activity) instead of hard wall-clock caps. A healthy long turn may now run past the configured value; a frozen one dies on the same value as before.review_agent_timeoutis now per agent turn (fix turn and rereview turn each get a fresh budget) instead of one shared per-round budget.Risks
Rollback
Revert this commit. All legacy behavior is restorable: the watchdog is confined to
internal/pipeline/liveness.go+agent_run.go, the pi watcher is additive ininternal/agent/pi_liveness.go, and adapter edits are one-line argument additions.Note on process: this PR was raised directly (not via the no-mistakes pipeline), so it deliberately carries no pipeline-attestation marker and the
no-mistakes / requiredcheck is expected to fail — the validator being repaired here cannot safely self-certify this fix. Diagnosis report:omp-deepseek-qwen-speed-quick-eval-20260825(firstmate scout, 2026-08-26), which code-level proved both failure classes against runs01M0WRW741G2YPQFXYE5TG6DZDand01M0WF7T6DEKM614JT3DM4ZT2F.