Repository navigation
t18650: fix(pulse): async housekeeping still dies before preflight_ownership_reconcile, starving parent/merged-PR reconcile #34287
Description
Task
Async post-dispatch housekeeping still dies before its last stage, preflight_ownership_reconcile, in about a third of runs. That stage runs reconcile_issues_single_pass, which closes completed parent-task trackers, merged-PR issues and stale status:done issues. Those reconciles are therefore starved for whole cycles, even after GH#33256.
Full task brief and audit context
Task Brief
What
Async post-dispatch housekeeping still dies before its last stage, preflight_ownership_reconcile, in about a third of runs. That stage runs reconcile_issues_single_pass, which closes completed parent-task trackers, merged-PR issues and stale status:done issues. Those reconciles are therefore starved for whole cycles, even after GH#33256.
Evidence (local pulse.log, 2026-10-10, ~130k lines)
Async post-dispatch housekeeping: started= 66,... complete= 43,... reclaiming stale lock= 23._pulse_acquire_post_dispatch_housekeeping_lockreclaims only whenkill -0 <pid>fails, so 23 runs died. They were not displaced while alive.- Stage order in
_pulse_run_post_dispatch_housekeeping_stages(.agents/scripts/pulse-dispatch-engine.sh): coderabbit_review → post_merge_scanner → auto_decomposer_scanner (257s observed) → dedup_cleanup → fast_fail_prune_expired →_pulse_run_blocker_refresh_catchup(build_dependency_graph_cache 482s + refresh_blocked_status_from_graph 128s + release_repaired_brief_holds 407s observed) →preflight_ownership_reconcile(598s observed). - The run that started at log line 127226 reached
release_repaired_brief_holds(128131) and then disappeared. The next cycle loggedreclaiming stale lock (pid=49956)(128643). The new run then restarted at blocker catch-up.preflight_ownership_reconcilelast ran at line 126078, so its starvation spanned several cycles. - Concrete impact: after GH#34281 (PRs t18648: GH#34281 close approved external parent trackers when sub-issue children complete #34282/t18648: GH#34281 route stale-approved external parents past stage 0b to parent close #34284) was deployed, external parents Automate JavaScript worktree tooling readiness in the harness with durable project policy #34180/security: enforce local-only processing and stop-and-notify on failed model readiness #34125 waited for a housekeeping run that survived long enough to reach reconcile.
Root cause (confirmed)
Setup reconciliation kills the housekeeping on every deploy. setup.sh runs from auto-update after each new main commit and from interactive deploys. It calls pulse-lifecycle-helper.sh _stop_reconciliation_processes, which SIGTERMs every Pulse runtime process. The detached housekeeping subshell's argv is bash pulse-wrapper.sh, so it matches _PULSE_PATTERN whatever its process group. GH#33256's set -m only protects it from process-group teardown.
- 19 of 24 dead housekeeping PIDs in pulse.log appear in
Stopping pre-reconciliation Pulse process(es): ...lists in~/.aidevops/logs/auto-update.log/pulse-wrapper.log(for example 32467 and 49038). The other 5 coincide with interactivesetup.sh --non-interactivedeploys, whose output is not written to those logs. - launchd shows the pulse job's last exit as
-15(SIGTERM). With no TERM trap the wrapper's EXIT trap never runs, which also explains the repeatedStale mkdir lock detected (owner PID … is dead)lines in pulse-wrapper.log. - Stopping old-bundle processes on deploy is intended, because new code must replace old code. The defect is that housekeeping takes 30+ minutes and always restarts from its first stage. On a busy day with many deploys, its last stage (
preflight_ownership_reconcile) is starved indefinitely.
Files Scope
- .agents/scripts/pulse-dispatch-engine.sh
- .agents/scripts/tests/test-pulse-post-dispatch-housekeeping.sh
Fix direction
- Persist per-stage completion (for example, a small state file next to the housekeeping lock with
stage=epochlines, written after each stage finishes). A run that reclaims a stale lock should skip stages completed within a freshness window (for example, < 30 min) and resume at the first incomplete stage. A clean, complete run clears or refreshes the state. - Move
preflight_ownership_reconcileahead of_pulse_run_blocker_refresh_catchup, so the cheap terminal reconciles (parent close, merged-PR close) are not queued behind roughly 17 minutes of graph rebuild and brief-hold release. - Add a TERM/HUP trap in the housekeeping subshell that logs
Async post-dispatch housekeeping: terminated by <signal> during <stage>before exiting, so future deaths are attributable from pulse.log alone. - Do not exempt housekeeping from setup reconciliation; old-bundle code must still be replaced on deploy. Keep stage timeouts and the GH#33246 catch-up semantics intact.
Verification
bash .agents/scripts/tests/test-pulse-post-dispatch-housekeeping.sh(extend it to cover the stage-order/resume contract).shellcheck -xon modified files.- Runtime: after deploy, the ratio of
Async post-dispatch housekeeping: completetostartedin pulse.log approaches 1.Stage start: preflight_ownership_reconcileappears in every housekeeping run that survives past the scanners.
Brief workflow contract
Brief Workflow
This issue body is composed under .agents/workflows/brief.md. Newly queued auto-dispatch work must pass its Dispatch Readiness Contract (brief schema v2): complete write surface, hazards and compatibility, executable verification mapped to affected surfaces, and positive plus negative/regression acceptance criteria.
Synced from TODO.md by issue-sync-helper.sh
aidevops.sh v3.39.0 plugin for OpenCode v1.18.35 with claude-opus-5-5 spent 4h 42m and 448,179 tokens on this with the user in an interactive session.
Activity
alex-solovyev commentedon Oct 10, 2026
Self-Hosting Tier Override
Pre-dispatch self-hosting detector replaced lower workload-tier labels with
tier:thinkingon this issue.Matched pattern:
pulse-dispatch-in issue bodyRationale: Issues modifying the dispatch path have a self-referential property — workers dispatched to fix them run through the code being fixed. Applying the terminal workload tier upfront avoids wasted lower-tier attempts while runtime routing retains control of the exact model and reasoning level.
Bypass:
AIDEVOPS_SKIP_SELF_HOSTING_DETECTOR=1Automated by
pre-dispatch-validator-helper.sh(t2819). This comment is posted once via the<!-- self-hosting-tier-override -->marker; re-runs are no-ops.aidevops.sh v3.39.0 automated scan.
alex-solovyev commentedon Oct 10, 2026
DISPATCH_CLAIM nonce=596f1a12bee762886ce61ec6c2b7e186 runner=alex-solovyev ts=2026-10-10T22:42:40Z max_age_s=1800 version=3.39.1 opencode_version=1.18.34 lease_token=596f1a12bee762886ce61ec6c2b7e186 device=device-1783824075-2103358-6141 session=issue-34287 phase=prelaunch expires_at=1791672281
alex-solovyev commentedon Oct 10, 2026
DISPATCH_LEASE phase=prelaunch lease_token=596f1a12bee762886ce61ec6c2b7e186 device=device-1783824075-2103358-6141 session=issue-34287 expires_at=1791672623 ts=2026-10-10T22:42:53Z attempt_id=attempt:84b96e62-dd5c-40cb-851b-0035c9eae67e
alex-solovyev commentedon Oct 10, 2026
Dispatching worker (deterministic).
alex-solovyev commentedon Oct 10, 2026
DISPATCH_LEASE phase=ready lease_token=596f1a12bee762886ce61ec6c2b7e186 device=device-1783824075-2103358-6141 session=issue-34287 expires_at=1791679409 ts=2026-10-10T22:43:29Z attempt_id=attempt:84b96e62-dd5c-40cb-851b-0035c9eae67e
aidevops.sh v3.39.1 automated scan.
alex-solovyev commentedon Oct 10, 2026
CLAIM_RELEASED reason=worker_complete runner=alex-solovyev ts=2026-10-10T22:50:52Z aidevops_version=3.39.1 opencode_version=1.18.34
aidevops.sh v3.39.1 plugin for OpenCode v1.18.34
alex-solovyev commentedon Oct 10, 2026
DISPATCH_LEASE phase=terminal lease_token=596f1a12bee762886ce61ec6c2b7e186 device=device-1783824075-2103358-6141 session=issue-34287 expires_at=0 ts=2026-10-10T22:51:13Z attempt_id=attempt:84b96e62-dd5c-40cb-851b-0035c9eae67e
aidevops.sh v3.39.1 automated scan.
vladimirdulov commentedon Oct 10, 2026
Completion Summary
aidevops.sh v3.39.1 plugin for OpenCode v1.18.34 with claude-opus-5-5 spent 4m and 12,698 tokens on this as a headless worker.
Merged via PR #34291 to main.
Merged by deterministic merge pass (pulse-wrapper.sh).