fix: harden watcher supervision during slow reply polls

[jokim1]

Intent

Fix two coupled watcher-supervision reliability defects: promptly retire only resolved or terminal answered pending-reply records while never expiring genuinely unanswered records; keep the beacon fresh during bounded slow polls only while the watcher retains identity-matched singleton lock ownership; and attach Stop-hook re-arms to a live identity-matched stale-beacon holder without starting a competing cycle, while newly started watchers still require a fresh beacon. Preserve singleton lock, self-eviction, unresolved-record durability, and guard fail-open/fail-closed behavior. Include the large resolved-population regression and deliver PR 3 green. Also fix the observed CI blockers without weakening behavior: suppress duplicate away-mode injection when a capacity-freeing status and same-window refill share a batch, bound daemon watcher teardown, make normal-injection waits event-driven, and launch the remote-worker test supervisor in the production process-group shape.

What Changed

• Retire resolved pending replies from the active poll, quarantine records whose escalation cannot close, and preserve unresolved records.
• Keep watcher beacons fresh during slow reconciliation only for the identity-matched lock generation, and attach re-arms to proven live holders without starting competing watchers.
• Harden supervision lifecycle handling with duplicate-refill suppression, bounded teardown, safe stale-lock reclamation, and expanded regression coverage and documentation.

Risk Assessment

✅ Low: The shared health predicate now requires generation-bound beacon proof, while the arm-only stale-holder path remains distinct and avoids competing watcher startup.

Testing

After reviewing the prior-round failures and base-to-target diff, I reran the watcher, pending-reply, Stop-hook, AFK injection, remote-worker, lock, guard, and full remote-retirement workflows; all passed and produced CLI evidence, while PR inspection showed target b875caa remains unpublished and PR 3 is still red. No screenshot applies because this is shell/CLI supervision behavior with no rendered UI.

Evidence: Core watcher end-to-end transcript

FM_TEST_BEGIN 2026-08-16T13:14:29Z tests/fm-pending-reply.test.sh family=unclassified expected_gate_skip=none
ok - normal correlated reply resolves once (idempotent)
ok - completed turn with no report triggers exactly one recovery
ok - recovery attempts reconcile without reinjection
ok - recovery reply resolves the original expectation
ok - second missed turn escalates once and remains durable
ok - failed escalation publication remains retryable and publishes once
ok - legacy escalation closes under the shared default key
ok - legacy escalation cannot close an unrelated default-key decision
ok - foreign correlated blocker cannot impersonate a pending-reply escalation
ok - concurrent resolution closes one keyed escalation exactly once
ok - concurrent escalation yields to a late correlated reply
ok - transport success cannot masquerade as reply success
ok - undelivered records remain immutable across scan paths
ok - delivery confirmation fallback reconciles durably
ok - unrelated events and stale correlation ids cannot resolve
ok - restart preserves expectation and exact parent destination
ok - wrong-home reports are detected but do not silently acknowledge
ok - direct unmarked captain input creates no expectation
ok - fm-send marked secondmate path creates pending and embeds corr
ok - status-pointed document resolves the expectation
ok - optional helper report resolves without being required for correctness
ok - backend busy/idle observation covers Pi/Claude paths without conversation scrape
ok - tmux and zellij unknown states use bounded capture fallback
ok - pending replies scope Kimi capture fallback by recorded harness
ok - tick retires resolved records and reuses target observations
ok - correlations are reused only for matching open task records
ok - tick end-to-end: miss -> one recovery -> escalate -> durable
ok - failed transport discards undelivered expectation only
ok - large resolved population retires and keeps the beacon fresh
ok - single slow observation beats only while its watcher owns the lock
ok - tick retires a record that resolves mid-poll
ok - blank parent_status resolved+escalated quarantines out of the hot walk
/Users/josephkim/.no-mistakes/worktrees/7ab7ae90c971/01M04SJHWB675N8JWEACA1KHZJ/bin/fm-pending-reply-lib.sh: line 1078: /var/folders/vx/kq2w6_xj1sq82jh1gtsjl0_m0000gn/T//fm-pending-reply.iE0Uck/stuck-unwritable-5902/state/hibit.status: Permission denied
ok - unwritable status quarantines without closing the open fold decision
ok - 200 stuck empty-parent_status records leave the hot walk in one tick
ok - unresolved escalated never quarantines
ok - quarantine mv failure keeps the hot record
ok - all pending-reply tests passed
FM_TEST_END 2026-08-16T13:20:23Z tests/fm-pending-reply.test.sh exit=0 duration_ms=354179 gate_skip=false
FM_TEST_BEGIN 2026-08-16T13:20:23Z tests/fm-watch-arm.test.sh family=watcher-wake-lock expected_gate_skip=none
ok - watch-arm: an attached arm reports the wake its cycle delivered instead of a false failure
ok - watch-arm: attaches to a live identity-matched holder even when the beacon is stale
ok - watch-arm: a peer must emit its first beacon before attachment
ok - watch-arm: a newly started child requires a fresh beacon
ok - watch-arm: a delivered wake consumed by the handling turn still closes the attached arm cleanly
ok - watch-arm: a cycle that delivered no wake of its own still fails loudly
/Users/josephkim/.no-mistakes/worktrees/7ab7ae90c971/01M04SJHWB675N8JWEACA1KHZJ/bin/fm-watch-arm.sh: line 631: 84689 Killed: 9                  "$WATCH" > "$child_out"
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 2 --recovery-generation 93752.1786886483.WuMYpn
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  Queued wakes need handling, but no live watcher process holds this home lock (last beat: 3s ago).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  After draining queued wakes, repair missing watcher supervision according to the session-start block for this harness; do not use shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - watch-arm: re-arm surfaces every queued wake and an open remote decision after downtime
ok - watch-arm: marker publication failure retains stale-lock recovery evidence
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 3 --recovery-generation 24673.1786886556.G5aWR6
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  Queued wakes need handling, but no live watcher process holds this home lock (last beat: 8s ago).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  After draining queued wakes, repair missing watcher supervision according to the session-start block for this harness; do not use shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 4 --recovery-generation 44792.1786886567.YT7mPE
WARNING: watcher still down (same stale episode; last beat: 4s ago, grace 900s) - full banner already printed this episode.
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - watch-arm: a wake queued after handling drain is recovered once at successor arm
ok - watch-arm: interrupted handling leaves its wake durable for successor re-drain
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 0 --recovery-generation 34295.1786886614.rBfRHw
ok - watch-arm: malformed recovery state is quarantined without a successor loop
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 1 --recovery-generation 62309.1786886629.jJcQBv
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  Queued wakes need handling, but no live watcher process holds this home lock (last beat: 4s ago).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  After draining queued wakes, repair missing watcher supervision according to the session-start block for this harness; do not use shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - watch-arm: publication after recovery handoff is surfaced
ok - watch-arm: restart publishes recovery before clearing a reused-pid watcher lock
WAKE_ACK_REQUIRED: after handling completes run bin/fm-wake-drain.sh --ack-through 7 --recovery-generation 84556.1786886646.cPVjoF
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  Queued wakes need handling, but no live watcher process holds this home lock (last beat: 3s ago).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  After draining queued wakes, repair missing watcher supervision according to the session-start block for this harness; do not use shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - watch-arm: markerless legacy queues are adopted and recovered
ok - watch-arm: a watcher close during handling keeps the printed acknowledgement valid
ok - watch-arm: a moved recovery generation consumes handled rows and names its remedy
ok - watch-arm: downtime marker publication does not follow symlinks
FM_TEST_END 2026-08-16T13:25:13Z tests/fm-watch-arm.test.sh exit=0 duration_ms=289319 gate_skip=false
FM_TEST_BEGIN 2026-08-16T13:25:13Z tests/fm-claude-stop-autoarm.test.sh family=unclassified expected_gate_skip=none
ok - auto-arm: inert in a linked child worktree even when in-flight
ok - auto-arm: inert with no session lock
ok - auto-arm: a demonstrably dead recorded session owner is reclaimed through fm-lock.sh before arming
ok - auto-arm: inert without arm, rewake, or lock replacement when another live harness owns the home
ok - auto-arm: inert while AFK owns supervision
ok - auto-arm: stale-owner recovery leaves the AFK and supervision-need gates unchanged
ok - auto-arm: resolves the outermost pid of a nested contiguous claude ancestry (bg-spare chain)
ok - auto-arm: inert with nothing in flight and no X-mode need
ok - auto-arm: actionable close translates to exactly one exit-2 rewake with reason
ok - auto-arm: queue-only refill wakes are owned and delivered
ok - auto-arm: actionable close survives a healthy successor without duplicate delivery
ok - auto-arm: bounded failure verification emits one automatic-mechanism alarm
ok - auto-arm: consecutive failures keep Stop-owned retry without repeating notice
ok - auto-arm: unverified clean close exhausts retries and fails closed
ok - auto-arm: post-alarm actionable outcomes cannot continue or reset failure state
ok - auto-arm: benign cycle end with a live watcher and fresh beacon stays silent across the next cycle
ok - auto-arm: FAILED arm with live-but-stale holder remains fail-closed (arm attach owns mid-poll)
ok - auto-arm: budget contention preserves the episode and forces a reset retry
ok - auto-arm: X-mode poll need arms the cycle even with no tasks in flight
ok - auto-arm: concurrent firings admit one owner and one rewake translation
ok - auto-arm: need vanishing mid-cycle closes without a rewake
ok - auto-arm: mid-cycle AFK hands triage to the daemon with no rewake
ok - auto-arm: active in a marked secondmate home
ok - fm-lock: shared session-lock lib preserves the status path
FM_TEST_END 2026-08-16T13:26:18Z tests/fm-claude-stop-autoarm.test.sh exit=0 duration_ms=65273 gate_skip=false
FM_TEST_SUMMARY total=3 failed=0 skipped_gate=0 duration_ms=709342
FM_TEST_SUMMARY_FAMILY family=unclassified count=2 duration_ms=419452 failed=0
FM_TEST_SUMMARY_FAMILY family=watcher-wake-lock count=1 duration_ms=289319 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-pending-reply.test.sh duration_ms=354179
FM_TEST_SLOWEST rank=2 script=tests/fm-watch-arm.test.sh duration_ms=289319
FM_TEST_SLOWEST rank=3 script=tests/fm-claude-stop-autoarm.test.sh duration_ms=65273
fm-test-run: wrote timing artifact: /var/folders/vx/kq2w6_xj1sq82jh1gtsjl0_m0000gn/T/no-mistakes-evidence/01M04SJHWB675N8JWEACA1KHZJ/core-watcher-tests.json

Evidence: CI-blocker behavior transcript

FM_TEST_BEGIN 2026-08-16T13:26:42Z tests/fm-daemon.test.sh family=watcher-wake-lock expected_gate_skip=none
ok - fm-afk-start.sh fails before daemon startup when the afk flag cannot be written
ok - fm-afk-start.sh ignores stale pidfile-only live pids
ok - fm-afk-start.sh reclaims stale daemon locks whose live pid identity no longer matches
ok - supervise daemon state root is scoped by FM_HOME
ok - routine signal self-handles
ok - captain-relevant status verbs escalate
ok - check + unknown escalate; heartbeat self-handles
ok - transient stale self-handles and records a persistence marker
ok - enriched stale wedges bypass status absorption without disturbing busy workers
ok - stale + terminal status escalates immediately
ok - paused reasons with captain phrases remain pause-classified
ok - handle_wake on a paused stale records a pause marker, drops the wedge marker, and does not escalate
ok - handle_wake records a declared pause from a routine signal for long-cadence rechecks
ok - a terminal signal clears pause and stale tracking across both supervisors
ok - housekeeping migrates a normal-watcher's declared pause into daemon tracking
ok - housekeeping clears an already-resumed watcher pause across both supervisors
ok - housekeeping seeds pause tracking from status without a watcher marker
ok - persistent stale escalates after threshold and clears its marker
ok - resumed (busy) stale clears its marker without escalating
ok - housekeeping re-surfaces a stale declared pause on the long cadence and resets its window
ok - housekeeping clears a paused marker whose pane became busy again, without escalating
ok - housekeeping clears a paused marker once the crew is no longer declaring the pause
ok - housekeeping moves an existing stale marker to pause before wedge escalation
ok - housekeeping clears tracking when a crew leaves pause
ok - persistent herdr stale resolves the target from metadata and escalates
ok - herdr idle busy-footer stale clears through capture corroboration
ok - resumed herdr stale clears through backend-aware busy state
ok - persistent Orca stale resolves the terminal from metadata
ok - multiple escalations flush as a single batched digest
ok - batch flush measures max-delay from the first append, not the last
ok - catch-all scan escalates a missed terminal once, not twice
ok - handle_wake routes routine->self and captain->escalate
ok - INJECT_SKIP forces self-handle, bypassing captain-relevant classification
ok - is_wake_reason distinguishes watcher wake reasons from singleton-status stdout
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  Queued wakes need handling, but no watcher has a fresh beacon (last beat: never, grace 900s).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  After draining queued wakes, repair missing watcher supervision according to the session-start block for this harness; do not use shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - durable mixed wake batches handle refill before acknowledgement
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  Queued wakes need handling, but no watcher has a fresh beacon (last beat: never, grace 900s).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  After draining queued wakes, repair missing watcher supervision according to the session-start block for this harness; do not use shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - capacity-freeing captain status covers same-window refill without dropping later refill-only wakes
ok - terminal-stale escalate removes its marker so housekeeping does not re-escalate
ok - captain signal escalate marks seen so the catch-all scan does not re-fire
ok - _collapse_newlines replaces newlines with literal separator
ok - afk flag absent: daemon does not inject, buffer preserved
ok - busy-guard defers injection when supervisor pane is busy
ok - marker detection: marker -> stay afk, no marker -> exit afk
ok - /afk invocation is exempt from afk exit (no self-cancel)
ok - should_exit_afk returns false when afk is not active
ok - strip_injection_marker removes the sentinel marker cleanly
ok - pane_input_pending detects partial input on the cursor line
ok - pane_input_pending: a blank unidentified cursor row defers (strict container-proof rule)
ok - pane_input_pending: only proven empty agent prompts pass
ok - fm_tmux_composer_state: a bare shell prompt ($/%/#/>) reads unknown, never empty (dead-shell injection safety)
ok - fm_tmux_composer_state: a bordered composer box and bare agent glyphs (❯/›) still read empty
ok - fm_tmux_composer_state: only matching edge borders form a composer box
ok - pane_input_pending preserves bright placeholder-like drafts in styled captures
ok - classify_signal dedupes against the catch-all scan seen marker
ok - classify_stale dedupes against the signal path seen marker
ok - AFK nonterminal working:+merged keeps wedge aging and re-escalates at bound
ok - genuine done: and merge-check events still escalate
ok - pane_input_pending: an idle bordered composer is NOT pending (afk-invx-i5)
ok - pane_input_pending: text inside a bordered composer is still pending
ok - submit-ACK confirms a submit when the composer returns to a bordered-empty box
ok - submit-ACK reports pending on a persistently swallowed Enter (type-once)
ok - max-defer on an empty stuck pane types once, alarms, and preserves the buffer
ok - max-defer flushes and clears the buffer on an empty bordered pane
ok - max-defer on a pending composer alarms without typing
ok - normal flush clears a stale wedge marker
ok - below MAX_DEFER: no inject, no alarm, buffer preserved
ok - max-defer does not flush or alarm while afk is inactive
ok - library mode: sourcing the daemon defaults FM_WEDGE_ALARM_EXEC to discard (no test can fire a real notification)
ok - wake helpers replace inherited notifier overrides with the safe recorder
ok - the discard seam suppresses every notifier, including command: (fires nothing)
ok - direct notifier helpers honor the discard seam, including command:
ok - osascript channel routes through the notifier seam with the summary (never a real notification)
ok - herdr channel routes through the notifier seam with the summary (never a real notification)
ok - command channel runs the captain command with the summary on $1 and on stdin
ok - command channel failures redact configured commands while logging their exit status
ok - unknown channel directives are redacted while the alarm keeps running
ok - off disables every active alert regardless of directive position (marker and tmux flash are unaffected)
ok - auto resolves to the macOS osascript notifier on Darwin (default-on)
ok - auto on a non-macOS platform selects no built-in OS channel (the marker or a configured command carries it)
ok - config/wedge-alarm selects every configured channel and skips comment and blank lines
ok - a failing channel logs and falls back to the next channel, never crashing the alarm
ok - a hung notifier is bounded, logged, and falls throug

... [6764 bytes truncated] ...

upervision warning only; the guarded operation WILL still run.
●  After draining queued wakes, repair missing watcher supervision according to the session-start block for this harness; do not use shell &.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - arm propagates an immediate watcher wake before confirmation
ok - arm attaches to a peer watcher after child stands down and surfaces a missing successor
watcher: lock held by live pid 12982 but heartbeat is stale for 840173588s (>300s); inspect or stop that watcher before re-arming.
ok - arm reports FAILED and exits non-zero when no fresh watcher can be confirmed
ok - cycle-exit ledger links a verified successor and remains size-capped
ok - SIGSTOP distinguishes live PID from stale beacon and termination records the exit class
FM_TEST_END 2026-08-16T13:34:14Z tests/fm-watcher-lock.test.sh exit=0 duration_ms=184452 gate_skip=false
FM_TEST_BEGIN 2026-08-16T13:34:14Z tests/fm-turnend-guard.test.sh family=watcher-wake-lock expected_gate_skip=none
ok - fm_supervision_unhealthy: false with no state/*.meta at all
ok - fm_supervision_unhealthy: true with in-flight task and no beacon ever
ok - fm_supervision_unhealthy: true with in-flight task and a beacon far outside the grace window
ok - fm_supervision_unhealthy: false with in-flight task and a fresh beacon
ok - fm_supervision_status: a pending wake queue requires supervision
ok - fm_supervision_needed: X-mode relay poll needs supervision
ok - fm_supervision_unhealthy: source-only home needs supervision
ok - fm-turnend-guard: silent no-op with nothing in flight
ok - fm-turnend-guard: blocks when a fresh beacon has no live watcher lock
ok - fm-turnend-guard: non-Claude path blocks a source-only home
ok - fm-turnend-guard: queue-only homes report queued wake supervision
ok - fm-turnend-guard: blocks on a dead watcher lock even when the beacon is fresh
ok - fm-turnend-guard: silent no-op with a live watcher lock and fresh beacon
ok - fm-turnend-guard: fresh leftover beacon cannot validate a new lock generation
ok - fm-turnend-guard: healthy non-Claude harness paths ignore Claude episode contention
ok - fm-turnend-guard: blocks on a live watcher lock with an ancient beacon
ok - fm-turnend-guard: blocks with the exact required reason in the primary when unhealthy
ok - fm-turnend-guard: blocks from active FM_HOME state, not only repo-root state
ok - fm-turnend-guard: X-mode repair reason sources the cadence config
ok - fm-turnend-guard: X-mode-only supervision remains guarded in default mode
ok - fm-turnend-guard: ignores stale repo-root state when FM_HOME is set
ok - fm-turnend-guard: uses FM_STATE_OVERRIDE ahead of FM_HOME/state
ok - fm-turnend-guard: stop_hook_active=true always allows the stop (never blocks twice in one turn)
ok - fm-turnend-guard: blocks a blind turn end in a secondmate's own home (.fm-secondmate-home no longer excludes it)
ok - fm-turnend-guard: idle-by-default - silent in a secondmate home with nothing in flight
ok - fm-turnend-guard: stop_hook_active=true allows the stop in a secondmate home (never blocks twice in one turn)
ok - fm-turnend-guard: secondmate deferred-death recovery - silent while watched, forces re-arm once the watcher exits
ok - fm-turnend-guard: inert in a secondmate's own child worktree (linked git worktree) even when unhealthy
ok - fm-turnend-guard: blocks a blind turn end in a treehouse-leased LINKED secondmate home (marker force-include)
ok - fm-turnend-guard: an invalid (empty) marker cannot spoof inclusion; linked worktree stays exempt
ok - fm-turnend-guard: a non-ASCII marker cannot spoof inclusion; linked worktree stays exempt
ok - fm-turnend-guard: inert in a crewmate/scout task worktree (linked git worktree) even when unhealthy
ok - fm-turnend-guard: fails open (never blocks) when jq is missing
ok - fm-turnend-guard: silent no-op on empty stdin
ok - fm-turnend-guard: runs well under the generous timing margin (1s)
ok - fm-turnend-guard-grok: forces one explicitly marked same-session resume when the shared predicate blocks
ok - fm-turnend-guard-grok: legacy environment loop guard prevents a nested resume loop
ok - fm-turnend-guard-grok: native false delegates blocking feedback with zero resume processes
ok - fm-turnend-guard-grok: native true remains bounded and starts no resume process
ok - fm-turnend-guard-grok: both spellings are typed and camelCase has deterministic precedence
ok - fm-turnend-guard-grok: malformed, invalidly typed, and missing-prerequisite payloads start neither path
ok - fm-turnend-guard-grok: missing jq and no-supervision-needed stops stay silent and bounded
ok - tracked .claude/settings.json entries: 6 inert under grok, the documented subagent exception still armed, all live under Claude
ok - .codex/hooks.json: Stop hook uses hook process root when payload cwd is outside
ok - .codex/hooks.json: Stop hook ignores nested git root guard scripts
ok - .opencode primary plugin: guard path is anchored to worktree, not directory
ok - .pi primary extension: no-tool and multi-tool runs each inject exactly one guard follow-up
ok - .pi primary extension: delivery failure resets the logical-run latch
ok - fm-turnend-guard --claude: re-blocks a loop-guarded stop while unhealthy and unclaimed (incident regression)
ok - fm-turnend-guard --claude: X-mode-only homes re-block when auto-arm recovery is absent
ok - fm-turnend-guard --claude: a live arming epoch advances once and repeated observation is idempotent
ok - fm-turnend-guard --claude: repeated failed-to-arming races make bounded monotonic progress
ok - fm-turnend-guard --claude: terminal owner boundary excludes a concurrent start without deadlock
ok - fm-turnend-guard --claude: fresh rewake epoch prevents a duplicate continuation for the same event
ok - fm-turnend-guard --claude: fresh failed epochs preserve and advance monotonic fail-open progression
ok - fm-turnend-guard --claude: integrated fresh failures reach one bounded fail-open, stop continuation, and reset on recovery
ok - fm-turnend-guard --claude: reset contention preserves all episode state until retry
ok - fm-turnend-guard --claude: concurrent auto-arm and guard resets are idempotent and deadlock-free
ok - fm-turnend-guard --claude: stale rewake epoch does not allow a blind stop
ok - fm-turnend-guard --claude: budget exhaustion alone cannot permit a blind stop
ok - fm-turnend-guard --claude: verified fail-open is loud, bounded, attended, and non-repeating
ok - fm-turnend-guard --claude: queue-only fail-open identifies queued wakes
ok - fm-turnend-guard --claude: fail-open requires both exhausted retries and consumed notice
ok - fm-turnend-guard --claude: away ownership excludes the Stop-autoarm fail-open
ok - fm-turnend-guard --claude: positive watcher recovery resets failure episode state
ok - fm-turnend-guard --claude: bounded claim wait avoids a token-consuming forced continuation
ok - fm-turnend-guard --claude: secondmate home re-blocks unclaimed and allows auto-arm-claimed stops
FM_TEST_END 2026-08-16T13:36:53Z tests/fm-turnend-guard.test.sh exit=0 duration_ms=158996 gate_skip=false
FM_TEST_SUMMARY total=5 failed=0 skipped_gate=0 duration_ms=611110
FM_TEST_SUMMARY_FAMILY family=afk count=1 duration_ms=40483 failed=0
FM_TEST_SUMMARY_FAMILY family=secondmate count=1 duration_ms=140301 failed=0
FM_TEST_SUMMARY_FAMILY family=watcher-wake-lock count=3 duration_ms=429234 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-watcher-lock.test.sh duration_ms=184452
FM_TEST_SLOWEST rank=2 script=tests/fm-turnend-guard.test.sh duration_ms=158996
FM_TEST_SLOWEST rank=3 script=tests/fm-remote-job.test.sh duration_ms=140301
FM_TEST_SLOWEST rank=4 script=tests/fm-daemon.test.sh duration_ms=85786
FM_TEST_SLOWEST rank=5 script=tests/fm-afk-inject-e2e.test.sh duration_ms=40483
fm-test-run: wrote timing artifact: /var/folders/vx/kq2w6_xj1sq82jh1gtsjl0_m0000gn/T/no-mistakes-evidence/01M04SJHWB675N8JWEACA1KHZJ/ci-blocker-tests.json

Evidence: Remote lifecycle end-to-end transcript

FM_TEST_BEGIN 2026-08-16T13:37:33Z tests/fm-remote-secondmate-lifecycle-e2e.test.sh family=secondmate expected_gate_skip=none
ok - overlapping remote home provisioning serializes through publication and rollback
ok - remote seed rollback preserves serialized competing routes
ok - unknown readiness preserves its route and brief for reconciliation
ok - remote seeding checks, repairs, and re-checks readiness, then stops on a remaining gap
ok - remote seeding proceeds once the repair closes every gap
ok - remote seeding provisions a supplied origin without touching the primary project tree
ok - remote provisioning re-validates a supplied origin at the receiving host
ok - seeding carries bitbucket, self-hosted, and scp-like origins through to the remote clone
ok - remote seed registers the route and provisions the whole home and project clone on that host
ok - remote inheritance rejects incomplete and superseded payload generations
ok - mixed local and remote routes validate without migration
ok - remote spawn launches on the remote-local backend and records a host-qualified route
ok - legacy and mismatched remote endpoints fail closed before backend access
ok - non-herdr remote endpoints are refused without changing either route
ok - remote spawn serializes inheritance through launch publication
ok - marked send and routed reply complete through the existing parent correlation owner
PR_CHECK_MIGRATION: watcher ownership is ambiguous; review state/.watch.lock before rearming polls
ok - partial remote inheritance retains reread intent through bootstrap convergence
ok - config push and bootstrap serialize remote inheritance convergence
WARNING: watcher still down (same stale episode; last beat: 299s ago, grace 900s) - full banner already printed this episode.
WARNING: queued wakes pending - drain them with bin/fm-wake-drain.sh before anything else.
ok - remote inherited config retains and retries a failed live reread nudge
ok - fleet snapshot projects mixed local and remote structured state
ok - remote update imports and fast-forwards the persistent home on its configured host
PR_CHECK_MIGRATION: watcher ownership is ambiguous; review state/.watch.lock before rearming polls
ok - startup repairs remote readiness before probing without relaunching
PR_CHECK_MIGRATION: watcher ownership is ambiguous; review state/.watch.lock before rearming polls
ok - startup reports alive legacy backends without changing their routes
ok - unreachable remote state remains unknown with no local respawn or failover
ok - remote retirement refuses child work, then removes only its own endpoint while a shared-session sibling survives
ALL TESTS PASSED
FM_TEST_END 2026-08-16T14:00:44Z tests/fm-remote-secondmate-lifecycle-e2e.test.sh exit=0 duration_ms=1390438 gate_skip=false
FM_TEST_SUMMARY total=1 failed=0 skipped_gate=0 duration_ms=1390661
FM_TEST_SUMMARY_FAMILY family=secondmate count=1 duration_ms=1390438 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-remote-secondmate-lifecycle-e2e.test.sh duration_ms=1390438
fm-test-run: wrote timing artifact: /var/folders/vx/kq2w6_xj1sq82jh1gtsjl0_m0000gn/T/no-mistakes-evidence/01M04SJHWB675N8JWEACA1KHZJ/remote-lifecycle-test.json

Evidence: PR 3 publication and CI status

Source: PR 3 publication and CI status

PR 3: 9 passed, 4 failed. Local target: b875caafa99da9db3b5965d2c61c9a99adc2e359. Published PR head: 4278da996e3e468dfc56020baf369aa7adf04d86.

- Outcome: ⚠️ 1 warning across 3 runs (5h23m58s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (3) ✅

• 🚨 bin/fm-watch-arm.sh:477 - Intent requires “attach Stop-hook re-arms to a live identity-matched stale-beacon holder without starting a competing cycle,” but this hunk explicitly starts the usual watcher child for that case. The child can exit nonzero on fm-watch.sh's stale-beacon lock check; lines 638-643 then report failure before the live-holder attach at line 650, recreating the false watcher-down race. Attach to the verified stale holder before spawning while retaining fresh-beacon confirmation for genuinely new watchers.

🔧 Fix: Attach stale-beacon watcher holders before spawning
1 error still open:

• 🚨 bin/fm-watch-arm.sh:481 - Intent requires stale-holder attachment while “newly started watchers still require a fresh beacon,” but live_watcher_holder verifies only PID/lock identity and this gate attaches even when .last-watcher-beat is absent. A peer watcher can publish its lock identity before its first beacon (fm-watch.sh:798-803 versus :856), then be reported as attached and followed indefinitely if startup wedges. Require proof that the current lock generation previously emitted a beacon before using the stale-holder path; otherwise retain bounded fresh-beacon confirmation.

🔧 Fix: Bind watcher attachment to generation beacon proof
1 error still open:

• 🚨 bin/fm-watch-arm.sh:258 - Intent requires “newly started watchers still require a fresh beacon” and preserved guard fail-closed behavior, but generation proof is enforced only by this local wrapper. fm-claude-stop-autoarm.sh:191 and fm-turnend-guard.sh:149,244,312,346,355 still call fm_watcher_healthy, which accepts a new live lock with a fresh leftover global beat without checking beacon-identity. An unbeaconed startup can therefore be suppressed as healthy/clean. Enforce generation proof in the shared health predicate or update every guard consumer while keeping stale-holder attachment distinct.

🔧 Fix: Enforce generation-bound health across watcher guards
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 warning

• 🚨 tests/fm-turnend-guard.test.sh:213 - The new guard regression aborts under set -u: record_watcher_lock expands $identity in the same local statement that assigns it. Split the declarations so identity is assigned first.
• 🚨 tests/fm-remote-job.test.sh:479 - The remote-worker suite failed twice at different points: an orphaned command mutated after crash recovery, then a rerun reported queue delay consuming the next job's execution timeout. The production-shape fixture is not reliably green.
• 🚨 tests/fm-watch-arm.test.sh:373 - The watcher-arm suite hung for over four minutes in the rearm-resurface fixture after intentional teardown, leaving the arm wrapper and watcher child alive until the test session was interrupted.
• ⚠️ PR 3 remains red, and its latest CI run tested head 4278da9 rather than supplied target fcc80cb. Green-PR evidence requires publishing a corrected target and running CI against it.
• git diff --stat 5b9e8629923a0b7623c82d5e151490e8185f59f7..fcc80cb17171c46af999bd1558ef76bb2f6d3b64
• tests/fm-pending-reply.test.sh
• tests/fm-watch-arm.test.sh - interrupted after the rearm-resurface hang
• tests/fm-daemon.test.sh
• tests/fm-afk-inject-e2e.test.sh
• tests/fm-remote-job.test.sh - run twice
• tests/fm-turnend-guard.test.sh
• tests/fm-claude-stop-autoarm.test.sh
• gh-axi pr checks 3
• gh-axi run view 31934901693 --log-failed
• git status --short and process checks for transient workspace artifacts

🔧 Fix: Bound watcher teardown and stabilize supervision tests
2 issues (1 error, 1 warning) still open:

• 🚨 bin/fm-wake-lib.sh:762 - Safe remote retirement reproducibly fails because fm_lock_try_acquire recursively acquires $lockdir.steal. When that mutex is stale, each recursion appends another .steal until paths exceed the filesystem limit. Acquire or reclaim the steal mutex without recursively calling fm_lock_try_acquire, while preserving refusal for live steal owners.
• ⚠️ PR 3 is not green and has no CI run for supplied target 6f23745. Its latest run tested older remote head 4278da9 and failed four jobs, so publishing the corrected target and running CI are required to satisfy the green-PR acceptance criterion.
• git status --short --branch and git rev-parse HEAD
• git diff --stat 5b9e8629923a0b7623c82d5e151490e8185f59f7..6f23745211432df7495561a8e01fdbd6d961b590
• /opt/homebrew/bin/gh-axi pr view 3 and /opt/homebrew/bin/gh-axi pr checks 3
• /opt/homebrew/bin/gh-axi run list --branch fm/fm-pending-reply-leak-slow-watch --limit 10 --fields headSha,number,updatedAt,url
• /opt/homebrew/bin/gh-axi run view 31934901693 --conclusion failure --log-failed
• bin/fm-test-run.sh tests/fm-pending-reply.test.sh tests/fm-watch-arm.test.sh tests/fm-claude-stop-autoarm.test.sh tests/fm-turnend-guard.test.sh tests/fm-watcher-lock.test.sh tests/fm-daemon.test.sh tests/fm-afk-inject-e2e.test.sh tests/fm-pi-watch-extension.test.sh tests/fm-remote-job.test.sh
• bin/fm-test-run.sh tests/fm-afk-inject-herdr-e2e.test.sh tests/fm-remote-secondmate-lifecycle-e2e.test.sh
• FM_SNAPSHOT_SECONDMATE_TIMEOUT=30 bin/fm-test-run.sh tests/fm-remote-secondmate-lifecycle-e2e.test.sh
• Final git status --short and transient-directory check

🔧 Fix: Fix stale lock reclamation and remote teardown completion
1 warning still open:

• ⚠️ The required green PR cannot be demonstrated: PR 3 still has four failing checks because its published head is 4278da9, not validated target b875caa. Publish the target through the authorized no-mistakes/PR workflow and run CI against it.
• git diff --stat 5b9e8629923a0b7623c82d5e151490e8185f59f7..b875caafa99da9db3b5965d2c61c9a99adc2e359 and intent-linked source/test inspection
• bin/fm-test-run.sh --json .../core-watcher-tests.json tests/fm-pending-reply.test.sh tests/fm-watch-arm.test.sh tests/fm-claude-stop-autoarm.test.sh
• bin/fm-test-run.sh --json .../ci-blocker-tests.json tests/fm-daemon.test.sh tests/fm-afk-inject-e2e.test.sh tests/fm-remote-job.test.sh tests/fm-watcher-lock.test.sh tests/fm-turnend-guard.test.sh
• bin/fm-test-run.sh --json .../remote-lifecycle-test.json tests/fm-remote-secondmate-lifecycle-e2e.test.sh
• gh-axi pr checks 3
• gh-axi run list --branch fm/fm-pending-reply-leak-slow-watch
• git ls-remote origin refs/heads/fm/fm-pending-reply-leak-slow-watch
• git status --short and transient-output inspection

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.