Skip to content

fix(daemon): suppress same-window refill double-injection and bound watcher stop - #5

Merged
jokim1 merged 9 commits into
lila-mainfrom
fm/fm-restore-green-main
Aug 18, 2026
Merged

fix(daemon): suppress same-window refill double-injection and bound watcher stop#5
jokim1 merged 9 commits into
lila-mainfrom
fm/fm-restore-green-main

Conversation

@jokim1

@jokim1 jokim1 commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Intent

Restore green CI on firstmate fork main (jokim1/firstmate) by fixing adopted-stack test regressions that block every firstmate PR. Captain ordered 2026-08-15 fix main. Named in-scope failures include afk-inject Scenario B duplicate U+2063 marker (real regression from cf26324/kunchenguid#2051; fix double-injection at source with regression assertion), pi-watch hung-successor, remote-job ready, and any other real adopted-stack regressions confirmed by local reproduction - fix code when behavior regressed, update tests only when expectation is stale against intentionally correct behavior; never weaken tests to hide regressions. Include fm-gotmp-fixture-lag only if not already on main. Shellcheck-clean; follow firstmate-coding-guidelines for shared tracked material. PR target origin jokim1/firstmate ONLY - never push/PR/touch upstream kunchenguid/firstmate. Full suite must pass locally and fork CI must go green; capture green CI run URL in PR body. Prior pipeline work on this branch (daemon refill suppression, watch-arm bounded stop, watcher-lock/fixture stabilizations through cb8cb58) must be preserved. Fix agent must NOT be codex (hangs on security-adjacent material); use non-codex agent (claude applied globally because grok is not a native no-mistakes agent and only codex/claude neutralize firstmate project instructions).

What Changed

  • Suppressed the same-window refill escalate in fm-supervise-daemon.sh so a status-driven capacity re-evaluation followed by a refill wake no longer double-injects the away-mode U+2063 marker; the refill is covered for one TTL-bounded shot while later refill-only wakes still escalate.
  • Bounded watcher stop in fm-watch-arm.sh (and wait_for_exit) by escalating TERM to KILL after a short wait, so signal handling and --restart cannot hang indefinitely on a child watcher whose EXIT cleanup blocks on recovery-marker locks; the arm re-arms fresh and reclaims any stale lock.
  • Stabilized the adopted-stack test suite across watcher, daemon, procevent, watcher-lock, and pi-watch coverage (deadline-based waits, bounded teardowns, fixture/lock synchronization) and added shellcheck source directives plus minor supporting adjustments in fm-watch-checkpoint.sh, fm-teardown.sh, fm-timeout-lib.sh, and fm-playbot-reconcile.mjs.

Risk Assessment

✅ Low: The change is well-bounded and overwhelmingly test-stabilization; the few source edits are narrow, defensive, intent-aligned, and covered by strengthened (not weakened) tests, with no upstream references introduced.

Testing

Ran the four named in-scope regression tests plus the slow bearings-snapshot test individually through the canonical bin/fm-test-run.sh runner; all passed (exit=0) with the afk-inject Scenario B duplicate-U+2063 assertion, pi-watch hung-successor fallback, remote-job ready bounds, and gotmp fixture all green. Inspected the test-file diffs to confirm the stabilizations replace fixed-count polling with wall-clock deadlines while keeping assertions intact rather than weakening them. The remaining changed-test batch (13 additional stabilized scripts, several multi-minute) was still running when I was required to return; nothing that executed failed. The changes are CLI/test-harness behavior with no rendered UI surface, so evidence is CLI transcripts (per-test ok - lines and FM_TEST_END exit=0 markers) rather than screenshots — appropriate for a shell toolbelt. Full local suite green and fork CI green-URL capture in the PR body remain the author's responsibility and were not verifiable in this isolated worktree.

Evidence: afk-inject + pi-watch named-regression transcript

ok - Scenario B: swallowed Enter produces exactly one clean digest ok - Pi hung successor falls back to one typed actionable wake ok - OpenCode hung successor falls back to one typed actionable wake FM_TEST_END tests/fm-afk-inject-e2e.test.sh exit=0 FM_TEST_END tests/fm-pi-watch-extension.test.sh exit=0

FM_TEST_BEGIN 2026-08-17T19:46:34Z tests/fm-afk-inject-e2e.test.sh family=afk expected_gate_skip=none
ok - Scenario A: partial input defers injection; digest arrives clean after idle
ok - Scenario B: swallowed Enter produces exactly one clean digest
ok - Scenario C: a normal captain status injects exactly one clean single-line sentinel digest
all e2e injection tests passed
FM_TEST_END 2026-08-17T19:47:10Z tests/fm-afk-inject-e2e.test.sh exit=0 duration_ms=36620 gate_skip=false
FM_TEST_BEGIN 2026-08-17T19:47:10Z tests/fm-pi-watch-extension.test.sh family=watcher-wake-lock expected_gate_skip=none
ok - Pi extension reports external healthy watcher output
ok - Pi custom tool exposes repair-only metadata and returns automatic-continuation guidance
ok - Pi redundant tool call returns ownership guidance and spawns no second child
ok - Pi scheduled retry remains extension-owned after another tool call
ok - Pi actionable close starts one successor before wake delivery settles
ok - Pi hung successor falls back to one typed actionable wake
ok - Pi unretired successor falls back without an overlapping retry
ok - Pi late unretired closes resume classified supervision
ok - Pi clean empty close triggers a bounded continuity retry
ok - Pi established clean closes stop at the configured retry limit
ok - Pi close handler verifies session-lock ownership before successor launch
ok - Pi watcher arm distinguishes all session lock ownership states
ok - Pi session transitions use a generation owner across /new /resume /fork, stale callbacks, and quit
ok - Pi process-exit cleanup listener remains singular across session replacement
ok - Pi process-exit cleanup stops the attached arm child
ok - OpenCode plugins have an explicit ESM boundary even under a typeless parent package
ok - OpenCode watcher plugin uses the effective FM_HOME state
ok - OpenCode watcher plugin sources the effective config
ok - OpenCode watcher plugin requires session lock ownership
ok - OpenCode watcher coordinator respects primary scope
ok - OpenCode watcher plugin starts one successor before wake prompt delivery settles
ok - OpenCode pre-ready actionable close preserves its successor
ok - OpenCode hung successor falls back to one typed actionable wake
ok - OpenCode unretired successor falls back without an overlapping retry
ok - OpenCode late unretired closes resume classified supervision
ok - OpenCode clean empty close triggers a bounded continuity retry
ok - OpenCode established clean closes stop at the configured retry limit
ok - OpenCode close handler verifies session-lock ownership before successor launch
ok - OpenCode watcher plugin coordinates with the turn-end guard
ok - OpenCode healthy arm output does not suppress the turn-end guard
FM_TEST_END 2026-08-17T19:47:51Z tests/fm-pi-watch-extension.test.sh exit=0 duration_ms=40222 gate_skip=false
FM_TEST_BEGIN 2026-08-17T19:47:51Z tests/fm-remote-job.test.sh family=secondmate expected_gate_skip=none
ok - default queue and execution bounds independently cover long polls
ok - operator PATH excludes a symlinked local bin
ok - operator PATH honors nvm defaults with a deterministic fallback
ok - operator PATH resolves the authorized Nix profile bin link
ok - the worker preserves bounded argv and stdin in an empty environment
ok - active jobs keep the worker ready for concurrent requests
ok - ensure replaces a live worker after its code changes
ok - worker identity binds the canonical configured code root
ok - stale ownership is reclaimed without signaling a reused pid
ok - the worker enforces the job timeout and publishes its result
ok - the worker expires queued jobs before they can mutate
ok - queued jobs receive a fresh bounded execution window
Evidence: remote-job + gotmp transcript

FM_TEST_END tests/fm-remote-job.test.sh exit=0 duration_ms=91095 gate_skip=false FM_TEST_END tests/fm-gotmp.test.sh exit=0 duration_ms=16439 gate_skip=false FM_TEST_SUMMARY total=2 failed=0 skipped_gate=0

FM_TEST_BEGIN 2026-08-17T19:48:48Z tests/fm-remote-job.test.sh family=secondmate expected_gate_skip=none
ok - default queue and execution bounds independently cover long polls
ok - operator PATH excludes a symlinked local bin
ok - operator PATH honors nvm defaults with a deterministic fallback
ok - operator PATH resolves the authorized Nix profile bin link
ok - the worker preserves bounded argv and stdin in an empty environment
ok - active jobs keep the worker ready for concurrent requests
ok - ensure replaces a live worker after its code changes
ok - worker identity binds the canonical configured code root
ok - stale ownership is reclaimed without signaling a reused pid
ok - the worker enforces the job timeout and publishes its result
ok - the worker expires queued jobs before they can mutate
ok - queued jobs receive a fresh bounded execution window
ok - a queued short command preempts a running long poll instead of waiting its window
ok - a poll re-armed after preemption reads the same cursor with nothing lost
ok - sibling polls never preempt each other into a re-arm churn loop
ok - worker shutdown terminates the active command tree before replacement
ok - Linux supervision recovers crashes and stops orphaned commands
ok - pre-execution validation obeys the job timeout
ok - the worker drains bounded output without changing command results
ok - the worker refuses symlinked job fields before command execution
ok - failed shutdown quarantines ownership against replacement workers
ok - quarantine clears only after recorded execution has stopped
ALL TESTS PASSED
FM_TEST_END 2026-08-17T19:50:19Z tests/fm-remote-job.test.sh exit=0 duration_ms=91095 gate_skip=false
FM_TEST_BEGIN 2026-08-17T19:50:19Z tests/fm-gotmp.test.sh family=session-bootstrap expected_gate_skip=none
ok - fm-teardown removes the dir pointed to by tasktmp= in meta
ok - fm-teardown skips gracefully when tasktmp= is absent (backward compat)
ok - fm-teardown skips gracefully when tasktmp= points to a nonexistent dir
FM_TEST_END 2026-08-17T19:50:36Z tests/fm-gotmp.test.sh exit=0 duration_ms=16439 gate_skip=false
FM_TEST_SUMMARY total=2 failed=0 skipped_gate=0 duration_ms=107935
FM_TEST_SUMMARY_FAMILY family=secondmate count=1 duration_ms=91095 failed=0
FM_TEST_SUMMARY_FAMILY family=session-bootstrap count=1 duration_ms=16439 failed=0
FM_TEST_SLOWEST rank=1 script=tests/fm-remote-job.test.sh duration_ms=91095
FM_TEST_SLOWEST rank=2 script=tests/fm-gotmp.test.sh duration_ms=16439

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

🔧 **Rebase** - 2 issues found → auto-fixed ✅
  • ⚠️ bin/fm-supervise-daemon.sh - merge conflict rebasing onto origin/lila-main
  • ⚠️ tests/fm-daemon.test.sh - merge conflict rebasing onto origin/lila-main

🔧 Fix applied.
✅ Re-checked - no issues remain.

⚠️ **Review** - 1 info
  • ℹ️ bin/fm-watch-arm.sh:438 - --restart behavior changed: a TERM-resistant but healthy watcher holding the watch lock is now force-killed via stop_pid_bounded (TERM → 5s wait → KILL) and the arm re-arms fresh, surfacing durable recovery; previously the restart attached to that live peer instead. The rewritten/renamed test test_watch_restart_bounds_term_resistant_healthy_peer codifies the new contract. This is the intended 'watch-arm bounded stop' work (commit 44e1132) and is intent-aligned, but it is a real semantics shift in a core supervisor path: a healthy-but-slow-to-TERM watcher can now be KILLed during an explicit restart, leaving a stale lock the fresh watcher must reclaim.
✅ **Test** - passed

✅ No issues found.

  • bin/fm-test-run.sh tests/fm-afk-inject-e2e.test.sh — Scenario A/B/C all ok; Scenario B swallowed-Enter produces exactly one clean U+2063 digest (exit=0)
  • bin/fm-test-run.sh tests/fm-pi-watch-extension.test.sh — Pi + OpenCode hung-successor fallback and 30 lifecycle assertions ok (exit=0)
  • bin/fm-test-run.sh tests/fm-remote-job.test.sh — remote-job ready/queue bounds ok (exit=0)
  • bin/fm-test-run.sh tests/fm-gotmp.test.sh — session-bootstrap fixture ok (exit=0)
  • bin/fm-test-run.sh tests/fm-bearings-snapshot.test.sh — snapshot-bearings ok (exit=0, 382s)
  • Reviewed git diff of tests/fm-pi-watch-extension.test.sh confirming assertions preserved under deadline-based waitFor
  • Launched remaining changed tests (calm-pi-extension, daemon, inactive-reconcile, pr-check-security, procevent, secondmate-safety, spawn-worktree-settle, watch-arm, watch-checkpoint, watch-triage, watcher-lock, herdr-lab, remote-secondmate-lifecycle-e2e) — still executing (calm-pi-extension in progress, no failures) when finalization was required
✅ **Document** - passed

✅ No issues found.

🔧 **Lint** - 1 issue found → auto-fixed ✅
  • ⚠️ linter found issues (exit code 1)

🔧 Fix: fm-watch-checkpoint.sh: add shellcheck source directives for sourced libs
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.

Green CI

Fork CI green: https://github.com/jokim1/firstmate/actions/runs/32068154155
Head: 036c6cdb27f3842f5f68b21963ab3d576501b784
Fixer: global agent: claude in ~/.no-mistakes/config.yaml (grok is not a native no-mistakes agent; firstmate disable_project_settings only neutralizes codex/claude).

@cursor

cursor Bot commented Aug 17, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

jokim1 added 9 commits August 17, 2026 13:31
Away-mode Scenario B double-injected U+2063 when a done/failed/blocked/
needs-decision status already forced capacity re-evaluation and a same-
window refill wake fired a second escalate. Cover the refill for one TTL-
bounded shot, keep later refill-only wakes escalating, and align gotmp
fixtures with teardown's remote-job lib sources.
…uite

Arm signal handling and --restart waited unboundedly on a child watcher
whose EXIT cleanup can block on recovery-marker locks. Escalate TERM to
KILL after a short bound, and make wait_for_exit do the same so bare
waits after intentional interrupts cannot hang the full portable suite.
…including replacement, ordering, and runner retirement
@jokim1
jokim1 force-pushed the fm/fm-restore-green-main branch from b20f281 to 036c6cd Compare August 17, 2026 20:52
@jokim1
jokim1 merged commit f6e599d into lila-main Aug 18, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant