Skip to content

fix(supervision): enforce owned pause proofs before suppressing wakes - #112

Open
ruby-dlee wants to merge 18 commits into
mainfrom
fm/pause-declared-but-never-registered
Open

fix(supervision): enforce owned pause proofs before suppressing wakes#112
ruby-dlee wants to merge 18 commits into
mainfrom
fm/pause-declared-but-never-registered

Conversation

@ruby-dlee

Copy link
Copy Markdown
Owner

What Changed

  • Require declared pauses to include a named owner and observable clearing condition, while keeping malformed, failure-bearing, or decision-blocked pauses actionable.
  • Make matching validation runs authoritative over pauses, preserve uncertain run/liveness lookups as unknown, and recheck measured progress before wedge escalation.
  • Register valid pause markers before returning signal wakes, reclaim timed-out watcher locks, and expand the related supervision documentation and regression coverage.

Risk Assessment

✅ Low: Captain, the bounded-capture fix correctly preserves normal exit codes, maps signaled children to nonzero failures, and keeps uncertain run discovery from authorizing pauses.

Testing

Completed 1 recorded test check.

  • Outcome: ⚠️ 1 error across 3 runs (11h12m37s)

Pipeline

Updates from git push no-mistakes

⏭️ **intent** - skipped

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 3 issues found → auto-fixed (4) ✅
  • 🚨 bin/fm-supervise-daemon.sh:402 - Away-mode pause handling bypasses the durable open-decision proof. After needs-decision [key=q], a valid paused: ...; owner=...; clears=... line is self-handled and registered solely through status_is_paused, allowing the unanswered decision to go quiet behind the pause cadence. Gate daemon pause registration/classification through crew_declared_pause_absorbable or its shared stream-proof boundary.
  • 🚨 bin/fm-crew-state.sh:900 - This assumes the pause is later than the run without establishing chronology. If a valid pause remains the last sparse status event, the wait clears, and the crewmate starts a run without another append, a subsequent done or parked run resurrects the stale pause and can hide a gate or PR-ready result. Record/require a resume transition or otherwise prove the pause postdates the run boundary before overriding run-step state.
  • ⚠️ bin/fm-watch-checkpoint.sh:94 - Forced-timeout cleanup can target a different state directory than the watcher: fm-watch.sh ignores ambient STATE, while the sourced wake library preserves it when FM_STATE_OVERRIDE is absent. With exported STATE plus FM_HOME, cleanup may report success after touching the wrong lock and leave the actual forced-kill lock behind. Resolve state using the watcher's exact precedence before sourcing/acquiring.

🔧 Fix: Captain: enforce pause proofs and watcher state precedence
1 error still open:

  • 🚨 bin/fm-classify-lib.sh:446 - Matching only run=<id> does not prove the pause postdates the current done/parked transition. Run R can be active or parked when the pause is appended, later resume, and then finish or park again without another status append; this check still accepts the stale pause because the run ID is unchanged, hiding the newer gate or PR-ready result. Bind pauses to a run transition/epoch or require a durable resume invalidation at this shared proof boundary.

🔧 Fix: Captain: keep matching run states authoritative over pauses
1 error still open:

  • 🚨 bin/fm-classify-lib.sh:562 - paused:status-log does not prove that no matching run exists. fm-crew-state.sh reaches this fallback when axi status returns empty or times out (lines 710-729), so a real done/parked run can transiently become paused; away-mode then self-handles and registers that pause, and later marker cleanup need not surface the run outcome. Preserve the run-query result and emit unknown unless absence of a matching run is affirmatively established before allowing this pause class.

🔧 Fix: Captain: make run discovery tri-state before pause fallback
1 error still open:

  • 🚨 bin/fm-crew-state.sh:231 - The Perl fallback maps signal termination to success because exit($? >> 8) discards the signal bits. If axi status or runs crashes or is killed before producing output, discovery becomes affirmatively absent and can again authorize a stale pause despite an existing run. Preserve signaled child status as nonzero at this shared capture boundary so uncertainty remains unknown.

🔧 Fix: Captain: preserve signaled child failures in bounded capture
✅ Re-checked - no issues remain.

⚠️ **Test** - 1 error
  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Captain: Stabilize lookup and watcher suppressor tests
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Captain: Stabilize pause suppressor observation under load
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"
✅ **Document** - passed

✅ No issues found.

🔧 **Lint** - 1 issue found → auto-fixed ✅
  • ⚠️ linter found issues (exit code 1)

🔧 Fix: Silence intentional negative-assertion stub lint warning
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant