fix(supervision): enforce owned pause proofs before suppressing wakes - #112
Open
ruby-dlee wants to merge 18 commits into
Open
fix(supervision): enforce owned pause proofs before suppressing wakes#112ruby-dlee wants to merge 18 commits into
ruby-dlee wants to merge 18 commits into
Conversation
…liveness contract
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
Risk Assessment
✅ Low: Captain, the bounded-capture fix correctly preserves normal exit codes, maps signaled children to nonzero failures, and keeps uncertain run discovery from authorizing pauses.
Testing
Completed 1 recorded test check.
Pipeline
Updates from git push no-mistakes
⏭️ **intent** - skipped
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 3 issues found → auto-fixed (4) ✅
bin/fm-supervise-daemon.sh:402- Away-mode pause handling bypasses the durable open-decision proof. Afterneeds-decision [key=q], a validpaused: ...; owner=...; clears=...line is self-handled and registered solely throughstatus_is_paused, allowing the unanswered decision to go quiet behind the pause cadence. Gate daemon pause registration/classification throughcrew_declared_pause_absorbableor its shared stream-proof boundary.bin/fm-crew-state.sh:900- This assumes the pause is later than the run without establishing chronology. If a valid pause remains the last sparse status event, the wait clears, and the crewmate starts a run without another append, a subsequentdoneorparkedrun resurrects the stale pause and can hide a gate or PR-ready result. Record/require a resume transition or otherwise prove the pause postdates the run boundary before overriding run-step state.bin/fm-watch-checkpoint.sh:94- Forced-timeout cleanup can target a different state directory than the watcher:fm-watch.shignores ambientSTATE, while the sourced wake library preserves it whenFM_STATE_OVERRIDEis absent. With exportedSTATEplusFM_HOME, cleanup may report success after touching the wrong lock and leave the actual forced-kill lock behind. Resolve state using the watcher's exact precedence before sourcing/acquiring.🔧 Fix: Captain: enforce pause proofs and watcher state precedence
1 error still open:
bin/fm-classify-lib.sh:446- Matching onlyrun=<id>does not prove the pause postdates the current done/parked transition. Run R can be active or parked when the pause is appended, later resume, and then finish or park again without another status append; this check still accepts the stale pause because the run ID is unchanged, hiding the newer gate or PR-ready result. Bind pauses to a run transition/epoch or require a durable resume invalidation at this shared proof boundary.🔧 Fix: Captain: keep matching run states authoritative over pauses
1 error still open:
bin/fm-classify-lib.sh:562-paused:status-logdoes not prove that no matching run exists.fm-crew-state.shreaches this fallback whenaxi statusreturns empty or times out (lines 710-729), so a real done/parked run can transiently becomepaused; away-mode then self-handles and registers that pause, and later marker cleanup need not surface the run outcome. Preserve the run-query result and emitunknownunless absence of a matching run is affirmatively established before allowing this pause class.🔧 Fix: Captain: make run discovery tri-state before pause fallback
1 error still open:
bin/fm-crew-state.sh:231- The Perl fallback maps signal termination to success becauseexit($? >> 8)discards the signal bits. Ifaxi statusorrunscrashes or is killed before producing output, discovery becomes affirmatively absent and can again authorize a stale pause despite an existing run. Preserve signaled child status as nonzero at this shared capture boundary so uncertainty remainsunknown.🔧 Fix: Captain: preserve signaled child failures in bounded capture
✅ Re-checked - no issues remain.
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"🔧 Fix: Captain: Stabilize lookup and watcher suppressor tests
1 error still open:
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"🔧 Fix: Captain: Stabilize pause suppressor observation under load
1 error still open:
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"✅ **Document** - passed
✅ No issues found.
🔧 **Lint** - 1 issue found → auto-fixed ✅
🔧 Fix: Silence intentional negative-assertion stub lint warning
✅ Re-checked - no issues remain.
✅ **Push** - passed
✅ No issues found.