Skip to content

fix: suppress false wedges for held green PRs - #116

Open
ruby-dlee wants to merge 3 commits into
mainfrom
fm/pause-vs-orphan-run-p6
Open

fix: suppress false wedges for held green PRs#116
ruby-dlee wants to merge 3 commits into
mainfrom
fm/pause-vs-orphan-run-p6

Conversation

@ruby-dlee

Copy link
Copy Markdown
Owner

Intent

Stop supervision from repeatedly wedge-escalating a lane that has finished, opened a green PR, and deliberately declared a pause because merge is held for a captain decision, while preserving the load-bearing rule that genuinely active authoritative working state outranks a stale pause and must still wedge-escalate. The implemented discriminator is conjunctive: the checks-green done event must be the immediately preceding non-empty status before the current pause, the ci log must be unknown with no newer re-arm or fixing marker, and the bounded liveness observation must specifically say the recorded run has no worktree. Treat that as heuristic corroboration, not a death verdict or abort authority; it could misclassify if a genuinely active CI run becomes undiscoverable while those adjacent stale events remain and its CI log is unavailable. Prove the negative direction explicitly: test_ci_pause_does_not_hide_a_real_active_run keeps positive work authoritative, and test_paused_authoritative_working_preserves_wedge_timer proves the watcher still escalates a real wedge. Do not claim to fix the no-mistakes lifecycle: the run itself still lacks a clean terminal state at the CI-ready point and can keep monitoring a held PR after its base moves; this change only suppresses the resulting false supervision alarm. The PR body must state this discriminator, its heuristic/misfire boundary, the named negative tests, and the remaining run-lifecycle defect so the alarm fix does not hide it.

What Changed

  • Captain, supervision now honors a held green-PR pause only when the immediately preceding non-empty status is checks-green done, the CI log is unknown with no newer re-arm/fixing marker, and bounded liveness reports that the recorded run has no worktree.
  • Documents that this is heuristic corroboration—not a death verdict or abort authority—and may misclassify an undiscoverable active CI run; the underlying no-mistakes run still lacks a clean CI-ready terminal state and may monitor a held PR after its base moves.
  • Preserves active-work wedge escalation through test_ci_pause_does_not_hide_a_real_active_run and test_paused_authoritative_working_preserves_wedge_timer, while isolating crosscheck evidence from provider credentials and bounding tmux smoke-test socket paths.

Risk Assessment

⚠️ Medium: Captain, the change is narrowly guarded and preserves positive active-work evidence, but it intentionally retains the documented heuristic risk of suppressing an alarm when a genuinely active CI run becomes undiscoverable.

Testing

The configured baseline and targeted tests passed, real CLI transcripts demonstrate suppression and preserved escalation, the worktree remains clean at the target commit, and only the future PR-body requirement remains unverifiable because no PR exists yet.

Evidence: Crew-state CLI matrix

CASE 1: exact orphan conjunction → paused CASE 2: live processes → working CASE 3: newer CI re-arm → working CASE 4: non-adjacent checks-green → working

CASE 1 — adjacent green report + pause + unknown CI log + recorded run has no worktree
state: paused · source: status-log · PR 2 green; merge held for captain decision · reported checks green immediately before pause · orphaned ci run record has no worktree
CASE 2 — same adjacent events, but recorded run has affirmative live processes
state: working · source: run-step · validating (running)
CASE 3 — no-worktree observation, but CI log has a newer base-advance re-arm marker
state: working · source: run-step · validating (running)
CASE 4 — no-worktree observation and unknown CI log, but checks-green is not immediately before pause
state: working · source: run-step · validating (running)
Evidence: Preserved wedge escalation

stale: test:fm-paused-working (idle 501s, possible wedge, escalation 1)

stale: test:fm-paused-working (idle 501s, possible wedge, escalation 1)
ok - a paused status overridden by authoritative working preserves its wedge timer and escalates
- Outcome: ⚠️ 1 warning across 8 runs (26h8m14s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - medium risk

✅ No issues found.

⚠️ **Test** - 1 warning
  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Leave code unchanged; failures are environment-contaminated
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Captain, leave code unchanged; failures are environment-contaminated
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Leave code unchanged; failures reproduce on pristine main
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Confirm targeted pause regressions pass unchanged
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Bound tmux smoke socket path
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Captain, isolate crosscheck evidence from provider credentials
1 error still open:

  • 🚨 tests failed with exit code 1
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"

🔧 Fix: Leave code unchanged; isolate macOS Keychain timeout
1 warning still open:

  • ⚠️ The required PR-body disclosure cannot yet be verified because GitHub has no open PR for fm/pause-vs-orphan-run-p6. Before delivery, the outer PR phase must verify the body states the conjunctive discriminator, heuristic/misfire boundary, both named negative tests, and remaining no-mistakes lifecycle defect.
  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; base=$(mktemp -d); mdir="$base/manifests"; mkdir -p "$mdir"; pids=""; for s in 1 2 3 4 5 6 7 8; do ( TMPDIR="$base/$s/tmp"; TMUX_TMPDIR="$base/$s/tmux"; export TMPDIR TMUX_TMPDIR; mkdir -p "$TMPDIR" "$TMUX_TMPDIR"; bin/fm-behavior-shards.sh --run "$s" 8 "$mdir/executed-$s.tsv" >"$base/shard-$s.log" 2>&1 ) & pids="$pids $!"; done; for p in $pids; do wait "$p" || rc=1; done; for s in 1 2 3 4 5 6 7 8; do echo "== behavior shard $s/8 =="; cat "$base/shard-$s.log"; done; bin/fm-behavior-shards.sh --verify 8 "$mdir" || rc=1; rm -rf "$base"; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"
  • Configured behavior-shard, agent-fleet pytest, and compileall baseline (already passed).
  • tests/fm-crew-state.test.sh
  • Focused test_paused_authoritative_working_preserves_wedge_timer from tests/fm-watch-triage.test.sh.
  • Hermetic real-CLI bin/fm-crew-state.sh matrix covering exact orphan corroboration, live-process precedence, newer CI re-arm precedence, and status adjacency.
  • ~/.local/bin/gh-axi api '/repos/ruby-dlee/firstmate/pulls?head=ruby-dlee:fm/pause-vs-orphan-run-p6'
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@ruby-dlee
ruby-dlee force-pushed the fm/pause-vs-orphan-run-p6 branch from c9e06ca to d9a2045 Compare August 12, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant