diff --git a/.agents/skills/firstmate-codexapp/SKILL.md b/.agents/skills/firstmate-codexapp/SKILL.md index 32eeb3f5ee2..c6e0dfccdd2 100644 --- a/.agents/skills/firstmate-codexapp/SKILL.md +++ b/.agents/skills/firstmate-codexapp/SKILL.md @@ -62,7 +62,7 @@ For a Firstmate-managed task, include an explicit status instruction: ```text Append supervisor-visible status lines to /state/.status. Use only these prefixes for status changes: working:, needs-decision:, blocked:, paused:, done:, failed:. -Use paused: only for a deliberate known external wait that should be rechecked later, never for a blocker that needs firstmate to act. +Before using paused:, follow the owned-and-clearing pause contract in docs/architecture.md; a bare paused: line is not absorbable. Before doing substantive work, append "working: Codex Desktop thread started". ``` diff --git a/AGENTS.md b/AGENTS.md index 26393457d8f..9e3428b528b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -626,9 +626,9 @@ Whenever at least one task is in flight, keep exactly one live supervision wait The emitted block is the only per-harness operating recipe in the session context. Do not substitute another harness's command shape for it. **Always-on wake triage (absorb only when proven benign).** -`bin/fm-watch.sh` classifies every wake in bash and absorbs the benign majority without waking you: crewmates with positive working evidence (an actively-running no-mistakes step for their branch, or a busy pane read via `bin/fm-crew-state.sh`) unless the separate permission-stall no-progress threshold has expired, a declared `paused:` external wait until its bounded recheck cadence under the proof and precedence contract owned by `docs/architecture.md`, and no-change heartbeats; that owner also records the known defect `herdr-push-transition-pause-gate-h8`, under which the Herdr native edge can silently absorb a lane that owes an unanswered keyed decision and let it go quiet. +`bin/fm-watch.sh` classifies every wake in bash and absorbs the benign majority without waking you: crewmates with positive working evidence (an actively-running no-mistakes step for their branch, a busy pane read via `bin/fm-crew-state.sh`, or the measured CPU/process-progress verdict owned by `docs/architecture.md`) unless the separate permission-stall no-progress threshold has expired, a valid owned-and-clearing `paused:` external wait until its bounded recheck cadence under the proof and precedence contract owned by `docs/architecture.md`, and no-change heartbeats. It never absorbs a crewmate that stopped without that evidence - whatever its stale status log claims - and only an actionable wake is queued durably and ends the supervision wait, so you resume the emitted protocol exactly once per actionable event. -A `paused:` status is a deliberate external wait, not `blocked:`; its initial signal still surfaces once, and a forgotten pause re-surfaces for a recheck once per window. +A `paused:` status is a deliberate external wait, not `blocked:`; `docs/architecture.md` owns the full owner/clearing contract, its initial signal surfaces once, incomplete declarations receive no absorption, and a forgotten valid pause re-surfaces for a recheck once per window. A pause gates starting new work only and never suspends custody of a validation run already in flight, because a gated run parks at its next gate whether or not anyone is watching; this is the in-flight validation-custody boundary. Repeated unchanged wedge or permission-stall escalations eventually add `demand-deep-inspection` to the wake reason so they are not mistaken for another routine validation wait. `docs/architecture.md` ("Event-driven supervision") owns the general classification mechanism and shared classifier library, while `docs/permission-stall-detection.md` owns permission-prompt matching and the macOS timeout heuristic; while `state/.afk` exists the daemon owns triage and the watcher surfaces every wake to it. diff --git a/bin/fm-brief.sh b/bin/fm-brief.sh index bd2468257e2..b7936dfaf17 100755 --- a/bin/fm-brief.sh +++ b/bin/fm-brief.sh @@ -37,8 +37,8 @@ # Scout tasks ignore mode - their deliverable is a report, not a merge. # Every scaffold's status protocol distinguishes the configured # declared-external-wait verb (FM_CLASSIFY_PAUSED_VERB, default "paused") from -# "blocked:": pause for a known external wait expected to clear on its own, -# blocked when firstmate must act. +# "blocked:": pause requires a named owner and observable clearing condition; +# blocked applies when firstmate must act. # Ship tasks include a project-memory section so durable project-intrinsic # learnings can be committed to AGENTS.md through the project's delivery path; # it carries the AGENTS.md authoring bar (widely useful knowledge only, pointers @@ -170,7 +170,8 @@ Handle routine work yourself. Report only true captain-relevant outcomes or a declared external wait by appending one line: \`echo "{state}: {one short line}" >> $STATUS_FILE\` States: working, needs-decision, blocked, $PAUSED_VERB, done, failed. -Use \`$PAUSED_VERB: {why}\` (distinct from \`blocked:\`) only when your domain is deliberately idling on a known external wait you expect to clear on its own; use \`blocked:\` when you are stuck and need firstmate to act. +Use \`$PAUSED_VERB: {reason}; owner={named owner}; clears={observable condition}\` (distinct from \`blocked:\`) only when your domain is deliberately idling on a known external wait. +The owner and clearing condition are required for pause absorption; use \`blocked:\` when you are stuck and need firstmate to act. Use this only for material phase changes, a captain decision, a real blocker, a failure, or work ready for review. This is also how you return the answer to a marked from-firstmate request above. When a decision you escalated is answered or a blocker clears and your domain resumes, append \`resolved: {how it was decided or unblocked}\` (keyed with \`[key=]\` if you opened it with one) so it is durably closed instead of resurfacing behind later unrelated events. @@ -250,8 +251,8 @@ The report is the only thing that survives, so anything worth keeping must be in Each append wakes firstmate, so report sparingly: only phase changes a supervisor would act on and the needs-decision/blocked/paused/done/failed states. No step-by-step FYI progress lines; firstmate reads your pane for that. - Use \`$PAUSED_VERB: {why}\` - distinct from \`blocked:\` - ONLY when you are deliberately idling on a - known external wait you expect to clear on its own (an upstream release, a rate-limit reset): + Use \`$PAUSED_VERB: {reason}; owner={named owner}; clears={observable condition}\` - distinct from \`blocked:\` - ONLY when you are deliberately idling on a + known external wait (an upstream release, a rate-limit reset). Both fields are required: firstmate then leaves your idle pane alone and rechecks it on a long cadence instead of treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help. 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help. @@ -387,9 +388,9 @@ $RULE1 would act on (setup done, bug reproduced, fix implemented, validation passed) and the needs-decision/blocked/paused/done/failed states. No step-by-step FYI progress lines; firstmate reads your pane for that. - Use \`$PAUSED_VERB: {why}\` - distinct from \`blocked:\` - ONLY when you are deliberately idling on a - known external wait you expect to clear on its own (an upstream release, a rate-limit reset, - a scheduled window): firstmate then leaves your idle pane alone and rechecks it on a long + Use \`$PAUSED_VERB: {reason}; owner={named owner}; clears={observable condition}\` - distinct from \`blocked:\` - ONLY when you are deliberately idling on a + known external wait (an upstream release, a rate-limit reset, a scheduled window). Both fields are required: + firstmate then leaves your idle pane alone and rechecks it on a long cadence instead of treating it as a possible wedge. Use \`blocked:\` when you are stuck and need help. 5. If you hit the same obstacle twice, append \`blocked: {why}\` and stop; firstmate will help. 6. If a decision belongs to a human (product choices, destructive actions, ask-user findings), diff --git a/bin/fm-classify-lib.sh b/bin/fm-classify-lib.sh index 3e8801cafaa..df92e40cd2b 100755 --- a/bin/fm-classify-lib.sh +++ b/bin/fm-classify-lib.sh @@ -39,7 +39,7 @@ FM_CREW_STATE_BIN="${FM_CREW_STATE_BIN:-$_FM_CLASSIFY_LIB_DIR/fm-crew-state.sh}" FM_CLASSIFY_CAPTAIN_RE_DEFAULT='done:|needs-decision:|blocked:|failed:|PR ready|checks green|ready in branch|merged' # The deliberate-external-wait verb. A crewmate (or firstmate steering it) appends -# paused: +# paused: ; owner=; clears= # to declare it is intentionally idling on a KNOWN external dependency - an # upstream release, a vendor rate-limit reset, a scheduled window. Unlike # `blocked:` (stuck, firstmate must help) an idle `paused:` pane is EXPECTED, so @@ -50,10 +50,18 @@ FM_CLASSIFY_CAPTAIN_RE_DEFAULT='done:|needs-decision:|blocked:|failed:|PR ready| # (status_is_paused) rather than hardcoding the literal, so the vocabulary cannot # drift between the two consumers. FM_CLASSIFY_PAUSED_VERB overrides it. # -# The pause verb alone is NOT sufficient to prove a deliberate wait: see the -# failure-pause discriminator immediately below. +# The pause verb alone is NOT sufficient to prove a deliberate wait. The +# declaration earns absorption only when status_is_paused validates the owner and +# clearing-condition fields below, and the failure-pause discriminator remains an +# independent refusal. FM_CLASSIFY_PAUSED_VERB_DEFAULT='paused' +# The two structured field names a pause declaration must carry. Defined once here +# and read by every consumer - the field reader, the contract check, and the prose +# scan that must skip them - so the grammar cannot drift between them. +_FM_CLASSIFY_PAUSE_OWNER_KEY='owner' +_FM_CLASSIFY_PAUSE_CLEARS_KEY='clears' + # Failure vocabulary for the FAILURE-PAUSE discriminator. # # A crewmate reporting a FAILURE frequently reaches for the pause verb - it has @@ -109,6 +117,7 @@ status_is_captain_relevant() { # (the discriminator has its own FM_CLASSIFY_PAUSE_FAILURE_RE knob). status_pause_is_failure "$line" && return 0 status_is_paused "$line" && return 1 + status_pause_is_incomplete "$line" && return 0 if [ -z "${FM_CAPTAIN_RE+x}" ]; then verb=$(status_line_verb "$line") case "$verb" in @@ -128,58 +137,133 @@ _fm_status_has_pause_verb() { # [ "$verb" = "${FM_CLASSIFY_PAUSED_VERB:-$FM_CLASSIFY_PAUSED_VERB_DEFAULT}" ] } -# Print the HEADLINE of a pause reason: its leading clause, bounded to -# FM_CLASSIFY_PAUSE_HEAD_WORDS words. +# Print the HEADLINE of every PROSE clause of a pause reason, one per line. # -# POSITION IS THE DISCRIMINATOR. A crewmate writes a status reason as +# POSITION IS THE DISCRIMINATOR. A crewmate writes each clause of a status reason as # "[: ]", and the headline is its own one-phrase answer to "why am # I stopped". A failure report puts the failure there ("error: ...", "... fresh run # error: ...", "run drive failed on ...: ..."), while a deliberate wait puts the # wait there ("waiting on upstream CI", "rate limit until 15:00") and can only mention # a failure LATER, in passing ("waiting for the captain to decide how to handle the # failed Shopify webhook"). Scanning the whole reason would escalate that last line; -# scanning only the headline keeps it absorbed, which is the point of the pause verb. +# scanning only the headlines keeps it absorbed, which is the point of the pause verb. # -# Two bounds define the headline, and both are needed: -# - the text before the first colon of the reason, so a failure headline is not +# Two bounds define a clause headline, and both are needed: +# - the text before that clause's first colon, so a failure headline is not # diluted by however long its stack-trace/detail tail happens to be; and -# - a word cap, because a colon-free prose reason would otherwise make the ENTIRE +# - a word cap, because a colon-free prose clause would otherwise make the ENTIRE # sentence the headline and re-admit exactly the passing-mention false positive # the clause bound was meant to avoid. # The cap is deliberately loose (8) rather than tight: a real failure headline names # the run, the tool, and the verb before it gets to the failure word, and the cost # asymmetry runs one way - a false wake is one cheap read, a false absorb hid six # stalled crewmates for hours - so ambiguity resolves toward escalating. -status_pause_headline() { # +# +# Semicolons separate clauses, and EVERY prose clause is a headline of its own: a +# crewmate that opens with a wait and then reports a failure in a later clause is +# still reporting a failure, so bounding the scan to the leading clause would hide +# it. The recognized owner=/clears= fields are the one exception and are dropped - +# they are structured contract metadata, and the observable condition that ends a +# wait may legitimately name a failure ("clears=the upstream timeout stops"), which +# would otherwise escalate exactly the declarations the contract asks for. +_fm_status_pause_prose_headlines() { # local n=${FM_CLASSIFY_PAUSE_HEAD_WORDS:-$FM_CLASSIFY_PAUSE_HEAD_WORDS_DEFAULT} - status_line_note "$1" | awk -v n="$n" ' - { sub(/:.*/, "") - out = "" - for (i = 1; i <= NF && i <= n; i++) out = out (i > 1 ? " " : "") $i - print out }' -} - -# 0 if uses the pause verb but its HEADLINE carries failure vocabulary -# (FM_CLASSIFY_PAUSE_FAILURE_RE) - i.e. the crewmate reported a FAILURE under the -# pause verb rather than declaring an external wait. Such a line is not a pause at all -# for triage purposes: status_is_paused rejects it (so no consumer absorbs it on the -# pause cadence) and status_is_captain_relevant escalates it like blocked:/failed:. -# Pure read of the line; both supervisors inherit it through those two predicates. + status_line_note "$1" | awk -F';' -v n="$n" \ + -v owner_key="$_FM_CLASSIFY_PAUSE_OWNER_KEY" -v clears_key="$_FM_CLASSIFY_PAUSE_CLEARS_KEY" ' + { for (i = 1; i <= NF; i++) { + clause = $i + sub(/^[[:space:]]+/, "", clause) + sub(/[[:space:]]+$/, "", clause) + if (index(clause, owner_key "=") == 1 || index(clause, clears_key "=") == 1) continue + sub(/:.*/, "", clause) + words = split(clause, word, /[[:space:]]+/) + out = "" + for (w = 1; w <= words && w <= n; w++) out = out (w > 1 ? " " : "") word[w] + if (out != "") print out + } + }' +} + +# The leading prose clause's headline - the reason's own one-phrase answer, used to +# prove a declaration actually states a reason rather than only its contract fields. +status_pause_headline() { # + _fm_status_pause_prose_headlines "$1" | sed -n '1p' +} + +# 0 if uses the pause verb but one of its prose-clause HEADLINES carries +# failure vocabulary (FM_CLASSIFY_PAUSE_FAILURE_RE) - i.e. the crewmate reported a +# FAILURE under the pause verb rather than declaring an external wait. Such a line is +# not a pause at all for triage purposes: status_is_paused rejects it (so no consumer +# absorbs it on the pause cadence) and status_is_captain_relevant escalates it like +# blocked:/failed:. Pure read of the line; both supervisors inherit it through those +# two predicates. status_pause_is_failure() { # _fm_status_has_pause_verb "$1" || return 1 - status_pause_headline "$1" \ + _fm_status_pause_prose_headlines "$1" \ | grep -qiE "${FM_CLASSIFY_PAUSE_FAILURE_RE:-$FM_CLASSIFY_PAUSE_FAILURE_RE_DEFAULT}" } -# 0 if a status line declares a DELIBERATE external wait (paused: ). A pure -# read of the line itself, so the daemon's classify_stale can reuse the last line -# it already read without a fm-crew-state.sh call. Matches only the verb before the -# first colon, so a reason mentioning "paused" elsewhere does not false-match, and -# rejects a failure reported under the pause verb (status_pause_is_failure) so a -# failure can never be absorbed as a wait. +# Print one semicolon-delimited field from a pause note. Explicit field names make +# the absorption contract mechanically auditable; inferring ownership or +# observability from free prose would make malformed waits look healthy again. +status_pause_field() { # + local key=$2 + status_line_note "$1" | awk -F';' -v key="$key" ' + { for (i = 1; i <= NF; i++) { + field = $i + sub(/^[[:space:]]*/, "", field) + sub(/[[:space:]]*$/, "", field) + if (index(field, key "=") == 1) { + value = substr(field, length(key) + 2) + sub(/^[[:space:]]*/, "", value) + sub(/[[:space:]]*$/, "", value) + print value + exit + } + } + }' +} + +status_pause_owner() { status_pause_field "$1" "$_FM_CLASSIFY_PAUSE_OWNER_KEY"; } +status_pause_clearing_condition() { status_pause_field "$1" "$_FM_CLASSIFY_PAUSE_CLEARS_KEY"; } + +_fm_pause_field_is_named() { # + local value normalized + value=$1 + [ -n "$value" ] || return 1 + normalized=$(printf '%s' "$value" | tr '[:upper:]' '[:lower:]') + case "$normalized" in + unknown|none|nobody|unowned|n/a|na|tbd) return 1 ;; + esac + printf '%s' "$value" | grep -q '[[:alnum:]]' +} + +# 0 only when the pause declaration names both its owner and the observable fact +# that clears it. This is the earned-absorption boundary: the literal pause verb is +# intentionally insufficient. +status_pause_has_contract() { # + local line=$1 owner clears + _fm_status_has_pause_verb "$line" || return 1 + status_pause_is_failure "$line" && return 1 + [ -n "$(status_pause_headline "$line")" ] || return 1 + owner=$(status_pause_owner "$line") + clears=$(status_pause_clearing_condition "$line") + _fm_pause_field_is_named "$owner" || return 1 + _fm_pause_field_is_named "$clears" +} + +# 0 if a status line declares an OWNED, CLEARING external wait. A pure read of +# the line itself, so both supervisors can use it before a current-state read. status_is_paused() { # + status_pause_has_contract "$1" +} + +# 0 when the line attempted a non-failure pause but omitted the auditable contract. +# Such a signal is actionable and never receives pause absorption. +status_pause_is_incomplete() { # _fm_status_has_pause_verb "$1" || return 1 - ! status_pause_is_failure "$1" + status_pause_is_failure "$1" && return 1 + ! status_is_paused "$1" } # --- durable keyed decisions ------------------------------------------------ @@ -321,8 +405,8 @@ signal_reason_is_actionable() { # ... # STATE/FM_STATE_OVERRIDE resolution window_to_task already relies on. Prints nothing # when no state dir is known or the file is absent - which the pause gate below treats # as "this pause cannot be proven", never as "no decisions are open". -_fm_status_file_for() { # - local id=$1 state=${STATE:-${FM_STATE_OVERRIDE:-}} +_fm_status_file_for() { # [state-dir] + local id=$1 state=${2:-${STATE:-${FM_STATE_OVERRIDE:-}}} [ -n "$id" ] && [ -n "$state" ] || return 0 [ -f "$state/$id.status" ] || return 0 printf '%s/%s.status' "$state" "$id" @@ -336,7 +420,6 @@ _fm_status_file_for() { # # rejects a FAILURE reported under the pause verb (status_pause_is_failure), and # 2. the keyed open/resolved fold (status_open_decisions) is empty, so no question # is still unanswered underneath that pause. -# # (2) is the safety boundary and is not redundant with (1): the status stream is an # append-only EVENT log, so a later `paused:` line MASKS an earlier still-open # needs-decision from any last-line read. A lane with an unanswered question must @@ -345,9 +428,9 @@ _fm_status_file_for() { # # Fails closed in both directions. An id with no locatable status file cannot be # proven either way, so it is refused rather than absorbed: absence of a signal is # never evidence of a pause. -crew_declared_pause_absorbable() { # [declared-pause-status-line] - local id=$1 declared=${2:-} f snapshot current before after - f=$(_fm_status_file_for "$id") +crew_declared_pause_absorbable() { # [declared-pause-status-line] [state-dir] + local id=$1 declared=${2:-} state=${3:-} f snapshot current before after + f=$(_fm_status_file_for "$id" "$state") [ -n "$f" ] || return 1 before=$(_fm_status_file_sig "$f") [ -n "$before" ] || return 1 @@ -373,32 +456,17 @@ _fm_status_file_sig() { # working - an actively-running no-mistakes step (running/fixing/ci) or a busy # pane; the crewmate is legitimately mid-work on a static-looking pane # (e.g. waiting on CI); -# paused - the crewmate declared an external wait and nothing is outstanding -# (crew_declared_pause_absorbable), so its pane is EXPECTED to idle; -# none - neither, so the wake must surface. -# -# PRECEDENCE, and why it is this way round. bin/fm-crew-state.sh's authoritative -# verdict ("state: · source: · ") answers "what is the PIPELINE -# doing"; a declared pause answers "what is the WORK doing". Only one of those can -# overrule the other, and active work is the one that does: a crewmate that appended -# `paused:` and then STARTED a run has superseded its own declaration, so a `working` -# run-step or busy pane still wins. Every OTHER verdict yields to a proven pause. +# paused - the crewmate declared an owned-and-clearing external wait and nothing +# is outstanding (crew_declared_pause_absorbable), so its pane is +# EXPECTED to idle; +# none - neither, so the wake must surface (including dead and unknown +# liveness, and an unreadable verdict). # -# That last sentence is the 2026-08-03 fix. This used to admit a pause only against a -# `done` run-step (the narrower #53 carve-out), which made the pipeline verdict the -# discriminator for a signal that is not about the pipeline at all: -# - `parked` vetoed it, so a lane whose recycled worktree had been re-checked-out -# onto another lane's branch inherited that foreign run's parked gate and -# wedge-escalated eight times in a row against an explicit, resolved pause; -# - `failed` vetoed it, though fm-crew-state.sh maps a routinely CANCELLED run here -# too, so ordinary teardown-cancelled runs silenced legitimate pauses; and -# - `unknown` vetoed it, so a pause whose pane was GONE - the case with the least -# reason to suspect a wedge - surfaced as one. -# Absorption never consulted pane liveness in any branch; the pipeline verdict was -# always the discriminator. Safety does not rest on those vetoes: it rests on the two -# proofs in crew_declared_pause_absorbable (a non-failure pause verb, and an empty -# open-decision fold) and on the caller's bounded FM_PAUSE_RESURFACE_SECS recheck, -# which re-surfaces an absorbed pause once per window so one cannot rot invisibly. +# PRECEDENCE. Every current run-step verdict is authoritative because a pause gates +# starting new validation work and cannot suspend an in-flight run. Only a no-run +# status-log pause can be absorbed. Every absorbed pause also passes the durable +# declaration and open-decision proofs above and re-surfaces on the caller's bounded +# FM_PAUSE_RESURFACE_SECS cadence. # # One fm-crew-state.sh read serves both absorb reasons at once. The optional second # argument lets stale-pane triage pass the status line it already read; omitted, the @@ -468,31 +536,59 @@ scan_crew_liveness_observations() { # done } -crew_absorb_class() { # [declared-pause-status-line] - local id=$1 declared_pause=${2:-} line state src liveness +crew_absorb_class() { # [declared-pause-status-line] [current-state-line] [state-dir] + local id=$1 declared_pause=${2:-} line=${3:-} state_dir=${4:-} state src liveness [ -n "$id" ] || { printf 'none'; return; } - line=$(crew_state_line "$id") + if [ -z "$line" ]; then + if [ -n "$state_dir" ]; then + line=$(FM_STATE_OVERRIDE="$state_dir" crew_state_line "$id") + else + line=$(crew_state_line "$id") + fi + fi case "$line" in state:*) state=${line#state: }; state=${state%% *} src=${line#*source: }; src=${src%% *} if [ "$state" = working ]; then liveness=$(crew_state_liveness_verdict "$line") + # Only a positive or absent verdict is working evidence; `dead` and + # `unknown` both fall through to the pause gate and then to `none`. case "$src:$liveness" in run-step:alive|run-step:|pane:alive|pane:) printf 'working'; return ;; esac fi + case "$state:$src" in + paused:status-log) + if crew_declared_pause_absorbable "$id" "$declared_pause" "$state_dir"; then + printf 'paused' + else + printf 'none' + fi + return + ;; + esac ;; esac - # An unreadable verdict falls through here too: it is not evidence against a pause - # the crewmate durably declared, and it is exactly what a torn-down pane produces. - if crew_declared_pause_absorbable "$id" "$declared_pause"; then - printf 'paused' - return - fi printf 'none' } +# 0 only when the current run-step carries the probe's positive progress verdict. +# The probe owns the CPU-versus-elapsed measurement and threshold; this consumer +# deliberately trusts only `alive`, while `dead` and `unknown` - the grade the probe +# gives presence without established progress - stay on the surface side. Used at the +# wedge threshold to distinguish a starved-but-computing static pane from a process +# set accumulating near-zero CPU. +crew_has_measured_progress() { # + local line state src liveness + line=$(crew_state_line "$1") + case "$line" in state:*) ;; *) return 1 ;; esac + state=${line#state: }; state=${state%% *} + src=${line#*source: }; src=${src%% *} + liveness=$(crew_state_liveness_verdict "$line") + [ "$state" = working ] && [ "$src" = run-step ] && [ "$liveness" = alive ] +} + # 0 if crewmate shows POSITIVE evidence it is still working (crew_absorb_class # reports `working`). This is the "provably working" predicate at the heart of # absorb-only-when-provably-working: a no-verb turn-end or stale wake is absorbed @@ -513,11 +609,11 @@ crew_is_provably_working() { # # 0 if crewmate is in a declared external-wait pause with nothing outstanding. # The stale path absorbs such a crewmate (on a long re-surface cadence) instead of # escalating a possible wedge. See crew_absorb_class for the precedence and -# crew_declared_pause_absorbable for the two proofs a pause must satisfy. -crew_is_paused() { # +# crew_declared_pause_absorbable for the pause proof contract. +crew_is_paused() { # [declared-pause-status-line] # This predicate needs only pause-or-not; dead and unknown arrive as `none` # and stay distinct from the one absorbable paused outcome. - case "$(crew_absorb_class "$1")" in + case "$(crew_absorb_class "$1" "${2:-}")" in paused) return 0 ;; working|none) return 1 ;; *) return 1 ;; diff --git a/bin/fm-crew-state.sh b/bin/fm-crew-state.sh index 45e27c50dde..1aeb99b7bd4 100755 --- a/bin/fm-crew-state.sh +++ b/bin/fm-crew-state.sh @@ -15,14 +15,14 @@ # and log reads plus fixed mapping logic, no heuristics and no LLM. Output is one # stable, parseable, token-tight line firstmate can read every heartbeat: # -# state: · source: · +# state: · source: · # ... · liveness: · step: # # Logic, in order: # 1. Resolve worktree + backend target + kind from state/.meta. # 2. Matching no-mistakes run for this crewmate's branch, active or terminal # (from `axi status`, or the coarse `no-mistakes runs` fallback)? -# The run-step is AUTHORITATIVE for active, parked, and failed states: +# The run-step is AUTHORITATIVE for every matching run state: # running/fixing -> working, ci -> working, awaiting_approval/fix_review -> # parked (with gate findings), and failed/cancelled -> failed. A PR-ready # passed/checks-passed claim becomes done only after GitHub resolves the @@ -39,10 +39,10 @@ # the run-step shows the run moved on, the log is deterministically stale and # is flagged superseded. A genuinely parked run plus a needs-decision log # agree, and are reported as parked. -# 4. No run for this crewmate (pre-validation, or kind=scout): fall back to the -# recorded backend's pane busy state, then the status log's last line only -# when its verb maps to a recognized run-state. Decision-only events such as -# `resolved` never become current state or detail. +# 4. Only an affirmatively absent run (pre-validation, or kind=scout) permits +# fallback to the recorded backend's pane busy state, then the status log's +# last line when its verb maps to a recognized run-state. An unreadable run +# lookup reports unknown; decision-only events never become current state. # 5. Missing meta or torn-down worktree: report unknown · none. If no run is # attributed to this crewmate, a dead endpoint also reports unknown · none rather # than trusting a stale status log. @@ -121,7 +121,7 @@ log_last_line() { } # Map a status-log verb onto a canonical state for the fallback path. `paused` is # the deliberate-external-wait verb (fm-classify-lib.sh's FM_CLASSIFY_PAUSED_VERB): -# a crewmate with no active run and an idle pane that declared a known external wait +# a crewmate with no active run and an idle pane that declared a valid owned-and-clearing wait # reports `paused` distinctly, so a supervisor reading this sees a declared pause # and its reason rather than a wedge-suspect idle. A line that uses the pause verb to # report a FAILURE is not a pause (fm-classify-lib.sh's status_pause_is_failure); it @@ -216,19 +216,28 @@ strip_quotes() { trim "$s" } -# Bounded no-mistakes call in the worktree; stdout only, never fails the script. +# Bounded no-mistakes call in the worktree. HAVE_TIMEOUT=none if command -v timeout >/dev/null 2>&1; then HAVE_TIMEOUT=timeout elif command -v gtimeout >/dev/null 2>&1; then HAVE_TIMEOUT=gtimeout elif command -v perl >/dev/null 2>&1; then HAVE_TIMEOUT=perl fi -nm_run() { # +nm_run_capture() { # + local output_var=$1 _nm_captured _nm_status + shift case "$HAVE_TIMEOUT" in - timeout) ( cd "$WT" && timeout "$NM_TIMEOUT" no-mistakes "$@" ) 2>/dev/null || true ;; - gtimeout) ( cd "$WT" && gtimeout "$NM_TIMEOUT" no-mistakes "$@" ) 2>/dev/null || true ;; - perl) ( cd "$WT" && perl -e 'my $t = shift; my $pid = fork; die "fork failed" unless defined $pid; if (!$pid) { setpgrp(0, 0); exec @ARGV } local $SIG{ALRM} = sub { kill "TERM", -$pid; select undef, undef, undef, 0.2; kill "KILL", -$pid; exit 124 }; alarm $t; waitpid $pid, 0; exit($? >> 8)' "$NM_TIMEOUT" no-mistakes "$@" ) 2>/dev/null || true ;; - *) true ;; + timeout) _nm_captured=$(cd "$WT" && timeout "$NM_TIMEOUT" no-mistakes "$@" 2>/dev/null); _nm_status=$? ;; + gtimeout) _nm_captured=$(cd "$WT" && gtimeout "$NM_TIMEOUT" no-mistakes "$@" 2>/dev/null); _nm_status=$? ;; + perl) _nm_captured=$(cd "$WT" && perl -e 'my $t = shift; my $pid = fork; die "fork failed" unless defined $pid; if (!$pid) { setpgrp(0, 0); exec @ARGV; exit 127 } local $SIG{ALRM} = sub { kill "TERM", -$pid; select undef, undef, undef, 0.2; kill "KILL", -$pid; exit 124 }; alarm $t; waitpid $pid, 0; my $status = $?; exit(($status & 127) ? 128 + ($status & 127) : $status >> 8)' "$NM_TIMEOUT" no-mistakes "$@" 2>/dev/null); _nm_status=$? ;; + *) _nm_captured=; _nm_status=127 ;; esac + printf -v "$output_var" '%s' "$_nm_captured" + return "$_nm_status" +} +nm_run() { # + local captured= + nm_run_capture captured "$@" || true + printf '%s' "$captured" } # Bounded, read-only GitHub query in the worktree; stdout only, never fails the @@ -237,7 +246,7 @@ gh_axi_run() { # case "$HAVE_TIMEOUT" in timeout) ( cd "$WT" && timeout "$GH_TIMEOUT" gh-axi "$@" ) 2>/dev/null || true ;; gtimeout) ( cd "$WT" && gtimeout "$GH_TIMEOUT" gh-axi "$@" ) 2>/dev/null || true ;; - perl) ( cd "$WT" && perl -e 'my $t = shift; my $pid = fork; die "fork failed" unless defined $pid; if (!$pid) { setpgrp(0, 0); exec @ARGV } local $SIG{ALRM} = sub { kill "TERM", -$pid; select undef, undef, undef, 0.2; kill "KILL", -$pid; exit 124 }; alarm $t; waitpid $pid, 0; exit($? >> 8)' "$GH_TIMEOUT" gh-axi "$@" ) 2>/dev/null || true ;; + perl) ( cd "$WT" && perl -e 'my $t = shift; my $pid = fork; die "fork failed" unless defined $pid; if (!$pid) { setpgrp(0, 0); exec @ARGV; exit 127 } local $SIG{ALRM} = sub { kill "TERM", -$pid; select undef, undef, undef, 0.2; kill "KILL", -$pid; exit 124 }; alarm $t; waitpid $pid, 0; my $status = $?; exit(($status & 127) ? 128 + ($status & 127) : $status >> 8)' "$GH_TIMEOUT" gh-axi "$@" ) 2>/dev/null || true ;; *) true ;; esac } @@ -408,11 +417,10 @@ verify_ready_head_or_emit() { # # both runs validate the same commit and the live PR-head comparison therefore # cannot distinguish them. verify_no_newer_active_run_or_emit() { # - local branch=$1 newest newest_status newest_rest newest_head - newest=$(nm_runs_status_for_branch "$branch") - if [ -z "$newest" ]; then - emit unknown run-step "PR-ready run-step could not verify the newest branch run; do not merge" - fi + local branch=$1 newest newest_status newest_rest newest_head status + newest=$(nm_runs_status_for_branch "$branch") || status=$? + [ "${status:-0}" -eq 0 ] \ + || emit unknown run-step "PR-ready run-step could not verify the newest branch run; do not merge" newest_status=${newest%%|*} newest_rest=${newest#*|} newest_head=${newest_rest%%|*} @@ -539,12 +547,14 @@ nm_active_step_name() { # One-line liveness verdict for the active step's own processes. A one-second # in-invocation membership sample lets this one-shot caller establish child # turnover without paying the 20-second window CPU rates require. Stable process -# membership falls back to the probe's preserved long-window CPU baseline. This -# consumer accepts exactly three verdicts: alive, dead, or unknown. A missing, -# failed, empty, or malformed probe is unknown, never silence and never evidence -# of health or death. +# membership falls back to the probe's preserved long-window CPU baseline. The +# probe owns the CPU-versus-elapsed measurement and its threshold; this consumer +# accepts exactly three verdicts: alive, dead, or unknown - presence the probe +# could not prove is progressing arrives as a graded unknown, not a fourth verdict. +# A missing, failed, empty, or malformed probe is unknown, never silence and never +# evidence of health or death. nm_step_liveness() { - local run_id out status=0 verdict procs doing grade detail detail_fields line rest reported_run structured_detail + local run_id out status=0 verdict procs doing grade detail detail_fields line rest reported_run structured_detail progress run_id=$(strip_quotes "$(nm_field id)") [ -n "$run_id" ] || { printf 'unknown (probe unreadable: run id unavailable)'; return; } [ -x "$NM_LIVENESS_BIN" ] || { printf 'unknown (probe unreadable: executable unavailable)'; return; } @@ -555,7 +565,7 @@ nm_step_liveness() { # with neither timeout nor gtimeout - the ordinary macOS default, including # this one - fell through to an UNBOUNDED probe call, so a slow lsof or # process scan could block the supervision read for as long as it took. - perl) out=$(perl -e 'my $t = shift; my $pid = fork; die "fork failed" unless defined $pid; if (!$pid) { setpgrp(0, 0); exec @ARGV } local $SIG{ALRM} = sub { kill "TERM", -$pid; select undef, undef, undef, 0.2; kill "KILL", -$pid; exit 124 }; alarm $t; waitpid $pid, 0; exit($? >> 8)' \ + perl) out=$(perl -e 'my $t = shift; my $pid = fork; die "fork failed" unless defined $pid; if (!$pid) { setpgrp(0, 0); exec @ARGV; exit 127 } local $SIG{ALRM} = sub { kill "TERM", -$pid; select undef, undef, undef, 0.2; kill "KILL", -$pid; exit 124 }; alarm $t; waitpid $pid, 0; my $status = $?; exit(($status & 127) ? 128 + ($status & 127) : $status >> 8)' \ "$NM_TIMEOUT" "$NM_LIVENESS_BIN" "$run_id" --sample 1 2>/dev/null) || status=$? ;; *) printf 'unknown (probe unreadable: no bounded runner available)'; return ;; esac @@ -638,6 +648,7 @@ nm_step_liveness() { doing=${out##*doing: } [ "$doing" = "$out" ] && doing="" line="$verdict ($procs procs)" + progress="" if [ "$verdict" = unknown ]; then case "$grade" in unreadable|present-unproven|present-no-progress|transition) ;; @@ -649,29 +660,36 @@ nm_step_liveness() { "$detail_fields"|grade:*) detail="probe reported unknown" ;; esac line="unknown (grade: $grade; $procs procs; $detail)" + else + # A decided verdict keeps the probe's own CPU-versus-elapsed evidence, which + # is what separates measured work from a process set sitting at the near-zero + # floor. `doing:` is already excluded: structured_detail comes from the + # prefix before it, so truncated argv can never be read as progress. + progress=$structured_detail fi + [ -n "$progress" ] && line="$line; $progress" [ -n "$doing" ] && line="$line on $doing" printf '%s' "$line" } nm_runs_status_for_branch() { # - local branch=$1 out row st rest br head pr field - out=$(nm_run runs --limit "$FM_CREW_STATE_RUNS_LIMIT") - [ -n "$out" ] || return 0 + local branch=$1 out row st br head pr field + nm_run_capture out runs --limit "$FM_CREW_STATE_RUNS_LIMIT" || return 2 + [ -n "$out" ] || return 1 while IFS= read -r row; do row=$(trim "$row") [ -n "$row" ] || continue - st=${row%% *} - rest=${row#* } - rest=$(trim "$rest") - br=${rest%% *} + # shellcheck disable=SC2086 # Intentional whitespace tokenization of CLI output. + set -- $row + [ "$#" -ge 4 ] || return 2 + st=$1 + br=$2 + head=$3 + case "$st" in running|completed|failed|cancelled) ;; *) return 2 ;; esac if [ "$br" = "$branch" ]; then # The human-oriented runs row is whitespace-delimited and always starts # with status, branch, and short head. Preserve the optional PR URL so a # coarse branch match can still verify a checks-green status-log event. - # shellcheck disable=SC2086 # Intentional whitespace tokenization of CLI output. - set -- $row - head=${3:-} pr= for field in "$@"; do case "$field" in https://github.com/*/*/pull/[0-9]*) pr=$field ;; esac @@ -680,7 +698,20 @@ nm_runs_status_for_branch() { # return 0 fi done <<< "$out" - return 0 + return 1 +} + +nm_status_output_valid() { + local id branch status + printf '%s\n' "$RUN_OUT" | grep -q '^run:$' || return 1 + id=$(strip_quotes "$(nm_field id)") + branch=$(strip_quotes "$(nm_field branch)") + status=$(strip_quotes "$(nm_field status)") + [ -n "$id" ] && [ -n "$branch" ] || return 1 + case "$status" in + running|fixing|ci|awaiting_approval|fix_review|completed|failed|cancelled) return 0 ;; + esac + return 1 } # CREW_BRANCH is empty at detached HEAD (a just-spawned crewmate, or a scout's @@ -688,6 +719,7 @@ nm_runs_status_for_branch() { # CREW_BRANCH=$(git -C "$WT" symbolic-ref --quiet --short HEAD 2>/dev/null || true) HAVE_RUN=0 +RUN_DISCOVERY=unknown # RUN_SOURCE distinguishes the two ways HAVE_RUN=1 can happen: "full" means # $RUN_OUT is real `axi status` TOON with step/gate detail; "coarse" means only # a bare status word came back from the runs-list fallback above, so the @@ -696,25 +728,32 @@ RUN_SOURCE=full COARSE_STATUS="" # Scouts and secondmates never drive a no-mistakes validation of their own # worktree, so skip the lookup for them and read state from pane/log directly. -if [ "$KIND" = ship ] && [ -n "$CREW_BRANCH" ] && command -v no-mistakes >/dev/null 2>&1; then - RUN_OUT=$(nm_run axi status) - if [ -n "$RUN_OUT" ]; then - run_branch=$(strip_quotes "$(nm_field branch)") - if [ -n "$run_branch" ] && [ "$run_branch" = "$CREW_BRANCH" ]; then - HAVE_RUN=1 - else - # The active-or-most-recent run is for another branch (the CLI is alive - # and answered; only the attribution missed) - try the coarse fallback. - # Deliberately nested inside `[ -n "$RUN_OUT" ]`: an empty/timed-out - # primary call means the CLI itself did not respond, so retrying it - # immediately with a second bounded call would just double the wait - # for no better answer. - coarse_run=$(nm_runs_status_for_branch "$CREW_BRANCH") - if [ -n "$coarse_run" ]; then - COARSE_STATUS=${coarse_run%%|*} +if [ "$KIND" != ship ]; then + RUN_DISCOVERY=absent +elif [ -n "$CREW_BRANCH" ] && command -v no-mistakes >/dev/null 2>&1; then + if nm_run_capture RUN_OUT axi status; then + if [ -z "$RUN_OUT" ]; then + RUN_DISCOVERY=absent + elif nm_status_output_valid; then + run_branch=$(strip_quotes "$(nm_field branch)") + if [ "$run_branch" = "$CREW_BRANCH" ]; then HAVE_RUN=1 - RUN_SOURCE=coarse + RUN_DISCOVERY=found + else + if coarse_run=$(nm_runs_status_for_branch "$CREW_BRANCH"); then + COARSE_STATUS=${coarse_run%%|*} + HAVE_RUN=1 + RUN_DISCOVERY=found + RUN_SOURCE=coarse + else + case "$?" in + 1) RUN_DISCOVERY=absent ;; + *) RUN_DISCOVERY=unknown ;; + esac + fi fi + else + RUN_DISCOVERY=unknown fi fi fi @@ -892,6 +931,8 @@ fi # liveness, so a finished-but-pane-closed crewmate never reaches here. Down here there # is no run to consult, so a dead/unreadable target means the crewmate is gone: report # unknown rather than trusting a possibly-stale status log as the current state. +[ "$RUN_DISCOVERY" = absent ] \ + || emit unknown none "no-mistakes run lookup unavailable for ${CREW_BRANCH:-this lane}" [ -n "$BACKEND_TARGET" ] || emit unknown none "no backend target recorded" pane_readable "$BACKEND_TARGET" || emit unknown none "backend target gone: $BACKEND_TARGET" @@ -917,9 +958,16 @@ fi # `unknown` verdict as the "not a state" test needs no second verb list here. if [ -n "$LOG_VERB" ]; then LOG_STATE=$(map_log_state "$LOG_LINE") - if [ "$LOG_STATE" != unknown ]; then - emit "$LOG_STATE" status-log "$(status_line_note "$LOG_LINE")" - fi + case "$LOG_STATE" in + working) + [ "$KIND" = secondmate ] \ + && emit working status-log "$(status_line_note "$LOG_LINE")" + emit wedged status-log "stopped without positive working evidence; last event: $(status_line_note "$LOG_LINE")" + ;; + unknown) ;; + *) emit "$LOG_STATE" status-log "$(status_line_note "$LOG_LINE")" ;; + esac fi -emit unknown none "no current-state source available" +[ "$KIND" = secondmate ] && emit unknown none "idle secondmate with no current-state event" +emit unknown none "idle lane with no current-state event" diff --git a/bin/fm-supervise-daemon.sh b/bin/fm-supervise-daemon.sh index 2cf08fc1b05..d3990fb00db 100755 --- a/bin/fm-supervise-daemon.sh +++ b/bin/fm-supervise-daemon.sh @@ -330,7 +330,7 @@ _collapse_newlines() { # classify_signal() { # local reason=$1 state=$2 f last distilled="" rel="" all_seen=1 task seen - local current liveness liveness_rel="" liveness_seen="" + local current liveness pause_class liveness_rel="" liveness_seen="" for f in $reason; do [ -e "$f" ] || continue last=$(last_status_line "$f") @@ -358,8 +358,21 @@ classify_signal() { # fi [ -n "$last" ] || continue distilled="${distilled}$(basename "$f"): ${last} | " - status_is_captain_relevant "$last" || continue - rel=1 + if status_is_paused "$last"; then + if ! crew_declared_pause_absorbable "$task" "$last" "$state"; then + rel=1 + else + pause_class=$(crew_absorb_class "$task" "$last" "$current" "$state") + case "$pause_class" in + paused|working) continue ;; + *) rel=1 ;; + esac + fi + elif status_is_captain_relevant "$last"; then + rel=1 + else + continue + fi # Dedupe against the catch-all scan: if this status was already escalated # (seen marker matches), skip escalating again. The seen marker is the # single source of truth shared between the per-wake signal path and the @@ -387,7 +400,7 @@ classify_signal() { # # first sight of a non-terminal stale it returns "self" and the caller records a # timestamp marker; persistence is escalated by housekeeping's recheck, not here. classify_stale() { # - local win=$1 state=$2 task last seen current liveness + local win=$1 state=$2 task last seen current liveness pause_class task=$(window_to_task "$win" "$state") current=$(crew_state_line "$task") liveness=$(crew_state_liveness_verdict "$current") @@ -399,15 +412,19 @@ classify_stale() { # ;; esac last=$(last_status_line "$state/$task.status") - if [ -n "$last" ] && status_is_paused "$last"; then + pause_class=$(crew_absorb_class "$task" "$last" "$current" "$state") + if [ -n "$last" ] && [ "$pause_class" = paused ]; then # A DECLARED external-wait pause (fm-classify-lib.sh): an idle pane is EXPECTED, # so this is not a wedge. The caller records a pause marker (long re-surface - # cadence in housekeeping) rather than a wedge stale marker. Cheap: reuses the - # status line already read, no fm-crew-state.sh call, mirroring the daemon's - # existing status-log classification. + # cadence in housekeeping) rather than a wedge stale marker. The current-state + # read above and the shared durable stream proof jointly establish this result. printf 'pause|paused (awaiting external), rechecked on a long cadence: %s' "$last" return fi + if [ -n "$last" ] && status_is_paused "$last"; then + printf 'escalate|declared pause is not absorbable because its durable status stream is unresolved or changed: %s' "$last" + return + fi if [ -n "$last" ] && status_is_captain_relevant "$last"; then # Dedupe against the signal path: if this status was already escalated # (seen marker matches), self-handle to avoid a duplicate in the digest. @@ -500,13 +517,18 @@ clear_pause_tracking() { # "$state/.stale-$watcher_key" "$state/.stale-since-$watcher_key" "$state/.wedge-escalations-$watcher_key" } -reconcile_pause_tracking() { # - local win=$1 state=$2 last=$3 task key marker watcher_key +reconcile_pause_tracking() { # [pause-class] + local win=$1 state=$2 last=$3 task key marker watcher_key pause_class task=$(window_to_task "$win" "$state") key=$(_stale_key "$task") marker="$state/.subsuper-paused-$key" watcher_key=$(_stale_key "$win") - if status_is_paused "$last"; then + if [ "$#" -ge 4 ]; then + pause_class=$4 + else + pause_class=$(crew_absorb_class "$task" "$last" "" "$state") + fi + if [ "$pause_class" = paused ]; then stale_marker_remove "$win" "$state" pause_marker_record "$win" "$state" elif [ -e "$marker" ] || [ -e "$state/.paused-$watcher_key" ]; then @@ -524,7 +546,8 @@ migrate_watcher_pause_markers() { # key=$(_stale_key "$task") watcher_key=$(_stale_key "$win") last=$(last_status_line "$state/$task.status") - if status_is_paused "$last" || [ -e "$state/.subsuper-paused-$key" ] || [ -e "$state/.paused-$watcher_key" ]; then + if status_is_paused "$last" \ + || [ -e "$state/.subsuper-paused-$key" ] || [ -e "$state/.paused-$watcher_key" ]; then reconcile_pause_tracking "$win" "$state" "$last" fi done @@ -985,7 +1008,7 @@ _oldest_line_age() { # -> seconds since the oldest buffered item first ar # 3) heartbeat scan: every HEARTBEAT_SCAN_SECS, grep state/*.status for a # captain-relevant line the per-wake classifier missed and escalate it. housekeeping() { # - local state=$1 now due f key task win marker age last max_defer oldest pause_secs + local state=$1 now due f key task win marker age last owner clears max_defer oldest pause_secs pause_class now=$(_now) migrate_watcher_pause_markers "$state" @@ -1035,8 +1058,9 @@ housekeeping() { # fi task=$(window_to_task "$win" "$state") last=$(last_status_line "$state/$task.status") - if [ -n "$last" ] && status_is_paused "$last"; then - reconcile_pause_tracking "$win" "$state" "$last" + pause_class=$(crew_absorb_class "$task" "$last" "" "$state") + if [ -n "$last" ] && [ "$pause_class" = paused ]; then + reconcile_pause_tracking "$win" "$state" "$last" "$pause_class" continue fi age=$(( now - $(cat "$marker" 2>/dev/null || echo "$now") )) @@ -1066,8 +1090,9 @@ housekeeping() { # fi task=$(window_to_task "$win" "$state") last=$(last_status_line "$state/$task.status") - if [ -z "$last" ] || ! status_is_paused "$last"; then - reconcile_pause_tracking "$win" "$state" "$last" + pause_class=$(crew_absorb_class "$task" "$last" "" "$state") + if [ -z "$last" ] || [ "$pause_class" != paused ]; then + reconcile_pause_tracking "$win" "$state" "$last" "$pause_class" continue fi age=$(( now - $(cat "$marker" 2>/dev/null || echo "$now") )) @@ -1078,8 +1103,10 @@ housekeeping() { # 2) rm -f "$marker" ;; *) last=$(last_status_line "$state/$task.status") - if [ -n "$last" ] && status_is_paused "$last"; then - escalate_add "$state" "paused ${age}s (awaiting external, recheck whether the wait still holds): $win" + if [ -n "$last" ] && [ "$(crew_absorb_class "$task" "$last" "" "$state")" = paused ]; then + owner=$(status_pause_owner "$last") + clears=$(status_pause_clearing_condition "$last") + escalate_add "$state" "paused ${age}s (owner=$owner, clears=$clears; if the clearing condition now holds, resume or surface the still-paused lane): $win" _now > "$marker" else rm -f "$marker" diff --git a/bin/fm-watch-checkpoint.sh b/bin/fm-watch-checkpoint.sh index 9d153b1f692..d47882ee840 100755 --- a/bin/fm-watch-checkpoint.sh +++ b/bin/fm-watch-checkpoint.sh @@ -86,6 +86,22 @@ run_with_perl_timeout() { ' "$SECONDS_ARG" "$SCRIPT_DIR/fm-watch.sh" } +cleanup_timed_out_watcher_lock() { + local checkpoint_lock + FM_ROOT="${FM_ROOT_OVERRIDE:-$(cd "$SCRIPT_DIR/.." && pwd)}" + FM_HOME="${FM_HOME:-${FM_ROOT_OVERRIDE:-$FM_ROOT}}" + STATE="${FM_STATE_OVERRIDE:-$FM_HOME/state}" + # fm-wake-lib.sh owns the root/home/state resolution rule; sourcing it here reuses + # that one owner instead of keeping a second copy in step with it. + # shellcheck source=bin/fm-wake-lib.sh + . "$SCRIPT_DIR/fm-wake-lib.sh" || return 1 + checkpoint_lock="$STATE/.watch.lock" + if fm_lock_try_acquire "$checkpoint_lock"; then + fm_lock_release "$checkpoint_lock" + fi + [ ! -e "$checkpoint_lock" ] && [ ! -L "$checkpoint_lock" ] +} + set +e if command -v timeout >/dev/null 2>&1; then timeout --kill-after=2 "$SECONDS_ARG" "$SCRIPT_DIR/fm-watch.sh" >"$OUT" 2>"$ERR" @@ -113,6 +129,10 @@ if grep -E '^watcher: already running' "$OUT" "$ERR" >/dev/null 2>&1; then fi if [ "$RC" -eq 124 ]; then + cleanup_timed_out_watcher_lock || { + echo "checkpoint: timed-out watcher lock is still live or could not be reclaimed" >&2 + exit 1 + } printf 'checkpoint: no actionable wake within %ss\n' "$SECONDS_ARG" exit 124 fi diff --git a/bin/fm-watch.sh b/bin/fm-watch.sh index 10dd64880e1..23a9be5bc72 100755 --- a/bin/fm-watch.sh +++ b/bin/fm-watch.sh @@ -168,7 +168,7 @@ PERMISSION_STALL_ESCALATE_SECS=${FM_PERMISSION_STALL_ESCALATE_SECS:-900} case "$PERMISSION_STALL_ESCALATE_SECS" in ''|*[!0-9]*|0) PERMISSION_STALL_ESCALATE_SECS=900 ;; esac -# A crewmate that DECLARED a pause (paused: , fm-classify-lib.sh) is idling on +# A crewmate that declared an owned-and-clearing pause (fm-classify-lib.sh) is idling on # a known external wait, so its stale pane is absorbed rather than wedge-escalated; # it re-surfaces once for a recheck every PAUSE_RESURFACE_SECS - far longer than the # wedge threshold, but finite so a forgotten pause cannot rot invisibly. @@ -339,13 +339,14 @@ FM_WEDGE_DEMAND_INSPECT_COUNT=${FM_WEDGE_DEMAND_INSPECT_COUNT:-3} # Repeat-poll wedge-timer bookkeeping for an already-classified stale hash # absorbed as provably-working - repairs a missing/corrupt timer (self-heals a # watcher restart between recording the hash and recording the timer), or -# escalates once STALE_ESCALATE_SECS have elapsed. Never re-reads the crewmate -# state (the costly check already ran once, at classification time). Shared by +# escalates once STALE_ESCALATE_SECS have elapsed. At that boundary it re-reads +# only the measured-progress verdict so a starved process accumulating CPU is not +# called wedged from pane staleness alone. Shared by # both places a hash can be absorbed this way: the plain non-terminal path, # and the stale_is_terminal-overridden path (a captain-relevant status-log # line that an active run/busy pane outranked). wedge_timer_check() { # - local win=$1 since_file=$2 label=$3 escalation_file=$4 since age n reason + local win=$1 since_file=$2 label=$3 escalation_file=$4 since age n reason task since=$(cat "$since_file" 2>/dev/null || true) case "$since" in ''|*[!0-9]*) @@ -355,6 +356,12 @@ wedge_timer_check() { # "$since_file" + triage_log "absorbed stale (measured worktree CPU/process progress at wedge threshold): $win" + return + fi n=$(( $(cat "$escalation_file" 2>/dev/null || echo 0) + 1 )) echo "$n" > "$escalation_file" reason="stale: $win (idle ${age}s, possible wedge, escalation $n)" @@ -369,7 +376,7 @@ wedge_timer_check() { # flag), so @@ -380,19 +387,22 @@ wedge_timer_check() { # and flags the key paused. handle_paused_stale() { # - local win=$1 task=$2 h=$3 key statusf mtime age rf rf_age reason + local win=$1 task=$2 h=$3 key statusf last owner clears mtime age rf rf_age reason key=$(printf '%s' "$win" | tr ':/.' '___') printf '%s' "$h" > "$STATE/.stale-$key" - : > "$STATE/.paused-$key" - rm -f "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key" statusf="$STATE/$task.status" + last=$(last_status_line "$statusf") + printf '%s\n' "$last" > "$STATE/.paused-$key" + rm -f "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key" mtime=$(stat_mtime "$statusf") case "$mtime" in ''|*[!0-9]*) mtime=$(date +%s) ;; esac age=$(( $(date +%s) - mtime )) rf="$STATE/.paused-resurfaced-$key" rf_age=$(age_of "$rf") # 999999 when no prior re-surface if [ "$age" -ge "$PAUSE_RESURFACE_SECS" ] && [ "$rf_age" -ge "$PAUSE_RESURFACE_SECS" ]; then - reason="stale: $win (paused ${age}s, awaiting external - declared pause, rechecked on a long cadence not a wedge; confirm the wait still holds)" + owner=$(status_pause_owner "$last") + clears=$(status_pause_clearing_condition "$last") + reason="stale: $win (paused ${age}s, owner=$owner, clears=$clears - bounded pause recheck, not a wedge; if the clearing condition now holds, resume or surface the still-paused lane)" fm_wake_append stale "$win" "$reason" || exit 1 wake "$reason" "$rf" fi @@ -416,6 +426,46 @@ clear_pause_tracking() { # rm -f "$STATE/.stale-$key" "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key" } +# Register a valid pause declaration as soon as its status signal is observed. +# The initial signal still surfaces once, but marker creation must happen before +# that return; otherwise a freshly re-armed watcher loses the declaration and +# repeatedly enters stale handling. Registration proves only the pause CONTRACT +# (the line itself), never the keyed open-decision fold, so it deliberately does not +# write the .paused-rechecked- proof signature. Any stale signature from an +# earlier declaration is dropped, leaving the first stale classification to establish +# a fresh one. +register_pause_declaration() { # + local win=$1 last=$2 key + key=$(printf '%s' "$win" | tr ':/.' '___') + printf '%s\n' "$last" > "$STATE/.paused-$key" + rm -f "$STATE/.paused-rechecked-$key" "$STATE/.paused-resurfaced-$key" \ + "$STATE/.stale-since-$key" "$STATE/.wedge-escalations-$key" +} + +# Watcher-local signal-path reconciliation. Named apart from the away-mode daemon's +# own sync_pause_markers_from_signal (bin/fm-supervise-daemon.sh), which takes a +# different argument shape: both files are routinely sourced for their pure +# functions, and a shared name would silently redefine one of them. +register_pause_markers_from_signal() { # + local paths=$1 f task meta win last key + for f in $paths; do + case "$f" in "$STATE"/*.status) ;; *) continue ;; esac + [ -e "$f" ] || continue + task=${f##*/}; task=${task%.status} + meta="$STATE/$task.meta" + [ -f "$meta" ] || continue + win=$(fm_backend_target_of_meta "$meta") + [ -n "$win" ] || continue + last=$(last_status_line "$f") + if status_is_paused "$last"; then + register_pause_declaration "$win" "$last" + else + key=$(printf '%s' "$win" | tr ':/.' '___') + [ -e "$STATE/.paused-$key" ] && clear_pause_tracking "$win" + fi + done +} + pause_state_class() { # local win=$1 task=$2 key last recheck_file statusf sig class key=${win//:/_} @@ -429,22 +479,7 @@ pause_state_class() { # crew_absorb_class "$task" return fi - # Cached verdict: skip the costly authoritative re-read (an fm-crew-state.sh call - # plus the keyed open/resolved fold) while a recent recheck still stands AND the - # status stream it was taken from is byte-identical. Both halves of the pause proof - # are pure functions of that stream, so an unchanged signature means the earlier - # verdict is still exactly as true - while any append, including one that OPENS a - # decision after the pause flag was written, invalidates the cache and forces a - # fresh proof rather than riding out the age window behind a stale `paused`. - # The recheck marker carries the signature as its content; callers read only its - # mtime for the age bound, and a marker left by an older watcher simply mismatches - # and re-proves. sig=$(stat_sig "$statusf") - if [ -e "$STATE/.paused-$key" ] && [ "$(age_of "$recheck_file")" -lt "$STALE_ESCALATE_SECS" ] \ - && [ -n "$sig" ] && [ "$sig" = "$(cat "$recheck_file" 2>/dev/null || true)" ]; then - printf 'paused' - return - fi class=$(crew_absorb_class "$task" "$last") case "$class" in paused) @@ -946,16 +981,14 @@ event_wait_or_sleep() { # machinery already understands it (queued by key=window, so a later poll-path # stale for the same pane collapses on drain). handle_push_transition() { # - local backend=$1 session=$2 record=$3 pane_id to window task key h reason + local backend=$1 session=$2 record=$3 pane_id to window task key h reason last pane_id=$(fm_transition_pane_id "$record") to=$(fm_transition_to_status "$record") [ -n "$pane_id" ] || { sleep 1; return; } window="$session:$pane_id" task=$(window_to_task "$window" "$STATE") - if status_is_paused "$(last_status_line "$STATE/$task.status")"; then - # The durable status is the trusted gate on this native edge. Commit the - # handled transition, then enter the shared pause path so it owns the marker - # and bounded re-surface cadence even when no auxiliary crew-state read exists. + last=$(last_status_line "$STATE/$task.status") + if status_is_paused "$last" && [ "$(pause_state_class "$window" "$task")" = paused ]; then key=$(printf '%s' "$window" | tr ':/.' '___') h=$(cat "$STATE/.hash-$key" 2>/dev/null || true) fm_backend_commit_transition "$backend" "$STATE" "$session" "$record" || exit 1 @@ -1084,6 +1117,7 @@ while :; do $pending EOF reason="signal:$files" + register_pause_markers_from_signal "$files" # A scout's terminal done event is its automatic reaping trigger. Ship work # waits for the separate merged-PR or approved local-merge authority. for f in $files; do @@ -1279,7 +1313,7 @@ EOF # - working: an actively-running pipeline legitimately sits on a static # pane (e.g. waiting on CI), so absorb and start the wedge timer so a # genuinely frozen run still escalates past STALE_ESCALATE_SECS; - # - paused: the crewmate DECLARED an external wait (paused:), so absorb on the + # - paused: the crewmate declared an owned-and-clearing wait, so absorb on the # long PAUSE_RESURFACE_SECS recheck cadence instead of wedge-escalating; # - none: no running pipeline, idle pane, no busy signature, no declared # pause - the crewmate has STOPPED. Surface immediately so firstmate peeks @@ -1288,7 +1322,7 @@ EOF # wait out the timer. if [ "$(cat "$sf" 2>/dev/null || true)" != "$h" ]; then task=$(window_to_task "$w" "$STATE") - case "$(crew_absorb_class "$task")" in + case "$(crew_absorb_class "$task" "$last")" in working) clear_pause_tracking "$w" printf '%s' "$h" > "$sf" diff --git a/docs/architecture.md b/docs/architecture.md index 175070883e2..8d63d03e50d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -15,14 +15,26 @@ The permission-prompt matcher and the explicitly heuristic macOS system-dialog f Repeated unchanged wedge or permission-stall escalations add an escalation count to the wake reason and, at `FM_WEDGE_DEMAND_INSPECT_COUNT`, a `demand-deep-inspection` marker. Those actionable wakes are written to a durable local queue (`state/.wake-queue`) before detector state advances, so a missed process exit can be recovered by draining the queue. No-verb wakes, such as `working:` notes and bare turn-ended signals, are benign only when `bin/fm-crew-state.sh` reports positive evidence that the crewmate is still working: an actively running no-mistakes step for that crewmate's branch or a backend busy signature. -A crewmate that declares `paused:` for a known external wait is separately absorbed while idle and re-surfaced only on the longer pause cadence, rather than being treated as a possible wedge. -A pause is a statement about the work rather than about the terminal, so it is honoured whether the crewmate's pane is alive, idle, or gone, and whatever its attributed no-mistakes run reports - parked, failed or cancelled, or unreadable. -The single exception is an actively `working` run-step or busy pane, which supersedes the declaration because the crewmate resumed after making it. -Absorption is gated on two proofs taken from one immutable read of the crewmate's current durable status stream: the pause verb carrying no failure vocabulary in its headline, and an empty keyed open/resolved fold, so a pause can never mask a still-unanswered decision. -The Herdr native blocked-transition edge does not yet honour this invariant, which is a known defect tracked as `herdr-push-transition-pause-gate-h8`: on that edge a lane that owes an unanswered keyed decision can be silently absorbed and go quiet. -A crewmate with no locatable status stream is refused rather than absorbed, and stopped crewmates without a declared pause surface immediately. +When such a pane stays static through `FM_STALE_ESCALATE_SECS`, the watcher re-reads the step-liveness verdict before escalating. +The step-liveness probe owns the CPU-versus-elapsed measurement and threshold: `alive` means worktree processes measurably advanced CPU or process membership, while presence without established progress is graded `unknown` rather than called alive. +Only `alive` refreshes the bounded wedge observation; every other verdict remains actionable, so process presence alone never suppresses a wedge. + +The pause contract is `paused: ; owner=; clears=`. +The owner may be a person, team, service, or other accountable role, and `clears` must name a fact, event, or time a supervisor can observe rather than a vague intention. +Only a non-failure declaration carrying both non-placeholder fields earns pause absorption. +A bare pause verb, a missing owner, or a missing clearing condition is actionable stopped work and receives no marker or pause cadence. +The normal watcher writes `.paused-` and clears any stale wedge timing before returning the declaration's initial one-time signal wake, so re-arming cannot lose the state transition. +Registration proves only the declaration itself, so it never seeds the `.paused-rechecked-` proof signature; the first stale classification writes that signature after the keyed open-decision fold passes, while every later classification still rechecks authoritative run state. +An owned pause is separately absorbed while idle and re-surfaced only on the longer pause cadence, rather than being treated as a possible wedge. +A pause is a statement about the work rather than about terminal liveness, but authoritative run state still constrains it. +Every matching no-mistakes run verdict supersedes the declaration because a pause gates starting new validation work and cannot suspend an in-flight run. +Absorption is gated on two proofs taken from one immutable read of the crewmate's current durable status stream: an owned-and-clearing declaration carrying no failure vocabulary in any of its prose-clause headlines, and an empty keyed open/resolved fold, so a pause can never mask a still-unanswered decision. +The Herdr native blocked-transition edge uses this same proof and precedence boundary before absorbing a transition. +A crewmate with no locatable status stream is refused rather than absorbed, and stopped crewmates without a valid declaration surface immediately. +Inside `bin/fm-crew-state.sh` a declared pause becomes the reported current state only when no matching run supplies current state. Pause cadence markers remain in force while the latest durable status still declares the pause and are cleared only after that status resumes, so every continuously declared pause still re-surfaces on the bounded long cadence. Its initial normal-mode status signal still surfaces through the no-verb path, while away mode self-handles that routine signal and owns the later recheck. +The bounded recheck includes the named owner and clearing condition, so a supervisor that observes the condition has passed can resume or surface a lane whose declaration still persists. Fresh stale panes use the same current-state read before trusting the status log, so an active run or busy pane outranks an old captain-relevant status-log line left behind before validation. Heartbeats are benign only when the fleet scan finds neither an unsurfaced captain-relevant status nor a `dead` or `unknown` command-step liveness observation. Absorbed wakes advance their suppression markers, log to `state/.watch-triage.log`, and keep the watcher blocking without a queue record or LLM turn. @@ -35,9 +47,11 @@ When a terminal run reports `outcome: passed`, the helper verifies the PR detail An open-PR `passed` or `checks-passed` outcome and a checks-green CI marker are classified by the exact remote-only currentness contract in `bin/fm-crew-state.sh`'s header; only `done` authorizes the PR-ready workflow. During no-mistakes' `ci` monitor phase, it also reads the ci step log tail because `axi status` reports both "still waiting on checks" and "checks green, waiting on merge" as `ci,running`. The most recent recognized ci log marker wins, so checks-green monitoring supplies the ready claim while a later re-arm, failed-check, or issue marker returns the crewmate to working before remote currentness is considered. -Only when no matching run exists does it fall back to the pane busy-signature and then a status-log event whose verb maps to a recognized run-state; a dead pane without a run reports unknown instead of trusting a stale log. +Only when run discovery affirmatively proves no matching run exists does it fall back to the pane busy-signature and then a status-log event whose verb maps to a recognized run-state; a timeout, malformed response, or other lookup uncertainty reports unknown instead of trusting the pane or a stale log. Decision-only events such as `resolved` never become current state or leak their prose into the current-state detail. -In that status-log fallback, a declared external wait reports the distinct `paused` state with its reason. +In that status-log fallback, a valid external wait reports `paused` with its reason, owner, and clearing condition. +An idle ordinary crewmate whose last event still claims work in progress, with no positive working evidence and no valid pause, reports `wedged`; decision-only events, unrecognized verbs, and evidence-free silence remain `unknown`. +Neither state is treated as healthy. For herdr, that pane fallback trusts a native `busy` verdict outright, but corroborates native `idle` or unknown verdicts against the rendered busy signature before deciding the crewmate is not working. For whole-fleet read-only review, `bin/fm-fleet-snapshot.sh --json` emits schema `fm-fleet-snapshot.v1` from the backlog, task metadata, current crewmate state, endpoint probes, PR/report pointers, scout reports, the bounded landed-work roll-up from registered secondmate homes, and secondmate return-channel guidance. `bin/fm-fleet-view.sh` renders that snapshot as Markdown for humans, while `bin/fm-bearings-snapshot.sh` provides the bounded bearings projection, so both views consume one structured contract instead of reparsing raw fleet files. diff --git a/docs/configuration.md b/docs/configuration.md index a28f7bb91eb..ae0dcd8aaac 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -712,7 +712,7 @@ FM_WATCHER_STALE_GRACE=300 # defaults to FM_GUARD_GRACE; seconds a live watche FM_SIGNAL_GRACE=30 # seconds to coalesce nearby status and turn-end signals into one wake FM_CAPTAIN_RE='done:|needs-decision:|blocked:|failed:|PR ready|checks green|ready in branch|merged' # status regex that makes watcher and daemon signal/stale/scan output captain-relevant FM_CLASSIFY_PAUSED_VERB=paused # leading status verb for a declared external wait; excluded from FM_CAPTAIN_RE and distinct from blocked -FM_STALE_ESCALATE_SECS=240 # idle seconds before a provably-working stale pane escalates; stale panes whose crewmate is not provably working surface immediately unless they declare the pause verb +FM_STALE_ESCALATE_SECS=240 # idle seconds before a provably-working stale pane escalates; stale panes whose crewmate is not provably working surface immediately unless their latest status is a valid declared pause FM_PERMISSION_STALL_ESCALATE_SECS=900 # busy seconds without meaningful pane/status/turn-end progress before a possible macOS permission/system-dialog block surfaces; timeout heuristic, not direct OS detection FM_PAUSE_RESURFACE_SECS=3600 # seconds before an idle declared external wait re-surfaces for a recheck in the watcher or away-mode daemon FM_WEDGE_DEMAND_INSPECT_COUNT=3 # consecutive unchanged wedge or permission-stall escalations before demand-deep-inspection is added diff --git a/docs/herdr-backend.md b/docs/herdr-backend.md index 98844cce3f6..3e42b86507d 100644 --- a/docs/herdr-backend.md +++ b/docs/herdr-backend.md @@ -857,8 +857,7 @@ This is the follow-up the former "No `events.subscribe` native push" gap note de **Mechanism (one owner per contract).** `bin/fm-transition-lib.sh` owns the backend-neutral normalized-transition record shape and the single-owner status->action policy table (`fm_transition_policy`: `blocked`=actionable, `working`=absorb-and-clear-dedupe, `idle`/`done`=defer, anything else=fall back to polling). `bin/backends/herdr.sh` (`fm_backend_herdr_wait_transition`) subscribes to `pane.agent_status_changed` for this home's herdr panes over ONE raw `AF_UNIX` connection via `bin/backends/herdr-eventwait.py`, subscribing to ALL statuses (so `working` edges clear the per-pane dedupe marker) and returning the first fresh `blocked` edge; after the subscription acknowledgement it level-reconciles each pane's current state while the stream remains live, so a pane that went blocked during the gap is caught once and transitions during reconciliation are buffered. -`bin/fm-watch.sh` splices this in as the watcher's terminal wait (`event_wait_or_sleep`, replacing the blind `sleep POLL` for push-capable homes): on a returned `blocked` it maps `pane_id -> : -> task`, exempts `kind=secondmate` endpoints and waits whose current status line has the `paused:` verb, and enqueues an immediate `stale` wake otherwise. -This native edge violates the pause absorption contract owned by [`architecture.md`](architecture.md) because it does not prove that the keyed open/resolved fold is empty before absorbing the transition; this is a known defect tracked as `herdr-push-transition-pause-gate-h8`, and it can silently absorb a lane that owes an unanswered keyed decision and let that lane go quiet. +`bin/fm-watch.sh` splices this in as the watcher's terminal wait (`event_wait_or_sleep`, replacing the blind `sleep POLL` for push-capable homes): on a returned `blocked` it maps `pane_id -> : -> task`, exempts `kind=secondmate` endpoints, absorbs a pause only after the shared declaration, current-state precedence, and keyed open/resolved proofs pass, and enqueues an immediate `stale` wake otherwise. There is no second watcher process: the reader is a short-lived subprocess of the single watcher, so the "exactly one live supervision cycle" invariant and every guard/beacon/arm/turn-end mechanism are unchanged. **Polling is the permanent fail-closed backstop.** @@ -950,7 +949,7 @@ Covered by the unit cases in `tests/fm-afk-launch.test.sh` (clear-on-fresh-entry Herdr 0.7.3 preserves the harness's own de-emphasis style (dim/faint and truecolor foreground) in `pane read --format ansi`, and `fm_backend_herdr_composer_state` extracts real typed content with the shared `fm_composer_strip_ghost` (`bin/fm-composer-lib.sh`), which drops dim/faint AND dark-truecolor runs to distinguish ghost suggestions/placeholders from real typed text. If a future herdr build strips ANSI style from `--format ansi`, the classifier loses its ghost signal and falls back to reading the suggestion text as `pending` - the fail-safe direction for terminal-backed compatibility injection (it defers rather than risks overwriting a human draft), which the max-defer alarm then surfaces. - **RESOLVED: a "paused / awaiting-external" crewmate state for the stale-wedge escalation.** Raised alongside the 2026-07-07 incident: an in-flight crewmate intentionally idling on a known external wait (a vendor rate limit, say) still tripped `bin/fm-supervise-daemon.sh`'s "stale persisted ... (possible wedge)" escalation exactly like a genuinely wedged crewmate, with no way to mark the wait as expected. - Fixed by the `paused:` external-wait verb: a crewmate declares a deliberate wait, and both `bin/fm-watch.sh` and `bin/fm-supervise-daemon.sh` absorb its idle pane through the shared `bin/fm-classify-lib.sh` vocabulary (`status_is_paused`, `crew_absorb_class`, `FM_PAUSE_RESURFACE_SECS`), re-surfacing it for a recheck on a long cadence instead of a wedge escalation. + Fixed by the owned-and-clearing `paused:` external-wait contract: a crewmate declares the wait, owner, and observable clearing condition, and both `bin/fm-watch.sh` and `bin/fm-supervise-daemon.sh` absorb its idle pane through the shared `bin/fm-classify-lib.sh` vocabulary (`status_is_paused`, `crew_absorb_class`, `FM_PAUSE_RESURFACE_SECS`), re-surfacing it for a recheck on a long cadence instead of a wedge escalation. See `AGENTS.md` section 8 and the crewmate-facing brief contract in `bin/fm-brief.sh`. - **Not implemented: mid-session secondmate liveness.** The `fm_backend_agent_alive`-driven respawn sweep (`bin/fm-bootstrap.sh`, see "Agent liveness probe reuses the husk classifier" above) only runs at session start. A secondmate dying mid-session is a harder follow-on: the watcher deliberately exempts secondmates from stale-pane detection (an idle secondmate pane is healthy by design), so catching a mid-session death would need a periodic liveness beacon distinct from that exemption, not implemented here. diff --git a/tests/fm-bearings-snapshot.test.sh b/tests/fm-bearings-snapshot.test.sh index ad49dda61f1..608be4f382b 100755 --- a/tests/fm-bearings-snapshot.test.sh +++ b/tests/fm-bearings-snapshot.test.sh @@ -118,7 +118,7 @@ EOF "harness=codex" \ "kind=ship" \ "mode=no-mistakes" - printf 'paused: declared external-wait for upstream release\n' > "$home/state/external-wait.status" + printf 'paused: declared external-wait for upstream release; owner=release team; clears=release artifact is published\n' > "$home/state/external-wait.status" # The secondmate's OWN home backlog records a merge it managed. This lands in the # secondmate home, never the main backlog, so landed-work views only see it via the # bounded cross-home Done roll-up. diff --git a/tests/fm-brief.test.sh b/tests/fm-brief.test.sh index 12ff4d995f6..5db582c673c 100755 --- a/tests/fm-brief.test.sh +++ b/tests/fm-brief.test.sh @@ -446,13 +446,15 @@ test_pause_verb_override_renders_all_brief_scaffolds() { assert_grep "States: working, needs-decision, blocked, awaiting, done, failed." "$brief" \ "$kind brief did not render the configured pause verb in its states list" # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded. - assert_grep 'Use `awaiting: {why}`' "$brief" \ + assert_grep 'Use `awaiting: {reason}; owner={named owner}; clears={observable condition}`' "$brief" \ "$kind brief did not instruct the configured pause status" # shellcheck disable=SC2016 # Literal backticks and braces must remain unexpanded. - assert_no_grep '`paused: {why}`' "$brief" \ + assert_no_grep '`paused: {reason}; owner={named owner}; clears={observable condition}`' "$brief" \ "$kind brief still instructs the default paused status" assert_grep 'or a blocker clears' "$brief" \ "$kind brief did not require durable resolution when a blocker clears" + assert_no_grep 'run={current run id}' "$brief" \ + "$kind brief still claimed a run id could make a pause override run state" done pass "fm-brief.sh: custom pause verb renders in every scaffold" } diff --git a/tests/fm-crew-state.test.sh b/tests/fm-crew-state.test.sh index 6bc9ed0c450..13a1355ac9f 100755 --- a/tests/fm-crew-state.test.sh +++ b/tests/fm-crew-state.test.sh @@ -70,8 +70,16 @@ case "${1:-}" in case "${1:-}" in status) shift - if [ "${1:-}" = --run ]; then printf '%s\n' "${FM_FAKE_AXI_STATUS_RUN:-}" - else printf '%s\n' "${FM_FAKE_AXI_STATUS:-}"; fi ;; + if [ "${1:-}" = --run ]; then + printf '%s\n' "${FM_FAKE_AXI_STATUS_RUN:-}" + else + case "${FM_FAKE_AXI_STATUS_MODE:-ok}" in + fail) exit 9 ;; + signal) kill -TERM "$$" ;; + timeout) sleep 30 ;; + esac + printf '%s\n' "${FM_FAKE_AXI_STATUS:-}" + fi ;; logs) printf '%s\n' "${FM_FAKE_CI_LOGS:-}" ;; esac @@ -205,8 +213,9 @@ case "${FM_FAKE_LIVENESS_MODE:-}" in empty-procs) printf 'liveness: unknown · run: 01RUN · procs: · grade: unreadable · missing count · doing: bash t.sh (1:00)\n' ;; argv-fields) printf 'liveness: unknown · run: 01RUN · procs: 3 · grade: present-unproven · presence established · doing: python -c "procs: x grade: bogus" (1:00)\n' ;; nonzero) exit 9 ;; + no-progress) printf 'liveness: unknown · run: 01RUN · procs: 2 · grade: present-no-progress · PRESENT BUT NOT PROGRESSING: stable membership and no persistent process advanced cpu in 30s (best +0.01s)\n' ;; timeout) sleep 30 ;; - *) printf 'liveness: alive · run: 01RUN · procs: 1 · processes present\n' ;; + *) printf 'liveness: alive · run: 01RUN · procs: 1 · persistent process advanced cpu over 30s (best +0.04s)\n' ;; esac SH chmod +x "$fb/no-mistakes" "$fb/tmux" "$fb/gh-axi" "$fb/git" "$fb/herdr" "$fb/fake-liveness" @@ -247,6 +256,7 @@ new_case() { # -> echoes case dir with an empty state/ # command-substitution assignment (SC2155). reset_fakes() { FM_FAKE_AXI_STATUS="" + FM_FAKE_AXI_STATUS_MODE=ok FM_FAKE_AXI_STATUS_RUN="" FM_FAKE_RUNS_LIST="" FM_FAKE_BUSY=0 @@ -261,7 +271,7 @@ reset_fakes() { FM_FAKE_REMOTE_COMMIT_HEAD=abc1234cafebabeabc1234cafebabeabc1234caf FM_FAKE_REMOTE_COMMIT_FAIL=0 FM_FAIL_ON_LOCAL_COMMIT_LOOKUP=0 - export FM_FAKE_AXI_STATUS FM_FAKE_AXI_STATUS_RUN FM_FAKE_RUNS_LIST FM_FAKE_BUSY FM_FAKE_TMUX_MISSING + export FM_FAKE_AXI_STATUS FM_FAKE_AXI_STATUS_MODE FM_FAKE_AXI_STATUS_RUN FM_FAKE_RUNS_LIST FM_FAKE_BUSY FM_FAKE_TMUX_MISSING export FM_FAKE_HERDR_BUSY FM_FAKE_HERDR_MISSING FM_FAKE_HERDR_AGENT_STATUS FM_FAKE_CI_LOGS export FM_FAKE_PR_STATE FM_FAKE_PR_HEAD FM_FAKE_GH_AXI_FAIL export FM_FAKE_REMOTE_COMMIT_HEAD FM_FAKE_REMOTE_COMMIT_FAIL FM_FAIL_ON_LOCAL_COMMIT_LOOKUP @@ -552,6 +562,13 @@ test_quiet_step_probe_failures_are_unknown() { assert_contains "$out" "probe unreadable: active step unavailable" "a missing active step explains its unreadable cause" FM_FAKE_AXI_STATUS="$(run_running_quiet_step fm/feat-qu test)" + out=$(FM_CREW_STATE_NM_LIVENESS_BIN="$probe" FM_FAKE_LIVENESS_MODE=no-progress \ + run_crew_state "$d" feat-qu) + assert_contains "$out" "liveness: unknown (grade: present-no-progress; 2 procs" \ + "the probe's near-zero CPU grade stays an explicit unknown, never a fourth verdict" + assert_contains "$out" "no persistent process advanced cpu in 30s (best +0.01s)" \ + "the present-no-progress grade preserves its CPU-versus-elapsed evidence" + out=$(FM_CREW_STATE_NM_LIVENESS_BIN="$probe" FM_FAKE_LIVENESS_MODE=empty \ run_crew_state "$d" feat-qu) assert_contains "$out" "liveness: unknown" "an empty probe result is visibly unknown" @@ -719,6 +736,39 @@ test_gate_block_parked_not_superseded() { pass "gate block parked run is not flagged superseded" } +test_matching_run_state_overrides_pause() { + reset_fakes + local d out pause + d=$(new_case run-pause-chronology) + make_repo_on_branch "$d/wt" fm/run-pause-chronology + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/run-pause-chronology.meta" \ + "window=fm:fm-run-pause-chronology" "worktree=$d/wt" "kind=ship" + pause='paused: awaiting release; owner=release team; clears=release artifact is published' + + FM_FAKE_AXI_STATUS="$(run_passed fm/run-pause-chronology)" + printf '%s; run=01RUN\n' "$pause" > "$d/state/run-pause-chronology.status" + out=$(run_crew_state "$d" run-pause-chronology) + assert_contains "$out" "state: done" "a completed run hid behind a matching-run pause" + + FM_FAKE_AXI_STATUS="$(run_parked fm/run-pause-chronology)" + out=$(run_crew_state "$d" run-pause-chronology) + assert_contains "$out" "state: parked" "a parked run hid behind a matching-run pause" + + FM_FAKE_AXI_STATUS="$(run_running fm/run-pause-chronology)" + out=$(run_crew_state "$d" run-pause-chronology) + assert_contains "$out" "state: working" "an active run hid behind its associated pause" + + FM_FAKE_AXI_STATUS="$(run_failed fm/run-pause-chronology)" + out=$(run_crew_state "$d" run-pause-chronology) + assert_contains "$out" "state: failed" "a failed run hid behind its associated pause" + + FM_FAKE_AXI_STATUS="$(run_completed_without_outcome fm/run-pause-chronology)" + out=$(run_crew_state "$d" run-pause-chronology) + assert_contains "$out" "state: unknown" "an unknown run hid behind its associated pause" + pass "every matching run state remains authoritative over a pause declaration" +} + test_ci_ready_done_log_beats_monitoring_run() { reset_fakes local d; d=$(new_case ci-ready) @@ -1463,8 +1513,8 @@ test_skipped_run_lookup_rejects_checks_green_log() { FM_FAKE_BUSY=0 local out; out=$(run_crew_state "$d" feat-detached-ready) assert_contains "$out" "state: unknown" "skipped run lookup with ready log -> unknown" - assert_contains "$out" "currentness is unavailable" "skipped lookup names missing currentness" - assert_contains "$out" "do not merge" "skipped lookup is merge-safe" + assert_contains "$out" "no-mistakes run lookup unavailable for this lane" "skipped lookup names missing currentness" + assert_not_contains "$out" "state: paused" "skipped lookup must not authorize a pause" assert_not_contains "$out" "state: done" "skipped lookup must not authorize done" pass "skipped run lookup rejects a stale checks-green log" } @@ -1493,7 +1543,7 @@ test_no_run_idle_pane_paused() { make_repo_on_branch "$d/wt" fm/feat-pause make_fakebin "$d" >/dev/null fm_write_meta "$d/state/feat-pause.meta" "window=fm:fm-feat-pause" "worktree=$d/wt" "kind=ship" - printf 'paused: holding for the upstream tool release\n' > "$d/state/feat-pause.status" + printf 'paused: holding for the upstream tool release; owner=tool maintainer; clears=release artifact is published\n' > "$d/state/feat-pause.status" FM_FAKE_AXI_STATUS="" FM_FAKE_BUSY=0 local out; out=$(run_crew_state "$d" feat-pause) @@ -1503,13 +1553,83 @@ test_no_run_idle_pane_paused() { pass "no run + idle pane on a paused: status reports state: paused with its reason" } +test_unreadable_run_lookup_rejects_pause() { + reset_fakes + local d out + d=$(new_case unreadable-run-paused) + make_repo_on_branch "$d/wt" fm/unreadable-run-paused + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/unreadable-run-paused.meta" \ + "window=fm:fm-unreadable-run-paused" "worktree=$d/wt" "kind=ship" + printf 'paused: awaiting release; owner=release team; clears=release artifact is published\n' \ + > "$d/state/unreadable-run-paused.status" + FM_FAKE_AXI_STATUS='not a run status response' + FM_FAKE_BUSY=0 + out=$(run_crew_state "$d" unreadable-run-paused) + assert_contains "$out" "state: unknown" "malformed run lookup -> unknown" + assert_contains "$out" "run lookup unavailable" "malformed run lookup names the unavailable proof" + assert_not_contains "$out" "state: paused" "malformed run lookup trusted a stale pause" + + FM_FAKE_AXI_STATUS="" + FM_FAKE_AXI_STATUS_MODE=fail + out=$(run_crew_state "$d" unreadable-run-paused) + assert_contains "$out" "state: unknown" "failed run lookup -> unknown" + assert_not_contains "$out" "state: paused" "failed run lookup trusted a stale pause" + + FM_FAKE_AXI_STATUS_MODE=ok + FM_FAKE_AXI_STATUS="$(run_running fm/some-other)" + FM_FAKE_RUNS_LIST='not a runs-list row' + out=$(run_crew_state "$d" unreadable-run-paused) + assert_contains "$out" "state: unknown" "malformed runs-list fallback -> unknown" + assert_not_contains "$out" "state: paused" "malformed runs-list fallback trusted a stale pause" + pass "unreadable run discovery cannot authorize a status-log pause" +} + +test_timed_out_run_lookup_rejects_pause() { + reset_fakes + local d out + d=$(new_case timed-out-run-paused) + make_repo_on_branch "$d/wt" fm/timed-out-run-paused + make_fakebin "$d" >/dev/null + fm_write_meta "$d/state/timed-out-run-paused.meta" \ + "window=fm:fm-timed-out-run-paused" "worktree=$d/wt" "kind=ship" + printf 'paused: awaiting release; owner=release team; clears=release artifact is published\n' \ + > "$d/state/timed-out-run-paused.status" + FM_FAKE_AXI_STATUS_MODE=timeout + FM_FAKE_BUSY=0 + out=$(FM_CREW_STATE_NM_TIMEOUT=1 run_crew_state "$d" timed-out-run-paused) + assert_contains "$out" "state: unknown" "timed-out run lookup -> unknown" + assert_contains "$out" "run lookup unavailable" "timed-out run lookup names the unavailable proof" + assert_not_contains "$out" "state: paused" "timed-out run lookup trusted a stale pause" + pass "timed-out run discovery cannot authorize a status-log pause" +} + +test_signaled_run_lookup_rejects_pause() { + reset_fakes + local d out toolbin + d=$(new_case signaled-run-paused) + make_repo_on_branch "$d/wt" fm/signaled-run-paused + make_fakebin "$d" >/dev/null + toolbin=$(make_no_timeout_toolbin "$d") + fm_write_meta "$d/state/signaled-run-paused.meta" \ + "window=fm:fm-signaled-run-paused" "worktree=$d/wt" "kind=ship" + printf 'paused: awaiting release; owner=release team; clears=release artifact is published\n' \ + > "$d/state/signaled-run-paused.status" + FM_FAKE_AXI_STATUS_MODE=signal + out=$(PATH="$d/fakebin:$toolbin" FM_STATE_OVERRIDE="$d/state" "$CREW_STATE" signaled-run-paused) + assert_contains "$out" "state: unknown" "signaled run lookup -> unknown" + assert_contains "$out" "run lookup unavailable" "signaled run lookup names the unavailable proof" + assert_not_contains "$out" "state: paused" "signaled run lookup trusted a stale pause" + pass "signaled run discovery cannot authorize a status-log pause" +} + test_no_run_idle_pane_custom_paused_verb() { reset_fakes local d; d=$(new_case custom-paused) make_repo_on_branch "$d/wt" fm/feat-custom-pause make_fakebin "$d" >/dev/null fm_write_meta "$d/state/feat-custom-pause.meta" "window=fm:fm-feat-custom-pause" "worktree=$d/wt" "kind=ship" - printf 'awaiting: vendor maintenance window\n' > "$d/state/feat-custom-pause.status" + printf 'awaiting: vendor maintenance window; owner=vendor; clears=maintenance window closes\n' > "$d/state/feat-custom-pause.status" FM_FAKE_AXI_STATUS="" FM_FAKE_BUSY=0 local out; out=$(FM_CLASSIFY_PAUSED_VERB=awaiting run_crew_state "$d" feat-custom-pause) @@ -1518,10 +1638,62 @@ test_no_run_idle_pane_custom_paused_verb() { assert_contains "$out" "vendor maintenance window" "custom pause preserves its reason" printf 'paused: default verb no longer selected\n' > "$d/state/feat-custom-pause.status" out=$(FM_CLASSIFY_PAUSED_VERB=awaiting run_crew_state "$d" feat-custom-pause) - assert_contains "$out" "state: unknown" "custom paused verb replaces the default" + assert_contains "$out" "state: unknown" "custom paused verb leaves the unrecognized default a non-state, not paused" pass "no run + idle pane honors the configured paused verb" } +# The supervisor-facing reader must preserve three separate outcomes: a readable +# stopped lane without positive work evidence is wedged, an auditable declaration +# is paused, and a lane with no readable evidence remains unknown. In particular, +# silence must never inherit paused health from the pause vocabulary. +test_supervisor_read_separates_paused_wedged_unknown() { + reset_fakes + local wedged paused silent wedged_out paused_out silent_out correct=0 + + wedged=$(new_case supervisor-wedged) + make_repo_on_branch "$wedged/wt" fm/supervisor-wedged + make_fakebin "$wedged" >/dev/null + fm_write_meta "$wedged/state/supervisor-wedged.meta" \ + "window=fm:fm-supervisor-wedged" "worktree=$wedged/wt" "kind=ship" + printf 'working: implementation was in progress\n' > "$wedged/state/supervisor-wedged.status" + FM_FAKE_AXI_STATUS="" + FM_FAKE_RUNS_LIST="" + FM_FAKE_BUSY=0 + wedged_out=$(run_crew_state "$wedged" supervisor-wedged) + case "$wedged_out" in + *"state: wedged"*) case "$wedged_out" in *"state: paused"*) ;; *) correct=$((correct + 1)) ;; esac ;; + esac + + paused=$(new_case supervisor-paused) + make_repo_on_branch "$paused/wt" fm/supervisor-paused + make_fakebin "$paused" >/dev/null + fm_write_meta "$paused/state/supervisor-paused.meta" \ + "window=fm:fm-supervisor-paused" "worktree=$paused/wt" "kind=ship" + printf 'paused: validation hold; owner=supervisor; clears=supervisor lifts the hold after host load recovers\n' \ + > "$paused/state/supervisor-paused.status" + paused_out=$(run_crew_state "$paused" supervisor-paused) + case "$paused_out" in + *"state: paused"*"owner=supervisor"*"clears=supervisor lifts the hold after host load recovers"*) + correct=$((correct + 1)) + ;; + esac + + silent=$(new_case supervisor-silent) + make_fakebin "$silent" >/dev/null + silent_out=$(run_crew_state "$silent" supervisor-silent) + export FM_CREW_STATE_BIN="$silent/fakebin/fm-crew-state.sh" + export FM_FAKE_CREW_STATE='state: unknown · source: none · no metadata' + case "$silent_out:$(crew_absorb_class supervisor-silent)" in + *"state: unknown"*:none) case "$silent_out" in *"state: paused"*) ;; *) correct=$((correct + 1)) ;; esac ;; + esac + unset FM_FAKE_CREW_STATE + export FM_CREW_STATE_BIN="$CREW_STATE" + + [ "$correct" -eq 3 ] \ + || fail "$correct of 3 supervisor states were distinct: wedged=[$wedged_out] paused=[$paused_out] silent=[$silent_out]" + pass "supervisor read keeps wedged, owned pause, and unknown silence distinct" +} + # A trailing keyed resolved: event is a decision-CLOSING event, not a run-state # verb. It must never become the current state or leak its resolution prose as the # detail: a healthy idle secondmate that just closed a keyed decision falls through @@ -1553,6 +1725,18 @@ test_no_run_idle_secondmate_resolved_event_not_state() { out=$(run_crew_state "$d" mate) assert_contains "$out" "state: working" "a real trailing state verb still renders" assert_contains "$out" "reconciling routed items" "a real state line still carries its detail" + # The invariant is about the EVENT, not the kind: a ship whose last event only + # closed a decision has no current-state source either, so it must not be graded + # a stopped lane and must not render the resolution prose as what it is doing. + fm_write_meta "$d/state/hull.meta" "window=fm:fm-hull" "worktree=$d/wt" "kind=ship" + printf 'needs-decision [key=rollback]: merge as-is, or fix the empty-pointer refusal?\n' \ + > "$d/state/hull.status" + printf 'resolved [key=rollback]: option (a) approved - merge this rebase as-is\n' \ + >> "$d/state/hull.status" + out=$(run_crew_state "$d" hull) + assert_contains "$out" "state: unknown" "a resolved-then-idle ship is not graded a stopped lane" + assert_contains "$out" "source: none" "a resolved event is not a status-log state source for a ship either" + assert_not_contains "$out" "merge this rebase as-is" "ship resolution prose must not leak into the detail" pass "a trailing resolved: event does not corrupt state render (idle stays idle)" } @@ -1630,8 +1814,8 @@ SH start=$SECONDS out=$(FM_FAKE_NM_CALLS="$calls_file" PATH="$d/fakebin:$toolbin" FM_STATE_OVERRIDE="$d/state" FM_CREW_STATE_NM_TIMEOUT=1 "$CREW_STATE" feat-timeout) elapsed=$((SECONDS - start)) - assert_contains "$out" "state: working" "timed-out no-mistakes falls back to pane" - assert_contains "$out" "source: pane" "timed-out no-mistakes -> pane source" + assert_contains "$out" "state: unknown" "timed-out no-mistakes remains unknown" + assert_contains "$out" "run lookup unavailable" "timed-out no-mistakes names the unavailable proof" [ "$elapsed" -lt 5 ] || fail "perl timeout did not bound no-mistakes calls (elapsed ${elapsed}s)" calls=$(awk 'END { print NR + 0 }' "$calls_file" 2>/dev/null || echo 0) [ "$calls" -eq 1 ] || fail "empty no-mistakes status triggered extra lookups ($calls calls)" @@ -1699,7 +1883,9 @@ test_provably_working_via_runs_list_fallback() { running fm/feat-provable bbbbbbb 2026-07-02 22:05 EOF )" - PATH="$d/fakebin:$PATH" FM_STATE_OVERRIDE="$d/state" crew_is_provably_working feat-provable \ + PATH="$d/fakebin:$PATH" FM_STATE_OVERRIDE="$d/state" \ + FM_CREW_STATE_NM_LIVENESS_BIN="$d/fakebin/fake-liveness" \ + crew_is_provably_working feat-provable \ || fail "cross-branch attribution via the runs list was not treated as provably working" pass "crew_is_provably_working absorbs a validating crew found only via the runs-list fallback" } @@ -1733,6 +1919,32 @@ test_usage_error() { pass "usage error exits 2" } +if [ "${FM_TEST_FOCUSED:-}" = supervision-states ]; then + test_supervisor_read_separates_paused_wedged_unknown + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = state-consumer-regressions ]; then + test_matching_run_state_overrides_pause + test_supervisor_read_separates_paused_wedged_unknown + test_no_run_idle_secondmate_resolved_event_not_state + test_provably_working_via_runs_list_fallback + test_not_provably_working_when_stopped + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = run-discovery ]; then + test_matching_run_state_overrides_pause + test_cross_branch_attribution_via_runs_list + test_other_branch_run_ignored + test_no_run_idle_pane_paused + test_unreadable_run_lookup_rejects_pause + test_timed_out_run_lookup_rejects_pause + test_signaled_run_lookup_rejects_pause + test_no_timeout_uses_perl_bound + exit 0 +fi + test_active_run_is_authoritative test_quiet_step_reports_dead_liveness test_quiet_step_reports_alive_liveness @@ -1743,6 +1955,7 @@ test_stale_blocked_superseded test_genuine_parked_not_superseded test_scalar_gate_parked_not_superseded test_gate_block_parked_not_superseded +test_matching_run_state_overrides_pause test_ci_ready_done_log_beats_monitoring_run test_ci_ready_log_pr_url_does_not_supply_run_identity test_ci_monitoring_checks_green_surfaces_done @@ -1784,7 +1997,11 @@ test_empty_run_lookup_rejects_checks_green_log test_skipped_run_lookup_rejects_checks_green_log test_no_run_idle_pane_uses_keyed_log test_no_run_idle_pane_paused +test_unreadable_run_lookup_rejects_pause +test_timed_out_run_lookup_rejects_pause +test_signaled_run_lookup_rejects_pause test_no_run_idle_pane_custom_paused_verb +test_supervisor_read_separates_paused_wedged_unknown test_no_run_idle_secondmate_resolved_event_not_state test_dead_window_ignores_stale_status_log test_dead_window_still_reports_terminal_run_step diff --git a/tests/fm-daemon.test.sh b/tests/fm-daemon.test.sh index c637f3aa329..aeaeca0915f 100755 --- a/tests/fm-daemon.test.sh +++ b/tests/fm-daemon.test.sh @@ -198,29 +198,105 @@ test_stale_terminal_escalates() { # escalation: classify_stale returns the `pause` action so handle_wake records a # pause marker (long re-surface cadence) rather than a wedge stale marker. test_stale_paused_classifies_pause() { - local dir state out pause_reason + local dir state fakebin out pause_reason dir=$(make_supercase stale-paused) state="$dir/state" - pause_reason='paused: waiting for upstream checks green, merged, and blocked state to clear' + fakebin="$dir/fakebin" + make_fake_crew_state "$fakebin" >/dev/null + pause_reason='paused: waiting for upstream checks green, merged, and blocked state to clear; owner=upstream CI; clears=required checks become terminal' status_is_captain_relevant "$pause_reason" && fail "pause reason phrases made the status captain-relevant" printf '%s\n' "$pause_reason" > "$state/held-w9.status" - out=$(FM_STATE_OVERRIDE="$state" classify_stale "sess:fm-held-w9" "$state") + out=$(FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' \ + classify_stale "sess:fm-held-w9" "$state") case "$out" in pause\|*) ;; *) fail "declared pause did not classify as pause: $out" ;; esac pass "paused reasons with captain phrases remain pause-classified" } +test_stale_pause_with_open_decision_escalates() { + local dir state fakebin out + dir=$(make_supercase stale-paused-open-decision) + state="$dir/state" + fakebin="$dir/fakebin" + make_fake_crew_state "$fakebin" >/dev/null + printf 'needs-decision [key=q]: choose the release target\n' > "$state/held-open.status" + printf 'paused: awaiting release; owner=release team; clears=release artifact is published\n' >> "$state/held-open.status" + out=$(FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' \ + classify_stale "sess:fm-held-open" "$state") + case "$out" in + escalate\|*) ;; + *) fail "pause with an open decision was not escalated: $out" ;; + esac + pass "away-mode pause refuses an unresolved keyed decision" +} + +test_signal_pause_with_open_decision_escalates_without_marker() { + local dir state fakebin out key win + dir=$(make_supercase signal-paused-open-decision) + state="$dir/state" + fakebin="$dir/fakebin" + make_fake_crew_state "$fakebin" >/dev/null + win="sess:fm-signal-open" + printf 'window=%s\nkind=ship\n' "$win" > "$state/signal-open.meta" + printf 'needs-decision [key=q]: choose the release target\n' > "$state/signal-open.status" + printf 'paused: awaiting release; owner=release team; clears=release artifact is published\n' >> "$state/signal-open.status" + out=$(FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' \ + classify_signal "$state/signal-open.status" "$state") + case "$out" in + escalate\|*) ;; + *) fail "pause signal with an open decision was not escalated: $out" ;; + esac + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' \ + handle_wake "signal: $state/signal-open.status" "$state" + key=$(printf '%s' "signal-open" | tr '.:/' '___') + [ ! -e "$state/.subsuper-paused-$key" ] || fail "open decision registered a daemon pause marker" + pass "away-mode pause signal refuses an unresolved keyed decision" +} + +test_signal_pause_with_terminal_run_escalates_without_marker() { + local dir state fakebin out key win + dir=$(make_supercase signal-paused-terminal-run) + state="$dir/state" + fakebin="$dir/fakebin" + make_fake_crew_state "$fakebin" >/dev/null + win="sess:fm-signal-terminal" + printf 'window=%s\nkind=ship\n' "$win" > "$state/signal-terminal.meta" + printf 'paused: awaiting release; owner=release team; clears=release artifact is published\n' \ + > "$state/signal-terminal.status" + out=$(FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: done · source: run-step · checks green: PR ready for review' \ + classify_signal "$state/signal-terminal.status" "$state") + case "$out" in + escalate\|*) ;; + *) fail "pause signal hid a terminal run outcome: $out" ;; + esac + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: done · source: run-step · checks green: PR ready for review' \ + handle_wake "signal: $state/signal-terminal.status" "$state" + key=$(printf '%s' "signal-terminal" | tr '.:/' '___') + [ ! -e "$state/.subsuper-paused-$key" ] || fail "terminal run registered a daemon pause marker" + pass "away-mode pause signal preserves terminal run authority" +} + # handle_wake on a paused stale records a pause marker, drops any pre-existing wedge # marker (so a working->paused pane is not still wedge-aged), and does NOT escalate # on the wake itself - the recheck is housekeeping's job on the long cadence. test_handle_wake_paused_records_pause_marker() { - local dir state key win + local dir state fakebin key win dir=$(make_supercase handle-paused) state="$dir/state" + fakebin="$dir/fakebin" + make_fake_crew_state "$fakebin" >/dev/null win="sess:fm-held-w10" - printf 'paused: awaiting the vendor rate-limit reset\n' > "$state/held-w10.status" + printf 'paused: awaiting the vendor rate-limit reset; owner=vendor; clears=rate-limit reset time arrives\n' > "$state/held-w10.status" key=$(printf '%s' "held-w10" | tr ':/.' '___') date +%s > "$state/.subsuper-stale-$key" - FM_STATE_OVERRIDE="$state" handle_wake "stale: $win" "$state" + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' \ + handle_wake "stale: $win" "$state" [ -e "$state/.subsuper-paused-$key" ] || fail "pause marker not recorded by handle_wake" [ ! -e "$state/.subsuper-stale-$key" ] || fail "wedge marker not cleared when the crew declared a pause" [ ! -s "$state/.subsuper-escalations" ] || fail "a declared pause escalated on the wake itself (should defer to the long recheck)" @@ -228,15 +304,19 @@ test_handle_wake_paused_records_pause_marker() { } test_handle_wake_paused_signal_records_pause_marker() { - local dir state key win + local dir state fakebin key win dir=$(make_supercase handle-paused-signal) state="$dir/state" + fakebin="$dir/fakebin" + make_fake_crew_state "$fakebin" >/dev/null win="sess:fm-held-w10-signal" printf 'window=%s\nkind=ship\n' "$win" > "$state/held-w10-signal.meta" - printf 'paused: awaiting the vendor rate-limit reset\n' > "$state/held-w10-signal.status" + printf 'paused: awaiting the vendor rate-limit reset; owner=vendor; clears=rate-limit reset time arrives\n' > "$state/held-w10-signal.status" key=$(printf '%s' "held-w10-signal" | tr ':/.' '___') date +%s > "$state/.subsuper-stale-$key" - FM_STATE_OVERRIDE="$state" handle_wake "signal: $state/held-w10-signal.status" "$state" + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' \ + handle_wake "signal: $state/held-w10-signal.status" "$state" [ -e "$state/.subsuper-paused-$key" ] || fail "pause signal did not record a pause marker" [ ! -e "$state/.subsuper-stale-$key" ] || fail "pause signal did not clear the wedge marker" [ ! -s "$state/.subsuper-escalations" ] || fail "a declared pause signal escalated instead of self-handling" @@ -269,15 +349,18 @@ test_handle_wake_terminal_signal_clears_pause_tracking() { } test_housekeeping_migrates_watcher_pause_marker() { - local dir state key win + local dir state fakebin key win dir=$(make_supercase migrate-watcher-pause) state="$dir/state" + fakebin="$dir/fakebin" + make_fake_crew_state "$fakebin" >/dev/null win="sess:fm-held-w10-migrate" printf 'window=%s\nkind=ship\n' "$win" > "$state/held-w10-migrate.meta" - printf 'paused: awaiting the upstream release\n' > "$state/held-w10-migrate.status" + printf 'paused: awaiting the upstream release; owner=release team; clears=release artifact is published\n' > "$state/held-w10-migrate.status" key=$(printf '%s' "$win" | tr '.:/' '___') : > "$state/.paused-$key" - FM_STATE_OVERRIDE="$state" housekeeping "$state" + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' housekeeping "$state" key=$(printf '%s' "held-w10-migrate" | tr '.:/' '___') [ -e "$state/.subsuper-paused-$key" ] || fail "watcher pause marker was not migrated into daemon tracking" [ ! -e "$state/.subsuper-stale-$key" ] || fail "watcher pause migration left a wedge marker behind" @@ -303,14 +386,17 @@ test_housekeeping_migrates_watcher_unpaused_marker_to_clear() { } test_housekeeping_seeds_pause_marker_from_status() { - local dir state key win + local dir state fakebin key win dir=$(make_supercase seed-paused-status) state="$dir/state" + fakebin="$dir/fakebin" + make_fake_crew_state "$fakebin" >/dev/null win="sess:fm-held-w10-seed" printf 'window=%s\nkind=ship\n' "$win" > "$state/held-w10-seed.meta" - printf 'paused: awaiting the upstream release\n' > "$state/held-w10-seed.status" + printf 'paused: awaiting the upstream release; owner=release team; clears=release artifact is published\n' > "$state/held-w10-seed.status" key=$(printf '%s' "held-w10-seed" | tr '.:/' '___') - FM_STATE_OVERRIDE="$state" housekeeping "$state" + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' housekeeping "$state" [ -e "$state/.subsuper-paused-$key" ] || fail "paused status did not seed daemon pause tracking" [ ! -e "$state/.subsuper-stale-$key" ] || fail "paused status seeded wedge tracking" pass "housekeeping seeds pause tracking from status without a watcher marker" @@ -323,14 +409,18 @@ test_housekeeping_paused_resurfaces_and_resets() { local dir state fakebin win pane key age dir=$(make_supercase paused-resurface) state="$dir/state"; fakebin="$dir/fakebin" + make_fake_crew_state "$fakebin" >/dev/null win="sess:fm-held-w11"; pane="$dir/pane.txt" - printf 'paused: holding for the upstream tool release\n' > "$state/held-w11.status" + printf 'paused: holding for the upstream tool release; owner=tool maintainer; clears=release artifact is published\n' > "$state/held-w11.status" printf 'idle prompt $\n' > "$pane" key=$(printf '%s' "held-w11" | tr ':/.' '___') echo $(( $(date +%s) - 5000 )) > "$state/.subsuper-paused-$key" PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ - FM_STATE_OVERRIDE="$state" FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" - grep -F "awaiting external" "$state/.subsuper-escalations" >/dev/null 2>&1 || fail "declared pause was not re-surfaced as an awaiting-external recheck" + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' \ + FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" + grep -F "owner=tool maintainer" "$state/.subsuper-escalations" >/dev/null 2>&1 || fail "declared pause recheck omitted its owner" + grep -F "clears=release artifact is published" "$state/.subsuper-escalations" >/dev/null 2>&1 || fail "declared pause recheck omitted its clearing condition" grep -F "possible wedge" "$state/.subsuper-escalations" >/dev/null 2>&1 && fail "declared pause was mislabeled a possible wedge" [ -e "$state/.subsuper-paused-$key" ] || fail "pause marker cleared instead of reset for the next window" age=$(( $(date +%s) - $(cat "$state/.subsuper-paused-$key" 2>/dev/null || echo 0) )) @@ -344,13 +434,16 @@ test_housekeeping_paused_resumed_cleared() { local dir state fakebin win pane key dir=$(make_supercase paused-resumed) state="$dir/state"; fakebin="$dir/fakebin" + make_fake_crew_state "$fakebin" >/dev/null win="sess:fm-held-w12"; pane="$dir/pane.txt" - printf 'paused: holding for the upstream tool release\n' > "$state/held-w12.status" + printf 'paused: holding for the upstream tool release; owner=tool maintainer; clears=release artifact is published\n' > "$state/held-w12.status" printf 'Working...\n' > "$pane" key=$(printf '%s' "held-w12" | tr ':/.' '___') echo $(( $(date +%s) - 5000 )) > "$state/.subsuper-paused-$key" PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ - FM_STATE_OVERRIDE="$state" FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' \ + FM_PAUSE_RESURFACE_SECS=240 housekeeping "$state" [ -e "$state/.subsuper-paused-$key" ] && fail "resumed (busy) pause marker was not cleared" [ ! -s "$state/.subsuper-escalations" ] || fail "a resumed pause was escalated" pass "housekeeping clears a paused marker whose pane became busy again, without escalating" @@ -364,7 +457,7 @@ test_housekeeping_paused_unpaused_cleared() { dir=$(make_supercase paused-unpaused) state="$dir/state"; fakebin="$dir/fakebin" win="sess:fm-held-w13"; pane="$dir/pane.txt" - printf 'paused: holding for the upstream release\nworking: resumed, upstream landed\n' > "$state/held-w13.status" + printf 'paused: holding for the upstream release; owner=release team; clears=release artifact is published\nworking: resumed, upstream landed\n' > "$state/held-w13.status" printf 'idle prompt $\n' > "$pane" key=$(printf '%s' "held-w13" | tr ':/.' '___') echo $(( $(date +%s) - 5000 )) > "$state/.subsuper-paused-$key" @@ -379,12 +472,15 @@ test_housekeeping_stale_marker_transitions_to_pause() { local dir state fakebin win pane key dir=$(make_supercase stale-to-paused) state="$dir/state"; fakebin="$dir/fakebin"; win="sess:fm-held-w14"; pane="$dir/pane.txt" - printf 'paused: awaiting the upstream tool release\n' > "$state/held-w14.status" + make_fake_crew_state "$fakebin" >/dev/null + printf 'paused: awaiting the upstream tool release; owner=tool maintainer; clears=release artifact is published\n' > "$state/held-w14.status" printf 'idle prompt $\n' > "$pane" key=$(printf '%s' "held-w14" | tr ':/.' '___') echo $(( $(date +%s) - 5000 )) > "$state/.subsuper-stale-$key" PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$win" FM_FAKE_TMUX_CAPTURE="$pane" \ - FM_STATE_OVERRIDE="$state" FM_STALE_ESCALATE_SECS=240 housekeeping "$state" + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" \ + FM_FAKE_CREW_STATE='state: paused · source: status-log · declared pause' \ + FM_STALE_ESCALATE_SECS=240 housekeeping "$state" [ -e "$state/.subsuper-paused-$key" ] || fail "existing stale marker did not move to paused state" [ ! -e "$state/.subsuper-stale-$key" ] || fail "existing stale marker remained wedge-aged after pause" [ ! -s "$state/.subsuper-escalations" ] || fail "a newly declared pause was escalated as a possible wedge" @@ -1662,6 +1758,8 @@ test_inject_msg_herdr_pane_gone_defers() { afk_enter "$state" ( fm_backend_target_exists() { return 1; } + # Intentional negative-assertion stub that must never be called. + # shellcheck disable=SC2329 fm_backend_busy_state() { fail "busy_state should not be consulted once the pane-exists check already failed"; } fm_backend_send_text_submit() { fail "send_text_submit should not run when the pane does not exist"; } if FM_SUPERVISOR_BACKEND=herdr FM_SUPERVISOR_TARGET="default:w1:gone" inject_msg "hello" "$state"; then @@ -1715,6 +1813,20 @@ test_inject_msg_defers_on_dead_shell_unknown() { pass "inject_msg: defers on a dead-shell/unreadable composer (unknown), never typing the escalation into a shell" } +if [ "${FM_TEST_FOCUSED:-}" = pause-proof ]; then + test_stale_paused_classifies_pause + test_stale_pause_with_open_decision_escalates + test_signal_pause_with_open_decision_escalates_without_marker + test_signal_pause_with_terminal_run_escalates_without_marker + test_handle_wake_paused_records_pause_marker + test_handle_wake_paused_signal_records_pause_marker + test_housekeeping_migrates_watcher_pause_marker + test_housekeeping_seeds_pause_marker_from_status + test_housekeeping_paused_resurfaces_and_resets + test_housekeeping_stale_marker_transitions_to_pause + exit 0 +fi + if [ "${FM_TEST_FOCUSED:-}" = liveness-verdicts ]; then test_liveness_verdicts_surface_through_away_classifiers exit 0 @@ -1731,6 +1843,9 @@ test_liveness_verdicts_surface_through_away_classifiers test_stale_transient_self_records_marker test_stale_terminal_escalates test_stale_paused_classifies_pause +test_stale_pause_with_open_decision_escalates +test_signal_pause_with_open_decision_escalates_without_marker +test_signal_pause_with_terminal_run_escalates_without_marker test_handle_wake_paused_records_pause_marker test_handle_wake_paused_signal_records_pause_marker test_handle_wake_terminal_signal_clears_pause_tracking diff --git a/tests/fm-fleet-snapshot-view.test.sh b/tests/fm-fleet-snapshot-view.test.sh index 6625b641162..83876c61032 100755 --- a/tests/fm-fleet-snapshot-view.test.sh +++ b/tests/fm-fleet-snapshot-view.test.sh @@ -64,9 +64,16 @@ make_home() { # printf '%s\n' "$home" } +make_ship_worktree() { # + mkdir -p "$1" + git -C "$1" init -q + git -C "$1" symbolic-ref HEAD "refs/heads/$2" +} + write_fixture() { # local home=$1 - mkdir -p "$home/projects/alpha-worktree" "$home/projects/scout-worktree" "$home/secondmate-home" + make_ship_worktree "$home/projects/alpha-worktree" fm/ship-task + mkdir -p "$home/projects/scout-worktree" "$home/secondmate-home" cat > "$home/data/backlog.md" <> "$SLEEP_LOG"; } reset_state() { rm -f "$STATE_DIR"/*.meta "$STATE_DIR"/*.status "$STATE_DIR"/.wake-queue \ "$STATE_DIR"/.wake-queue.seq "$STATE_DIR"/.watch-triage.log \ - "$STATE_DIR"/.herdr-escalated-* "$TMP"/panes "$TMP"/wtcalls "$TMP"/wtcalled 2>/dev/null || true + "$STATE_DIR"/.herdr-escalated-* "$STATE_DIR"/.paused-* \ + "$STATE_DIR"/.stale-* "$TMP"/panes "$TMP"/wtcalls "$TMP"/wtcalled 2>/dev/null || true : > "$WAKE_LOG" : > "$SLEEP_LOG" _event_cap_key="" @@ -69,7 +70,8 @@ pass "handle_push_transition: enqueue failure cannot commit the Herdr dedupe mar reset_state fm_write_meta "$STATE_DIR/tk2.meta" "window=default:wG:pQ" "backend=herdr" "kind=ship" -printf 'paused: waiting on the upstream release\n' > "$STATE_DIR/tk2.status" +printf 'paused: waiting on the upstream release; owner=release team; clears=release artifact is published\n' > "$STATE_DIR/tk2.status" +crew_state_line() { printf 'state: paused · source: status-log · waiting on the upstream release\n'; } handle_push_transition herdr default "$(mkrec wG:pQ blocked)" if [ -e "$STATE_DIR/.wake-queue" ] && grep -q 'stale' "$STATE_DIR/.wake-queue"; then fail "a declared-pause crew must NOT be fast-escalated: $(cat "$STATE_DIR/.wake-queue")" @@ -80,6 +82,15 @@ grep -q 'absorbed push' "$STATE_DIR/.watch-triage.log" 2>/dev/null || fail "the [ -e "$STATE_DIR/.herdr-escalated-default_wG_pQ" ] || fail "the paused event path must commit the handled transition" pass "handle_push_transition: a declared-pause crew enters the shared pause cadence without a fast wake" +reset_state +fm_write_meta "$STATE_DIR/tk2.meta" "window=default:wG:pQ" "backend=herdr" "kind=ship" +printf 'needs-decision [key=q]: choose the release target\n' > "$STATE_DIR/tk2.status" +printf 'paused: waiting on the upstream release; owner=release team; clears=release artifact is published\n' >> "$STATE_DIR/tk2.status" +handle_push_transition herdr default "$(mkrec wG:pQ blocked)" +grep -q 'stale' "$STATE_DIR/.wake-queue" || fail "an open decision under a pause was absorbed by the event fast-path" +[ ! -e "$STATE_DIR/.paused-default_wG_pQ" ] || fail "an open decision created a pause marker on the event fast-path" +pass "handle_push_transition: an open keyed decision prevents pause absorption" + # --- event_wait_or_sleep: secondmate windows are excluded from the pane list -- reset_state diff --git a/tests/fm-watch-checkpoint.test.sh b/tests/fm-watch-checkpoint.test.sh index 1b0f1efd7fd..03e76290e52 100755 --- a/tests/fm-watch-checkpoint.test.sh +++ b/tests/fm-watch-checkpoint.test.sh @@ -34,6 +34,54 @@ test_quiet_checkpoint_exits_124_cleanly() { pass "quiet checkpoint exits 124 with a clean checkpoint line and no live lock" } +test_quiet_checkpoint_reclaims_forced_kill_lock() { + local home fakebin out err status + home=$(make_home forced-kill) + fakebin=$(fm_fakebin "$home") + out="$home/out.txt" + err="$home/err.txt" + cat > "$fakebin/timeout" <<'SH' +#!/usr/bin/env bash +mkdir -p "$FM_HOME/state/.watch.lock" +printf '99999999\n' > "$FM_HOME/state/.watch.lock/pid" +exit 124 +SH + chmod +x "$fakebin/timeout" + status=0 + PATH="$fakebin:$PATH" FM_HOME="$home" "$CHECKPOINT" --seconds 1 >"$out" 2>"$err" || status=$? + expect_code 124 "$status" "forced-kill checkpoint exit" + assert_contains "$(cat "$out")" "checkpoint: no actionable wake within 1s" "forced-kill checkpoint line missing" + assert_absent "$home/state/.watch.lock" \ + "forced-kill checkpoint left its stale watch lock: out=$(cat "$out"); err=$(cat "$err")" + pass "quiet checkpoint reclaims a forced-kill stale watcher lock" +} + +test_forced_kill_cleanup_ignores_ambient_state() { + local home ambient fakebin out err status + home=$(make_home forced-kill-ambient-state) + ambient="$home/ambient-state" + mkdir -p "$ambient/.watch.lock" + printf 'keep\n' > "$ambient/.watch.lock/sentinel" + fakebin=$(fm_fakebin "$home") + out="$home/out.txt" + err="$home/err.txt" + cat > "$fakebin/timeout" <<'SH' +#!/usr/bin/env bash +mkdir -p "$FM_HOME/state/.watch.lock" +printf '99999999\n' > "$FM_HOME/state/.watch.lock/pid" +exit 124 +SH + chmod +x "$fakebin/timeout" + status=0 + PATH="$fakebin:$PATH" FM_HOME="$home" STATE="$ambient" "$CHECKPOINT" --seconds 1 >"$out" 2>"$err" || status=$? + expect_code 124 "$status" "forced-kill checkpoint with ambient STATE exit" + assert_absent "$home/state/.watch.lock" \ + "forced-kill cleanup targeted ambient STATE instead of FM_HOME: out=$(cat "$out"); err=$(cat "$err")" + [ -e "$ambient/.watch.lock/sentinel" ] \ + || fail "forced-kill cleanup modified the ignored ambient STATE" + pass "forced-kill cleanup uses the watcher's state precedence" +} + test_signal_passes_through_and_exits_zero() { local home out err status drained home=$(make_home signal) @@ -89,6 +137,8 @@ test_existing_singleton_watcher_is_not_success() { } test_quiet_checkpoint_exits_124_cleanly +test_quiet_checkpoint_reclaims_forced_kill_lock +test_forced_kill_cleanup_ignores_ambient_state test_signal_passes_through_and_exits_zero test_check_uses_preserved_watcher_environment test_existing_singleton_watcher_is_not_success diff --git a/tests/fm-watch-pause-absorb.test.sh b/tests/fm-watch-pause-absorb.test.sh index 1ccac584a09..00c39c074f9 100755 --- a/tests/fm-watch-pause-absorb.test.sh +++ b/tests/fm-watch-pause-absorb.test.sh @@ -1,35 +1,6 @@ #!/usr/bin/env bash -# tests/fm-watch-pause-absorb.test.sh - a DECLARED pause must be honoured on its -# own evidence, independent of what the crewmate's terminal or its attributed -# no-mistakes run happens to be doing. -# -# The 2026-08-03 incident this suite pins down: lane priors-882-rebase-c8 declared -# `paused: work complete and verified; waiting on the merge of PR 882` with its one -# keyed decision opened AND explicitly resolved, yet wedge-escalated eight times in -# a row as a suspected wedge and never once created state/.paused-. The -# supervisor's hypothesis was pane liveness (its agent was alive and idle at the -# composer, while a comparison lane whose agent was quota-dead did absorb). That was -# WRONG, and this suite encodes the refutation: absorption never consulted pane -# liveness at all. crew_absorb_class asked fm-crew-state.sh for one authoritative -# verdict and admitted the declared pause only when that verdict was exactly `done` -# (bin/fm-classify-lib.sh, the `done`-only carve-out added in #53). Any other verdict -# - `parked` (c8: its recycled worktree had been re-checked-out onto ANOTHER lane's -# branch, so a foreign parked run was attributed to it), `failed` (a routinely -# CANCELLED run maps here too), or `unknown` (backend target gone, i.e. a genuinely -# DEAD pane) - discarded the pause and returned `none`, so the watcher took -# surface_nonterminal_stale and emitted a bare `stale: `. -# -# The dead-pane branch is the direct refutation: a gone pane classified `unknown` and -# was surfaced, so "only a dead pane reaches the pause path" had the truth backwards. -# The comparison lane absorbed because a quota-dead agent has no attributed run at -# all, which routes fm-crew-state.sh to its status-log fallback and reports `paused` - -# an absence-of-run effect, not a pane-liveness effect. -# -# Four branches, one per row of the truth table, driven through a REAL fm-watch.sh: -# live-idle paused run-step `parked`, pane alive and idle -> ABSORB -# dead paused pane gone, verdict `unknown` -> ABSORB -# paused + OPEN decision an unanswered question -> SURFACE -# paused + CLOSED decision opened then resolved (the c8 shape) -> ABSORB +# tests/fm-watch-pause-absorb.test.sh - pause absorption requires a durable +# declaration, no open keyed decision, and precedence-compatible current state. # # The third row is the safety boundary and the reason this is not simply "absorb # every pause": a lane with an unanswered captain question must never go quiet, so @@ -69,17 +40,12 @@ wait_live() { # surfaced), PAUSE_FLAG, WAKE_QUEUE, WATCH_OUT. Call it directly, never in a command # substitution, or the globals are lost with the subshell. # -# is the canned fm-crew-state.sh answer - the ONE input the old -# implementation keyed its whole decision on, which is why each branch below differs -# only in that string and in the status stream. The pane is primed as already-stale +# is the canned fm-crew-state.sh answer. The pane is primed as already-stale # (.hash/.count pre-seeded) so the poll reaches stale triage immediately, and # .seen-* is primed so the signal scan does not pre-empt it. A generous # FM_PAUSE_RESURFACE_SECS keeps a legitimate absorb silent, so any wake at all is a # genuine surface rather than the hourly pause recheck. -# PRESEED_PAUSED=1 makes the case start from an already-absorbed pause: the -# .paused- flag plus a fresh .paused-rechecked- marker, as a lane that -# absorbed on an earlier poll would carry. Used to prove the cached verdict cannot -# outlive the status stream it was proven from. +# PRESEED_PAUSED=1 starts from an already-absorbed pause with an old proof signature. PRESEED_PAUSED=0 run_pause_case() { # local name=$1 stream=$2 verdict=$3 @@ -96,8 +62,6 @@ run_pause_case() { # printf '%s' "$pane_hash" > "$state/.hash-$key" printf '1\n' > "$state/.count-$key" if [ "$PRESEED_PAUSED" = 1 ]; then - # A stale cache: flagged paused, rechecked seconds ago, but proven against a - # DIFFERENT (earlier) status stream than the one on disk now. : > "$state/.paused-$key" printf 'stale-signature-from-an-earlier-stream' > "$state/.paused-rechecked-$key" fi @@ -119,39 +83,34 @@ run_pause_case() { # } # --- branch 1: live-but-idle paused pane ----------------------------------------- -# The exact c8 shape. Its worktree had been recycled onto another lane's branch, so -# fm-crew-state.sh attributed that foreign run and answered `parked`. The pane was -# alive and idle at the composer the whole time. A pause is a statement about the -# WORK, so a run-step that is merely parked must not veto it. +# With no matching run, the status-log pause is current state. test_live_idle_paused_pane_absorbed() { run_pause_case live-idle-paused \ - 'paused: work complete and verified; waiting on the merge of PR 882 by the main firstmate + 'paused: work complete and verified; waiting on the merge of PR 882 by the main firstmate; owner=main firstmate; clears=PR 882 is merged ' \ - 'state: parked · source: run-step · parked at ci: 1 finding(s)' + 'state: paused · source: status-log · waiting on the merge' [ "$TRIAGE" = absorbed ] \ || fail "live-but-idle declared pause was surfaced as a suspected wedge: $WATCH_OUT" [ "$PAUSE_FLAG" = present ] \ || fail "live-but-idle declared pause never recorded .paused- (handle_paused_stale did not run)" [ -z "$WAKE_QUEUE" ] || fail "live-but-idle declared pause enqueued a wake: $WAKE_QUEUE" - pass "live-but-idle paused pane: a parked run-step no longer vetoes a declared pause" + pass "live-but-idle paused pane: a no-run status pause is absorbed" } # --- branch 2: dead paused pane -------------------------------------------------- -# The refutation of the pane-liveness hypothesis. A gone backend target makes -# fm-crew-state.sh answer `unknown · source: none`, which the old carve-out treated -# exactly like `parked` - discarded. So a DEAD pane was surfaced too, and pane -# liveness was never the discriminator in either direction. -test_dead_paused_pane_absorbed() { +# A gone backend target is unknown current state and remains actionable. +test_dead_paused_pane_surfaced() { run_pause_case dead-paused \ - 'paused: agent is quota-dead, work preserved and verified; waiting on the account reset + 'paused: agent is quota-dead, work preserved and verified; waiting on the account reset; owner=account owner; clears=the account quota resets ' \ 'state: unknown · source: none · backend target gone: test:fm-dead-paused' - [ "$TRIAGE" = absorbed ] \ - || fail "declared pause with a gone pane was surfaced as a suspected wedge: $WATCH_OUT" - [ "$PAUSE_FLAG" = present ] \ - || fail "declared pause with a gone pane never recorded .paused-" - [ -z "$WAKE_QUEUE" ] || fail "declared pause with a gone pane enqueued a wake: $WAKE_QUEUE" - pass "dead paused pane: a declared pause is honoured with no readable terminal at all" + [ "$TRIAGE" = surfaced ] \ + || fail "unknown current state was hidden behind a pause: $WATCH_OUT" + [ "$PAUSE_FLAG" = absent ] \ + || fail "unknown current state recorded .paused-" + printf '%s' "$WAKE_QUEUE" | grep -q 'stale' \ + || fail "unknown current state did not enqueue a stale wake: $WAKE_QUEUE" + pass "dead paused pane: unknown current state retains precedence" } # --- branch 3: paused with an OPEN decision -------------------------------------- @@ -161,9 +120,9 @@ test_dead_paused_pane_absorbed() { test_paused_with_open_decision_surfaced() { run_pause_case paused-open-decision \ 'needs-decision [key=rollback-empty-pointer]: merge as-is, or fix the empty-pointer refusal here? -paused: standing by for the decision on the rollback gap +paused: standing by for the decision on the rollback gap; owner=captain; clears=the rollback decision is answered ' \ - 'state: unknown · source: none · backend target gone: test:fm-paused-open-decision' + 'state: paused · source: status-log · standing by for the decision' [ "$TRIAGE" = surfaced ] \ || fail "a pause masking an UNANSWERED decision was absorbed and went quiet" [ "$PAUSE_FLAG" = absent ] \ @@ -183,9 +142,9 @@ test_paused_with_closed_decision_absorbed() { done: PR https://github.com/Ruby-Labs/relvino/pull/882 checks green; NOT merged needs-decision [key=rollback-empty-pointer]: merge as-is, or fix the empty-pointer refusal here? resolved [key=rollback-empty-pointer]: option (a) approved - merge this rebase as-is -paused: work complete and verified; waiting on the merge of PR 882 by the main firstmate +paused: work complete and verified; waiting on the merge of PR 882 by the main firstmate; owner=main firstmate; clears=PR 882 is merged ' \ - 'state: parked · source: run-step · parked at ci: 1 finding(s)' + 'state: paused · source: status-log · waiting on the merge' [ "$TRIAGE" = absorbed ] \ || fail "a pause whose only decision was explicitly resolved was surfaced: $WATCH_OUT" [ "$PAUSE_FLAG" = present ] \ @@ -196,28 +155,29 @@ paused: work complete and verified; waiting on the merge of PR 882 by the main f # --- the same four branches at the classifier level ------------------------------- # The behavioral cases above prove the watcher wires this correctly; this matrix pins -# the decision itself, including the run-step verdicts that used to veto a pause and -# the `working` verdict that legitimately still does. +# the decision itself. test_crew_absorb_class_pause_matrix() { local dir state fakebin dir=$(make_case absorb-pause-matrix); state="$dir/state"; fakebin="$dir/fakebin" export FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_FAKE_CREW_STATE FM_STATE_OVERRIDE="$state" - local paused='paused: waiting on the merge of PR 882' + local paused='paused: waiting on the merge of PR 882; owner=main firstmate; clears=PR 882 is merged' printf '%s\n' "$paused" > "$state/a.status" - # Every non-working verdict must yield to the declared pause, including the three - # that used to veto it. local v - for v in 'state: parked · source: run-step · parked at ci: 1 finding(s)' \ - 'state: failed · source: run-step · run cancelled' \ - 'state: unknown · source: none · backend target gone' \ - 'state: done · source: run-step · run passed: PR merged (verified)' \ - 'state: paused · source: status-log · waiting on the merge'; do + FM_FAKE_CREW_STATE='state: paused · source: status-log · waiting on the merge' + [ "$(crew_absorb_class a "$paused")" = paused ] \ + || fail "a no-run status-log pause was not honoured" + crew_is_paused a || fail "crew_is_paused disagreed with the no-run pause class" + + for v in 'state: parked · source: run-step · parked at ci: 1 finding(s) · run: 01RUN' \ + 'state: done · source: run-step · run passed: PR merged (verified) · run: 01RUN' \ + 'state: failed · source: run-step · run cancelled · run: 01RUN' \ + 'state: stale · source: run-step · stale run' \ + 'state: unknown · source: none · backend target gone'; do FM_FAKE_CREW_STATE="$v" - [ "$(crew_absorb_class a "$paused")" = paused ] \ - || fail "declared pause not honoured over verdict [$v]" - crew_is_paused a || fail "crew_is_paused disagreed with the class for verdict [$v]" + [ "$(crew_absorb_class a "$paused")" = none ] \ + || fail "authoritative verdict was hidden behind a pause [$v]" done # A crewmate that appended a pause and then STARTED working is working, not paused: @@ -231,15 +191,15 @@ test_crew_absorb_class_pause_matrix() { # The pause must be read from the durable stream even when no caller passes it, # so the two call sites in bin/fm-watch.sh cannot disagree. - FM_FAKE_CREW_STATE='state: parked · source: run-step · parked at ci: 1 finding(s)' + FM_FAKE_CREW_STATE='state: paused · source: status-log · waiting on the merge' [ "$(crew_absorb_class a)" = paused ] \ || fail "the declared pause was not read from the status stream when unspecified" # An open keyed decision blocks absorption on every verdict. printf 'needs-decision [key=api-shape]: which shape?\n%s\n' "$paused" > "$state/a.status" - for v in 'state: parked · source: run-step · parked at ci: 1 finding(s)' \ + for v in 'state: parked · source: run-step · parked at ci: 1 finding(s) · run: 01RUN' \ 'state: unknown · source: none · backend target gone' \ - 'state: done · source: run-step · run passed: PR merged (verified)' \ + 'state: done · source: run-step · run passed: PR merged (verified) · run: 01RUN' \ 'state: paused · source: status-log · waiting on the merge'; do FM_FAKE_CREW_STATE="$v" [ "$(crew_absorb_class a "$paused")" = none ] \ @@ -247,14 +207,14 @@ test_crew_absorb_class_pause_matrix() { done # ...and stops blocking once it is explicitly resolved. printf 'needs-decision [key=api-shape]: which shape?\nresolved [key=api-shape]: option (a)\n%s\n' "$paused" > "$state/a.status" - FM_FAKE_CREW_STATE='state: parked · source: run-step · parked at ci: 1 finding(s)' + FM_FAKE_CREW_STATE='state: paused · source: status-log · waiting on the merge' [ "$(crew_absorb_class a "$paused")" = paused ] \ || fail "a resolved decision still blocked absorption" # A FAILURE reported under the pause verb is not a wait and must never absorb. - local failpause='paused: error: drive run: reconcile run: read response: i/o timeout' + local failpause='paused: error: drive run: reconcile run: read response: i/o timeout; owner=drive; clears=the reconcile run returns' printf '%s\n' "$failpause" > "$state/a.status" - FM_FAKE_CREW_STATE='state: unknown · source: none · backend target gone' + FM_FAKE_CREW_STATE='state: paused · source: status-log · failure-shaped pause' [ "$(crew_absorb_class a "$failpause")" = none ] \ || fail "a failure reported under the pause verb was absorbed as a declared wait" @@ -265,38 +225,32 @@ test_crew_absorb_class_pause_matrix() { [ "$(crew_absorb_class "")" = none ] || fail "empty id not classed none" unset FM_FAKE_CREW_STATE FM_STATE_OVERRIDE - pass "crew_absorb_class: a declared pause outranks every verdict but working, and an open decision blocks it" + pass "crew_absorb_class enforces run precedence and the open-decision proof" } -# --- a cached pause verdict cannot outlive the stream it was proven from ---------- -# pause_state_class skips the authoritative re-read while a recent recheck marker -# stands, so that cache is the one way an open decision could still slip past the -# boundary: a lane absorbs cleanly, THEN appends needs-decision and re-declares its -# pause, leaving the pause verb on the last line again. The marker carries the status -# signature it was proven against, so any append invalidates it and forces a fresh -# proof instead of riding out the age window. -test_cached_pause_verdict_reproven_when_stream_changes() { +# --- a pause proof marker cannot outlive the stream it was proven from ------------- +test_pause_marker_reproven_when_stream_changes() { PRESEED_PAUSED=1 run_pause_case cached-pause-stale-proof \ - 'paused: waiting on the merge of PR 882 + 'paused: waiting on the merge of PR 882; owner=main firstmate; clears=PR 882 is merged needs-decision [key=rollback-empty-pointer]: merge as-is, or fix the empty-pointer refusal here? -paused: standing by for the decision on the rollback gap +paused: standing by for the decision on the rollback gap; owner=captain; clears=the rollback decision is answered ' \ - 'state: unknown · source: none · backend target gone: test:fm-cached-pause-stale-proof' + 'state: paused · source: status-log · standing by for the decision' PRESEED_PAUSED=0 [ "$TRIAGE" = surfaced ] \ || fail "a decision opened after the pause flag rode out the cache window and went quiet" printf '%s' "$WAKE_QUEUE" | grep -q 'stale' \ || fail "the re-proven pause did not enqueue a stale wake: $WAKE_QUEUE" - pass "a cached pause verdict is re-proven whenever the status stream changes under it" + pass "a pause proof marker is re-proven whenever the status stream changes" } test_pause_moving_during_pipeline_read_refused() { export FM_FAKE_CREW_STATE_APPEND_STATUS='blocked: stream advanced during pipeline state read' run_pause_case pause-moved-during-proof \ - 'paused: awaiting ordered PR merges + 'paused: awaiting ordered PR merges; owner=merge supervisor; clears=ordered merges complete ' \ - 'state: parked · source: run-step · parked at ci: 1 finding(s)' + 'state: paused · source: status-log · awaiting ordered PR merges' unset FM_FAKE_CREW_STATE_APPEND_STATUS [ "$TRIAGE" = surfaced ] \ || fail "a pause invalidated during the crew-state read was absorbed" @@ -306,9 +260,9 @@ test_pause_moving_during_pipeline_read_refused() { } test_live_idle_paused_pane_absorbed -test_dead_paused_pane_absorbed +test_dead_paused_pane_surfaced test_paused_with_open_decision_surfaced test_paused_with_closed_decision_absorbed -test_cached_pause_verdict_reproven_when_stream_changes +test_pause_marker_reproven_when_stream_changes test_pause_moving_during_pipeline_read_refused test_crew_absorb_class_pause_matrix diff --git a/tests/fm-watch-triage.test.sh b/tests/fm-watch-triage.test.sh index 8937798ad75..0fd4ffce161 100755 --- a/tests/fm-watch-triage.test.sh +++ b/tests/fm-watch-triage.test.sh @@ -227,16 +227,18 @@ test_crew_is_provably_working_classifier() { # that merely mentions "paused" does not false-match, and a genuine blocker stays a # blocker. test_status_is_paused_classifier() { - status_is_paused 'paused: holding for the upstream release' || fail "paused verb not recognized" - status_is_paused ' paused: waiting on a rate-limit reset' || fail "leading-space paused verb not recognized" + status_is_paused 'paused: holding for the upstream release; owner=release team; clears=release artifact is published' || fail "owned pause not recognized" + status_is_paused ' paused: waiting on a rate-limit reset; owner=vendor; clears=reset time arrives' || fail "leading-space owned pause not recognized" + status_is_paused 'paused: holding for the upstream release' && fail "a pause without owner and clearing condition was accepted" status_is_paused 'blocked: the build is paused upstream' && fail "a blocked line mentioning paused false-matched" status_is_paused 'working: paused the animation loop' && fail "a working line mentioning paused false-matched" status_is_paused 'done: shipped' && fail "done classified as paused" status_is_paused '' && fail "empty line classified as paused" # A pause is deliberately NOT captain-relevant: it is a stop-nagging signal, not # work to keep surfacing. - status_is_captain_relevant 'paused: holding for the upstream release' && fail "paused is captain-relevant (should not be)" - pass "status_is_paused: only the leading paused verb matches, and paused is not captain-relevant" + status_is_captain_relevant 'paused: holding for the upstream release; owner=release team; clears=release artifact is published' && fail "valid paused is captain-relevant (should not be)" + status_is_captain_relevant 'paused: holding for the upstream release' || fail "an incomplete pause was not surfaced as actionable" + pass "status_is_paused: only owned-and-clearing leading pause declarations match" } # status_pause_is_failure: the failure-pause discriminator. A crewmate reporting a @@ -266,11 +268,11 @@ test_failure_pause_is_failure_classifier() { # (2) Genuine deliberate pauses keep absorbing; regressing this would flood the # supervisor with the idle waits the pause verb exists to silence. local -a deliberate=( - "paused: waiting for the captain's decision on the IDC rollout" - 'paused: waiting on upstream CI' - 'paused: rate limit until 15:00' - 'paused: holding for the upstream tool release' - 'paused: awaiting PR review before the rebase' + "paused: waiting for the captain's decision on the IDC rollout; owner=captain; clears=captain records the IDC decision" + 'paused: waiting on upstream CI; owner=upstream CI; clears=required checks become terminal' + 'paused: rate limit until 15:00; owner=vendor; clears=clock reaches 15:00' + 'paused: holding for the upstream tool release; owner=tool maintainer; clears=release artifact is published' + 'paused: awaiting PR review before the rebase; owner=reviewer; clears=review verdict is posted' ) for line in "${deliberate[@]}"; do status_pause_is_failure "$line" && fail "a deliberate pause was flagged a failure: $line" @@ -280,7 +282,7 @@ test_failure_pause_is_failure_classifier() { # (3) The documented tradeoff: a deliberate wait that mentions a failure in PASSING, # past the headline, still absorbs - position is the discriminator, not presence. - line='paused: waiting for the captain to decide how to handle the failed Shopify webhook' + line='paused: waiting for the captain to decide how to handle the failed Shopify webhook; owner=captain; clears=captain records the webhook decision' status_pause_is_failure "$line" && fail "a passing failure mention past the headline escalated" status_is_paused "$line" || fail "a passing failure mention past the headline stopped being a pause" # ...but the SAME failure word inside the headline does escalate, which is the @@ -288,11 +290,26 @@ test_failure_pause_is_failure_classifier() { status_pause_is_failure 'paused: the Shopify webhook failed, waiting for the captain' \ || fail "a failure word inside the headline did not escalate" - # (4) The headline bound itself: text after the reason's first colon is detail, not + # (4) The headline bound itself: text after a clause's first colon is detail, not # headline, so a failure word only in the detail tail does not escalate. - status_pause_is_failure 'paused: waiting on the vendor window: previous attempt failed' \ + status_pause_is_failure 'paused: waiting on the vendor window: previous attempt failed; owner=vendor; clears=maintenance window opens' \ && fail "a failure word in the detail tail (past the first colon) escalated" + # (4b) The bound is per CLAUSE, not per line: a failure reported in a later prose + # clause is still a failure report, so the semicolons the contract fields need + # cannot become a place to hide one. + status_pause_is_failure 'paused: retrying the upstream fetch; the previous run failed with an i/o timeout; owner=vendor; clears=vendor API returns 200' \ + || fail "a failure reported in a later prose clause was not detected" + status_is_paused 'paused: retrying the upstream fetch; the previous run failed with an i/o timeout; owner=vendor; clears=vendor API returns 200' \ + && fail "a failure reported in a later prose clause was still absorbed as a declared wait" + # ...while the contract fields themselves are structured metadata, not prose: an + # observable clearing condition may legitimately name the failure it ends. + status_pause_is_failure 'paused: waiting on the vendor window; owner=vendor; clears=the upstream i/o timeout stops' \ + && fail "failure vocabulary inside a contract field escalated" + # A declaration with contract fields but no reason of its own is not a declaration. + status_is_paused 'paused: owner=vendor; clears=the vendor window opens' \ + && fail "a pause with no stated reason earned absorption" + # (5) Both knobs are data. A home can retune the vocabulary and the bound, and the # discriminator follows the configurable pause verb rather than the literal. # shellcheck disable=SC2034 # Read by fm-classify-lib.sh (sourced above), not here. @@ -333,10 +350,10 @@ test_failure_pause_is_failure_classifier() { # # The pause half is proven from the crewmate's own durable status stream, not from # the fm-crew-state.sh verdict, so every pause case here needs a real status file; -# the full pause truth table (including the run-step verdicts that used to veto a -# pause) lives in tests/fm-watch-pause-absorb.test.sh. +# the full pause truth table lives in tests/fm-watch-pause-absorb.test.sh. test_crew_absorb_class_classifier() { - local dir fakebin malformed state + local dir fakebin malformed state owned_pause + owned_pause='paused: awaiting ordered PR merges; owner=merge supervisor; clears=ordered merges complete' dir=$(make_case absorb-class); fakebin="$dir/fakebin"; state="$dir/state" export FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" export FM_FAKE_CREW_STATE FM_STATE_OVERRIDE="$state" @@ -345,6 +362,12 @@ test_crew_absorb_class_classifier() { [ "$(crew_absorb_class a)" = working ] || fail "active run-step not classed working" FM_FAKE_CREW_STATE='state: working · source: run-step · validating (running) · liveness: alive (2 procs) · step: test' [ "$(crew_absorb_class a)" = working ] || fail "alive command-step liveness not classed working" + crew_has_measured_progress a || fail "alive CPU/process progress was not recognized at the wedge boundary" + FM_FAKE_CREW_STATE='state: working · source: run-step · validating (running) · liveness: unknown (grade: present-no-progress; 2 procs; PRESENT BUT NOT PROGRESSING: stable membership and no persistent process advanced cpu in 30s (best +0.01s)) · step: test' + [ "$(crew_state_liveness_verdict "$FM_FAKE_CREW_STATE")" = unknown ] \ + || fail "the probe's present-no-progress grade was not read as unknown" + [ "$(crew_absorb_class a)" = none ] || fail "present-no-progress liveness was absorbed as working" + ! crew_has_measured_progress a || fail "near-zero CPU progress was treated as measured work" FM_FAKE_CREW_STATE='state: working · source: run-step · validating (running) · liveness: dead (0 procs) · step: test' [ "$(crew_absorb_class a)" = none ] || fail "dead command-step liveness was absorbed as working" ! crew_is_paused a || fail "dead command-step liveness was absorbed as paused" @@ -370,38 +393,37 @@ test_crew_absorb_class_classifier() { || fail "a state line without a liveness field changed classification" FM_FAKE_CREW_STATE='state: working · source: pane · harness busy' [ "$(crew_absorb_class a)" = working ] || fail "busy pane not classed working" - printf 'paused: awaiting upstream\n' > "$state/a.status" + printf 'paused: awaiting upstream; owner=upstream; clears=upstream release lands\n' > "$state/a.status" FM_FAKE_CREW_STATE='state: paused · source: status-log · awaiting upstream' - [ "$(crew_absorb_class a)" = paused ] || fail "declared pause not classed paused" - crew_is_paused a || fail "crew_is_paused did not recognize a paused verdict" + [ "$(crew_absorb_class a 'paused: awaiting upstream; owner=upstream; clears=upstream release lands')" = paused ] || fail "declared pause not classed paused" + crew_is_paused a 'paused: awaiting upstream; owner=upstream; clears=upstream release lands' || fail "crew_is_paused did not recognize a paused verdict" ! crew_is_provably_working a || fail "a paused crew was treated as provably working" - # A terminal run-step with NO pause in the stream is not absorbable... + # A terminal run-step remains authoritative with or without a pause in the stream. printf 'done: PR ready\n' > "$state/a.status" - FM_FAKE_CREW_STATE='state: done · source: run-step · checks green: PR ready for review' + FM_FAKE_CREW_STATE='state: done · source: run-step · checks green: PR ready for review · run: 01RUN' [ "$(crew_absorb_class a)" = none ] || fail "terminal run-step without pause context was absorbed" - # ...and yields to one when the caller supplies it. + printf '%s\n' "$owned_pause" > "$state/a.status" + [ "$(crew_absorb_class a "$owned_pause")" = none ] \ + || fail "terminal run-step yielded to a declared pause for absorb classification" + # Absorption is earned by the registration, not by the verb: a bare pause names + # nobody who owns the wait and no fact that ends it, so it stays actionable. + FM_FAKE_CREW_STATE='state: paused · source: status-log · awaiting ordered PR merges' printf 'paused: awaiting ordered PR merges\n' > "$state/a.status" - [ "$(crew_absorb_class a 'paused: awaiting ordered PR merges')" = paused ] \ - || fail "terminal run-step did not yield to a declared pause for absorb classification" - # A FAILED run-step yields to a declared pause too, as of 2026-08-03. It used not - # to, which silenced nothing but did surface plenty: fm-crew-state.sh maps a - # routinely CANCELLED run onto `failed`, so ordinary teardown-cancelled runs - # wedge-escalated lanes that had legitimately declared a wait. A crewmate that - # genuinely failed reports it with the captain-relevant `failed:` verb, and a - # failure written under the pause verb is caught by status_pause_is_failure below - - # neither route depends on this veto. - FM_FAKE_CREW_STATE='state: failed · source: run-step · validation failed' - [ "$(crew_absorb_class a 'paused: awaiting ordered PR merges')" = paused ] \ - || fail "failed run-step did not yield to a declared external wait" - printf 'paused: error: drive run: read response: i/o timeout\n' > "$state/a.status" - [ "$(crew_absorb_class a 'paused: error: drive run: read response: i/o timeout')" = none ] \ + [ "$(crew_absorb_class a 'paused: awaiting ordered PR merges')" = none ] \ + || fail "a pause with no owner and no clearing condition was absorbed" + printf '%s\n' "$owned_pause" > "$state/a.status" + FM_FAKE_CREW_STATE='state: failed · source: run-step · validation failed · run: 01RUN' + [ "$(crew_absorb_class a "$owned_pause")" = none ] \ + || fail "failed run-step yielded to a pause declaration" + printf 'paused: error: drive run: read response: i/o timeout; owner=drive; clears=the drive run returns\n' > "$state/a.status" + [ "$(crew_absorb_class a 'paused: error: drive run: read response: i/o timeout; owner=drive; clears=the drive run returns')" = none ] \ || fail "a failure reported under the pause verb was absorbed behind a failed run-step" printf 'blocked: stream advanced while pipeline state was read\n' > "$state/a.status" - [ "$(crew_absorb_class a 'paused: awaiting ordered PR merges')" = none ] \ + [ "$(crew_absorb_class a "$owned_pause")" = none ] \ || fail "a stale caller-supplied pause overrode the current status line" - printf 'paused: awaiting ordered PR merges\n' > "$state/a.status" + printf '%s\n' "$owned_pause" > "$state/a.status" export FM_FAKE_CREW_STATE_APPEND_STATUS='blocked: stream advanced during pipeline state read' - [ "$(crew_absorb_class a 'paused: awaiting ordered PR merges')" = none ] \ + [ "$(crew_absorb_class a "$owned_pause")" = none ] \ || fail "a pause invalidated during the crew-state read was absorbed" unset FM_FAKE_CREW_STATE_APPEND_STATUS FM_FAKE_CREW_STATE='state: working · source: status-log · working: compiling' @@ -446,22 +468,29 @@ test_signal_crew_provably_working_classifier() { # --- benign wakes are absorbed ONLY when the crewmate is provably working ----- test_provably_working_signal_absorbed() { - local dir state fakebin out status_file pid + local dir state fakebin out status_file sig pid dir=$(make_case provably-working-signal); state="$dir/state"; fakebin="$dir/fakebin"; out="$dir/watch.out" status_file="$state/task.status" printf 'working: compiling step 2\n' > "$status_file" + sig=$(seen_sig "$status_file") # The crewmate's pipeline is in an actively-running step: positive evidence it is # still working, so a no-verb working: signal is absorbed (the original low-churn # case during a long validation). export FM_FAKE_CREW_STATE='state: working · source: run-step · validating (running)' watch_bg "$state" "$fakebin" "$out" pid=$! - if ! wait_live "$pid" 30; then - reap "$pid"; fail "watcher exited for a working: signal whose crew is provably working (should absorb): $(cat "$out")" + if ! wait_file_value "$state/.seen-task_status" "$sig" 300; then + if ! kill -0 "$pid" 2>/dev/null; then + reap "$pid"; fail "watcher exited for a working: signal whose crew is provably working (should absorb): $(cat "$out")" + fi + reap "$pid"; fail "provably-working signal did not advance its .seen-* suppressor" fi + kill -0 "$pid" 2>/dev/null \ + || { reap "$pid"; fail "watcher exited after absorbing a provably-working signal: $(cat "$out")"; } [ ! -s "$out" ] || fail "provably-working signal printed a wake reason: $(cat "$out")" [ ! -s "$state/.wake-queue" ] || fail "provably-working signal enqueued a durable wake record" - [ -s "$state/.seen-task_status" ] || fail "provably-working signal did not advance its .seen-* suppressor" + [ "$(cat "$state/.seen-task_status" 2>/dev/null || true)" = "$sig" ] \ + || fail "provably-working signal did not persist the observed .seen-* suppressor" [ -e "$state/.last-watcher-beat" ] || fail "watcher beacon was not touched while absorbing" reap "$pid" pass "a no-verb signal whose crew is provably working is absorbed (no exit, no queue, suppressor advanced, beacon present)" @@ -735,7 +764,7 @@ $ wait-for-upstream-release Press enter to confirm or esc to cancel EOF printf 'window=%s\nkind=ship\nharness=codex\n' "$window" > "$state/paused-permission.meta" - printf 'paused: awaiting the upstream release\n' > "$statusf" + printf 'paused: awaiting the upstream release; owner=release team; clears=release artifact is published\n' > "$statusf" printf '%s' "$(seen_sig "$statusf")" > "$state/.seen-paused-permission_status" key=$(printf '%s' "$window" | tr ':/.' '___') export FM_FAKE_CREW_STATE='state: paused · source: status-log · awaiting the upstream release' @@ -744,10 +773,10 @@ EOF FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_PAUSE_RESURFACE_SECS=999 \ FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & pid=$! - if ! wait_live "$pid" 30; then - reap "$pid"; fail "a declared pause fell through to the permission-prompt stale path: $(cat "$out")" - fi - [ -e "$state/.paused-$key" ] || { reap "$pid"; fail "declared pause did not create its suppression marker before permission-prompt detection"; } + fm_test_wait_for_file "$state/.paused-$key" "$pid" \ + || { reap "$pid"; fail "declared pause did not create its suppression marker before permission-prompt detection: $(cat "$out")"; } + kill -0 "$pid" 2>/dev/null \ + || { reap "$pid"; fail "a declared pause fell through to the permission-prompt stale path: $(cat "$out")"; } [ ! -e "$state/.stale-permission-$key" ] || { reap "$pid"; fail "declared pause incorrectly entered permission-prompt escalation tracking"; } [ ! -s "$state/.wake-queue" ] || { reap "$pid"; fail "declared pause enqueued a permission-prompt stale wake"; } [ ! -s "$out" ] || { reap "$pid"; fail "declared pause printed a permission-prompt stale wake: $(cat "$out")"; } @@ -756,6 +785,128 @@ EOF pass "a declared pause enters pause suppression before permission-prompt stale detection" } +# The pause declaration itself is a state transition, so the normal watcher must +# register it before returning the initial one-time signal wake. This is the exact +# consequence the old classifier-only coverage missed: status_is_paused accepted the +# line, but the first watcher exited without ever creating .paused-. +# Start dirty to prove that declaration registration also leaves the ordinary +# wedge track immediately instead of inheriting escalation history forever. +test_owned_pause_signal_registers_marker() { + local dir state fakebin out capture_file statusf window key pid consequences= + dir=$(make_case owned-pause-signal-marker); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/owned-pause.status" + window="test:fm-owned-pause" + printf 'idle under a declared hold\n' > "$capture_file" + printf 'window=%s\nkind=ship\nharness=codex\n' "$window" > "$state/owned-pause.meta" + printf 'paused: validation hold; owner=supervisor; clears=supervisor lifts the hold after host load recovers\n' > "$statusf" + key=$(printf '%s' "$window" | tr ':/.' '___') + printf '1\n' > "$state/.stale-since-$key" + printf '4\n' > "$state/.wedge-escalations-$key" + export FM_FAKE_CREW_STATE='state: paused · source: status-log · validation hold' + + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_PAUSE_RESURFACE_SECS=999 \ + FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + wait_for_exit "$pid" 40 || { reap "$pid"; fail "the initial owned-pause signal did not surface once"; } + grep -F "signal: $statusf" "$out" >/dev/null \ + || fail "the initial owned-pause declaration did not produce its one-time signal wake" + [ -e "$state/.paused-$key" ] \ + || consequences="${consequences}pause marker absent; " + [ ! -e "$state/.paused-rechecked-$key" ] \ + || consequences="${consequences}registration seeded the recheck cache before the open-decision fold ran; " + [ ! -e "$state/.stale-since-$key" ] \ + || consequences="${consequences}pre-existing stale timer remains; " + [ ! -e "$state/.wedge-escalations-$key" ] \ + || consequences="${consequences}pre-existing wedge escalation count remains; " + [ -z "$consequences" ] || fail "pause declaration consequences failed: $consequences" + unset FM_FAKE_CREW_STATE + pass "a valid owned-and-clearing pause registers and clears dirty wedge history before its initial signal wake returns" +} + +# A registered owned pause must buy the lane the long pause cadence even when the +# pane hash changes between watcher processes. The real incident re-armed after +# every stale wake, observed a fresh rendered hash, and woke again because no pause +# marker had ever been committed. +test_owned_pause_absorbs_repeated_changed_stale() { + local dir state fakebin out capture_file statusf window key sig pid token pane_hash wakes=0 + dir=$(make_case owned-pause-repeat); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/owned-repeat.status" + window="test:fm-owned-repeat" + printf 'window=%s\nkind=ship\nharness=codex\n' "$window" > "$state/owned-repeat.meta" + printf 'paused: validation hold; owner=supervisor; clears=supervisor lifts the hold after host load recovers\n' > "$statusf" + sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-owned-repeat_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + export FM_FAKE_CREW_STATE='state: paused · source: status-log · validation hold' + + for token in one two; do + printf 'idle under declared hold, render %s\n' "$token" > "$capture_file" + pane_hash=$(hash_text "idle under declared hold, render $token") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + : > "$out" + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_PAUSE_RESURFACE_SECS=999 \ + FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + if wait_for_exit "$pid" 12; then + wakes=$((wakes + 1)) + else + reap "$pid" + fi + : > "$state/.wake-queue" + done + + [ "$wakes" -eq 0 ] \ + || fail "an owned pause woke on $wakes of 2 changed-hash polls instead of using the bounded pause cadence" + [ -e "$state/.paused-$key" ] || fail "repeat-poll absorption lost the owned-pause marker" + [ ! -e "$state/.stale-since-$key" ] || fail "an owned pause entered wedge timing" + unset FM_FAKE_CREW_STATE + pass "an owned pause absorbs repeated changed-hash polls on the bounded pause cadence" +} + +# The pause benefit is earned by naming both the wait owner and the observable +# condition that clears it. A bare pause verb, or a declaration with only one of +# those fields, remains stopped work and must surface without any pause marker. +test_unowned_or_uncleared_pause_does_not_absorb() { + local line name dir state fakebin out capture_file statusf window key pane_hash sig pid absorbed=0 markers=0 + for name in missing-owner missing-clearing; do + case "$name" in + missing-owner) line='paused: validation hold; clears=host load falls below the declared threshold' ;; + *) line='paused: validation hold; owner=supervisor' ;; + esac + dir=$(make_case "pause-$name"); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/pause-$name.status" + window="test:fm-pause-$name" + printf 'idle with incomplete pause declaration\n' > "$capture_file" + printf 'window=%s\nkind=ship\n' "$window" > "$state/pause-$name.meta" + printf '%s\n' "$line" > "$statusf" + sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-pause-${name}_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + pane_hash=$(hash_text "idle with incomplete pause declaration") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '1\n' > "$state/.count-$key" + export FM_FAKE_CREW_STATE='state: paused · source: status-log · incomplete declaration' + + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_STALE_ESCALATE_SECS=999 \ + FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + if wait_for_exit "$pid" 30; then + : + else + absorbed=$((absorbed + 1)) + reap "$pid" + fi + [ ! -e "$state/.paused-$key" ] || markers=$((markers + 1)) + done + unset FM_FAKE_CREW_STATE + if [ "$absorbed" -ne 0 ] || [ "$markers" -ne 0 ]; then + fail "$absorbed of 2 incomplete declarations were absorbed and $markers of 2 received pause markers" + fi + pass "a pause missing its owner or clearing condition surfaces without absorption" +} + # A macOS TCC dialog can block the foreground command while the harness footer # remains busy and its elapsed counter changes forever. The watcher therefore # hashes semantic progress without that footer: real output resets the timer, @@ -1009,17 +1160,15 @@ test_nonterminal_stale_not_working_surfaced() { pass "a not-provably-working non-terminal stale is surfaced immediately (never left to wait out the timer)" } -# A terminal no-mistakes run-step remains authoritative current-state evidence, -# but for stale-pane ABSORB classification a newer durable declared pause says why -# the finished crewmate is intentionally idle. The pause must therefore enter the -# long-cadence path without losing cadence markers from an earlier watcher cycle. -test_terminal_run_step_declared_pause_absorbed_with_markers() { +# A terminal no-mistakes run-step remains authoritative current-state evidence even +# when the durable status stream still ends in a declared pause. +test_terminal_run_step_overrides_declared_pause() { local dir state fakebin out capture_file window key pane_hash sig pid dir=$(make_case terminal-run-declared-pause); state="$dir/state"; fakebin="$dir/fakebin" out="$dir/watch.out"; capture_file="$dir/pane.txt"; window="default:w6:pKV" printf 'idle after checks passed, awaiting ordered merges\n' > "$capture_file" printf 'window=%s\nkind=ship\nharness=codex\n' "$window" > "$state/terminal-paused.meta" - printf 'paused: waiting on the captain to merge PRs #63, #64, and #65 in order before D4 can start\n' \ + printf 'paused: waiting on ordered PR merges before D4 can start; owner=captain; clears=PRs #63, #64, and #65 are merged in order\n' \ > "$state/terminal-paused.status" sig=$(seen_sig "$state/terminal-paused.status"); printf '%s' "$sig" > "$state/.seen-terminal-paused_status" key=$(printf '%s' "$window" | tr ':/.' '___') @@ -1027,33 +1176,27 @@ test_terminal_run_step_declared_pause_absorbed_with_markers() { printf '%s' "$pane_hash" > "$state/.hash-$key" printf '1\n' > "$state/.count-$key" : > "$state/.paused-resurfaced-$key" - export FM_FAKE_CREW_STATE='state: done · source: run-step · checks green: PR ready for review' + export FM_FAKE_CREW_STATE='state: done · source: run-step · checks green: PR ready for review · run: 01RUN' PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_PAUSE_RESURFACE_SECS=999 \ FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & pid=$! - if ! wait_live "$pid" 30; then - reap "$pid"; fail "terminal run-step overrode its declared pause and surfaced: $(cat "$out")" - fi - [ ! -s "$out" ] || { reap "$pid"; fail "terminal-run declared pause printed a stale wake: $(cat "$out")"; } - [ ! -s "$state/.wake-queue" ] || { reap "$pid"; fail "terminal-run declared pause enqueued a stale wake"; } - [ "$(cat "$state/.stale-$key" 2>/dev/null || true)" = "$pane_hash" ] \ - || { reap "$pid"; fail "terminal-run declared pause did not advance its stale suppressor"; } - [ -e "$state/.paused-$key" ] || { reap "$pid"; fail "terminal-run declared pause did not create its pause marker"; } - [ -e "$state/.paused-rechecked-$key" ] || { reap "$pid"; fail "terminal-run declared pause did not retain its authoritative recheck marker"; } - [ -e "$state/.paused-resurfaced-$key" ] || { reap "$pid"; fail "terminal-run declared pause lost its long-cadence resurface marker"; } - [ ! -e "$state/.stale-since-$key" ] || { reap "$pid"; fail "terminal-run declared pause started a wedge timer"; } + wait_for_exit "$pid" 40 || { reap "$pid"; fail "terminal run-step was absorbed behind a declared pause"; } + grep -F "stale: $window" "$out" >/dev/null \ + || fail "terminal run-step did not surface its stale pane: $(cat "$out")" + grep -F "$window" "$state/.wake-queue" >/dev/null \ + || fail "terminal run-step did not enqueue an actionable stale wake" reap "$pid" unset FM_FAKE_CREW_STATE - pass "terminal run-step plus durable declared pause is absorbed and preserves pause cadence markers" + pass "terminal run-step remains actionable over a durable declared pause" } test_surface_nonterminal_stale_clears_pause_only_after_status_resumes() { local dir state window key dir=$(make_case surface-stale-pause-markers); state="$dir/state"; window="default:w6:pKV" printf 'window=%s\nkind=ship\n' "$window" > "$state/marker-owner.meta" - printf 'paused: awaiting ordered PR merges\n' > "$state/marker-owner.status" + printf 'paused: awaiting ordered PR merges; owner=merge supervisor; clears=ordered merges complete\n' > "$state/marker-owner.status" key=$(printf '%s' "$window" | tr ':/.' '___') : > "$state/.paused-$key" : > "$state/.paused-rechecked-$key" @@ -1102,7 +1245,7 @@ test_nonterminal_stale_paused_absorbed_then_resurfaced() { statusf="$state/held.status" # A DECLARED pause (not captain-relevant), .seen-* primed so the signal scan does # not pre-empt the stale path. - printf 'paused: holding for the upstream tool release\n' > "$statusf" + printf 'paused: holding for the upstream tool release; owner=tool maintainer; clears=release artifact is published\n' > "$statusf" sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-held_status" key=$(printf '%s' "$window" | tr ':/.' '___') pane_hash=$(hash_text "idle, holding for upstream") @@ -1144,7 +1287,8 @@ test_nonterminal_stale_paused_absorbed_then_resurfaced() { pid=$! wait_for_exit "$pid" 40 || fail "watcher did not re-surface a declared pause past the threshold" grep -F "stale: $window" "$out" >/dev/null || fail "re-surface did not print a stale wake" - grep -F "awaiting external" "$out" >/dev/null || fail "re-surface was not labeled a paused/awaiting-external recheck" + grep -F "owner=tool maintainer" "$out" >/dev/null || fail "re-surface omitted the pause owner" + grep -F "clears=release artifact is published" "$out" >/dev/null || fail "re-surface omitted the clearing condition" grep -F "possible wedge" "$out" >/dev/null && fail "a declared pause was mislabeled a possible wedge" [ -e "$state/.paused-$key" ] || fail "long-cadence pause re-surface lost the pause marker" [ -e "$state/.paused-rechecked-$key" ] || fail "long-cadence pause re-surface lost the authoritative recheck marker" @@ -1222,7 +1366,7 @@ test_herdr_blocked_transition_enters_pause_absorb_path() { window="default:w6:p3H" key=$(printf '%s' "$window" | tr ':/.' '___') printf 'window=%s\nbackend=herdr\nkind=ship\n' "$window" > "$state/herdr-paused.meta" - printf 'paused: awaiting an external release\n' > "$state/herdr-paused.status" + printf 'paused: awaiting an external release; owner=release team; clears=release artifact is published\n' > "$state/herdr-paused.status" printf 'stable-pane-hash' > "$state/.hash-$key" export FM_FAKE_CREW_STATE='state: paused · source: status-log · awaiting an external release' record=$(fm_transition_record 'w6:p3H' 'w6' '' blocked codex) @@ -1251,7 +1395,7 @@ test_secondmate_paused_resurfaces_in_normal_mode() { window="test:fm-secondmate-held" printf 'idle awaiting external\n' > "$capture_file" printf 'window=%s\nkind=secondmate\n' "$window" > "$state/secondmate-held.meta" - printf 'paused: awaiting the upstream release\n' > "$statusf" + printf 'paused: awaiting the upstream release; owner=release team; clears=release artifact is published\n' > "$statusf" back=$(( $(date +%s) - 500 )) if [ "$(uname)" = Darwin ]; then touch -mt "$(date -r "$back" '+%Y%m%d%H%M.%S')" "$statusf" else touch -m -d "@$back" "$statusf"; fi @@ -1267,7 +1411,8 @@ test_secondmate_paused_resurfaces_in_normal_mode() { pid=$! wait_for_exit "$pid" 40 || fail "watcher did not re-surface a paused secondmate" grep -F "stale: $window" "$out" >/dev/null || fail "paused secondmate did not emit a stale recheck" - grep -F "awaiting external" "$out" >/dev/null || fail "paused secondmate recheck omitted its external-wait reason" + grep -F "owner=release team" "$out" >/dev/null || fail "paused secondmate recheck omitted its owner" + grep -F "clears=release artifact is published" "$out" >/dev/null || fail "paused secondmate recheck omitted its clearing condition" grep -F "possible wedge" "$out" >/dev/null && fail "paused secondmate was mislabeled a wedge" unset FM_FAKE_CREW_STATE pass "a declared paused secondmate re-surfaces on the bounded normal-mode cadence" @@ -1329,7 +1474,7 @@ test_nonterminal_stale_pause_transitions_reclassify_unchanged_hash() { out="$dir/watch.out"; capture_file="$dir/pane.txt"; window="test:fm-transition" printf 'idle awaiting external\n' > "$capture_file" printf 'window=%s\nkind=ship\n' "$window" > "$state/transition.meta" - printf 'paused: awaiting the upstream release\n' > "$state/transition.status" + printf 'paused: awaiting the upstream release; owner=release team; clears=release artifact is published\n' > "$state/transition.status" sig=$(seen_sig "$state/transition.status"); printf '%s' "$sig" > "$state/.seen-transition_status" key=$(printf '%s' "$window" | tr ':/.' '___') pane_hash=$(hash_text "idle awaiting external") @@ -1374,7 +1519,7 @@ test_nonterminal_paused_rechecks_authoritative_state() { out="$dir/watch.out"; capture_file="$dir/pane.txt"; window="test:fm-pause-recheck" printf 'idle awaiting external\n' > "$capture_file" printf 'window=%s\nkind=ship\n' "$window" > "$state/pause-recheck.meta" - printf 'paused: awaiting the upstream release\n' > "$state/pause-recheck.status" + printf 'paused: awaiting the upstream release; owner=release team; clears=release artifact is published\n' > "$state/pause-recheck.status" sig=$(seen_sig "$state/pause-recheck.status"); printf '%s' "$sig" > "$state/.seen-pause-recheck_status" key=$(printf '%s' "$window" | tr ':/.' '___') pane_hash=$(hash_text "idle awaiting external") @@ -1382,6 +1527,7 @@ test_nonterminal_paused_rechecks_authoritative_state() { printf '%s' "$pane_hash" > "$state/.stale-$key" printf '1\n' > "$state/.count-$key" : > "$state/.paused-$key" + printf '%s' "$sig" > "$state/.paused-rechecked-$key" export FM_FAKE_CREW_STATE='state: working · source: run-step · validating (running)' PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ @@ -1395,7 +1541,7 @@ test_nonterminal_paused_rechecks_authoritative_state() { [ -s "$state/.stale-since-$key" ] || { reap "$pid"; fail "authoritative active run did not resume wedge tracking"; } reap "$pid" unset FM_FAKE_CREW_STATE - pass "a declared pause is periodically rechecked against authoritative active-run state" + pass "a cached declared pause cannot hide a later active run" } test_paused_authoritative_working_preserves_wedge_timer() { @@ -1404,7 +1550,7 @@ test_paused_authoritative_working_preserves_wedge_timer() { out="$dir/watch.out"; capture_file="$dir/pane.txt"; window="test:fm-paused-working" printf 'idle awaiting external\n' > "$capture_file" printf 'window=%s\nkind=ship\n' "$window" > "$state/paused-working.meta" - printf 'paused: awaiting the upstream release\n' > "$state/paused-working.status" + printf 'paused: awaiting the upstream release; owner=release team; clears=release artifact is published\n' > "$state/paused-working.status" sig=$(seen_sig "$state/paused-working.status"); printf '%s' "$sig" > "$state/.seen-paused-working_status" key=$(printf '%s' "$window" | tr ':/.' '___') pane_hash=$(hash_text "idle awaiting external") @@ -1438,6 +1584,92 @@ test_paused_authoritative_working_preserves_wedge_timer() { pass "a paused status overridden by authoritative working preserves its wedge timer and escalates" } +# The step-liveness probe owns the CPU-versus-elapsed threshold and exposes its +# conclusion as alive (measured CPU/process progress) or, for near-zero CPU +# progress, a present-no-progress grade under unknown - never a fourth verdict. +# At the wedge threshold the watcher must consume that verdict again: +# pane staleness alone cannot distinguish the measured 2026-08-04 starved run from +# a hang. +test_cpu_progress_suppresses_static_pane_wedge() { + local dir state fakebin out capture_file statusf window key pane_hash sig pid before after waited + dir=$(make_case cpu-progress-static-pane); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/cpu-progress.status" + window="test:fm-cpu-progress" + printf 'static pane while no-mistakes owns the foreground wait\n' > "$capture_file" + printf 'window=%s\nkind=ship\n' "$window" > "$state/cpu-progress.meta" + printf 'working: validation still computing; nothing needs attention until the next gate\n' > "$statusf" + sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-cpu-progress_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + pane_hash=$(hash_text "static pane while no-mistakes owns the foreground wait") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '%s' "$pane_hash" > "$state/.stale-$key" + printf '1\n' > "$state/.count-$key" + before=$(( $(date +%s) - 500 )) + printf '%s\n' "$before" > "$state/.stale-since-$key" + # Field measurements: 594s CPU / 8,880s elapsed = 6.69%, plus 40s / 1,200s + # = 3.33%. Both are low-duty-cycle but steadily progressing, so the probe's + # public verdict is alive. + export FM_FAKE_CREW_STATE='state: working · source: run-step · validating · liveness: alive (2 procs; cpu/elapsed 594s/8880s=6.69%, 40s/1200s=3.33%; persistent cpu advanced) · step: test' + + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_STALE_ESCALATE_SECS=2 \ + FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + # Wait for the observation itself rather than for a fixed slice of wall clock: + # under load the watcher can still be building its stable-hash count when a + # fixed window expires, which says nothing about the verdict under test. + after=0 + waited=0 + while [ "$waited" -lt 300 ]; do + kill -0 "$pid" 2>/dev/null \ + || { reap "$pid"; fail "a static pane with measured CPU progress raised a wedge wake: $(cat "$out")"; } + after=$(cat "$state/.stale-since-$key" 2>/dev/null || echo 0) + [ "$after" -gt "$before" ] && break + sleep 0.1 + waited=$((waited + 1)) + done + [ "$after" -gt "$before" ] \ + || { reap "$pid"; fail "measured CPU progress did not refresh the bounded wedge observation"; } + kill -0 "$pid" 2>/dev/null \ + || { reap "$pid"; fail "a static pane with measured CPU progress raised a wedge wake: $(cat "$out")"; } + [ ! -s "$state/.wake-queue" ] \ + || { reap "$pid"; fail "measured CPU progress enqueued a possible-wedge wake"; } + reap "$pid" + unset FM_FAKE_CREW_STATE + pass "a static pane with measured low-duty-cycle CPU progress remains provably working" +} + +test_near_zero_cpu_static_pane_still_wedge_escalates() { + local dir state fakebin out capture_file statusf window key pane_hash sig pid + dir=$(make_case cpu-stalled-static-pane); state="$dir/state"; fakebin="$dir/fakebin" + out="$dir/watch.out"; capture_file="$dir/pane.txt"; statusf="$state/cpu-stalled.status" + window="test:fm-cpu-stalled" + printf 'static pane with a frozen worktree process\n' > "$capture_file" + printf 'window=%s\nkind=ship\n' "$window" > "$state/cpu-stalled.meta" + printf 'working: validation step claims to be running\n' > "$statusf" + sig=$(seen_sig "$statusf"); printf '%s' "$sig" > "$state/.seen-cpu-stalled_status" + key=$(printf '%s' "$window" | tr ':/.' '___') + pane_hash=$(hash_text "static pane with a frozen worktree process") + printf '%s' "$pane_hash" > "$state/.hash-$key" + printf '%s' "$pane_hash" > "$state/.stale-$key" + printf '1\n' > "$state/.count-$key" + printf '%s\n' $(( $(date +%s) - 500 )) > "$state/.stale-since-$key" + # The probe's measured trace-noise floor is +0.01s over 30s = 0.033%; presence it + # cannot prove is progressing stays a graded unknown, not a blanket presence-based + # working verdict. + export FM_FAKE_CREW_STATE='state: working · source: run-step · validating · liveness: unknown (grade: present-no-progress; 2 procs; PRESENT BUT NOT PROGRESSING: stable membership and no persistent process advanced cpu in 30s (best +0.01s)) · step: test' + + PATH="$fakebin:$PATH" FM_FAKE_TMUX_WINDOW="$window" FM_FAKE_TMUX_CAPTURE="$capture_file" \ + FM_STATE_OVERRIDE="$state" FM_CREW_STATE_BIN="$fakebin/fm-crew-state.sh" FM_STALE_ESCALATE_SECS=2 \ + FM_POLL=1 FM_SIGNAL_GRACE=1 FM_CHECK_INTERVAL=999999 FM_HEARTBEAT=999999 "$WATCH" > "$out" & + pid=$! + wait_for_exit "$pid" 40 || { reap "$pid"; fail "near-zero CPU progress received blanket wedge suppression"; } + grep -F "possible wedge" "$out" >/dev/null \ + || fail "near-zero CPU progress did not retain the possible-wedge escalation" + unset FM_FAKE_CREW_STATE + pass "a static pane with near-zero CPU progress still wedge-escalates" +} + # --- consecutive wedge escalations on the same pane demand deep inspection ---- # Root cause of the PR #252 incident's ~20 minutes of unnoticed green: each # wedge escalation fires, gets classified as "still validating" one poll later @@ -1747,7 +1979,7 @@ test_afk_paused_changed_pane_hands_off_plain_stale() { printf 'idle, awaiting upstream\n' > "$capture_file" printf 'window=%s\nkind=ship\n' "$window" > "$state/afk-held.meta" statusf="$state/afk-held.status" - printf 'paused: awaiting the upstream tool release\n' > "$statusf" + printf 'paused: awaiting the upstream tool release; owner=tool maintainer; clears=release artifact is published\n' > "$statusf" back=$(( $(date +%s) - 500 )) if [ "$(uname)" = Darwin ]; then touch -mt "$(date -r "$back" '+%Y%m%d%H%M.%S')" "$statusf" else touch -m -d "@$back" "$statusf"; fi @@ -1921,13 +2153,38 @@ fi if [ "${FM_TEST_FOCUSED:-}" = pause-regressions ]; then test_declared_pause_preempts_permission_prompt_stale test_nonterminal_stale_not_working_surfaced - test_terminal_run_step_declared_pause_absorbed_with_markers + test_terminal_run_step_overrides_declared_pause test_surface_nonterminal_stale_clears_pause_only_after_status_resumes test_nonterminal_stale_paused_absorbed_then_resurfaced test_herdr_blocked_transition_enters_pause_absorb_path exit 0 fi +if [ "${FM_TEST_FOCUSED:-}" = owned-pause-marker ]; then + test_owned_pause_signal_registers_marker + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = owned-pause-cadence ]; then + test_owned_pause_absorbs_repeated_changed_stale + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = incomplete-pause ]; then + test_unowned_or_uncleared_pause_does_not_absorb + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = cpu-progress ]; then + test_cpu_progress_suppresses_static_pane_wedge + exit 0 +fi + +if [ "${FM_TEST_FOCUSED:-}" = cpu-stalled ]; then + test_near_zero_cpu_static_pane_still_wedge_escalates + exit 0 +fi + if [ "${FM_TEST_FOCUSED:-}" = failure-pause ]; then test_failure_pause_is_failure_classifier test_failure_pause_stale_surfaced_not_absorbed @@ -1959,6 +2216,9 @@ test_working_note_not_working_surfaced test_actionable_signal_surfaced test_harness_permission_prompts_surface_immediately test_declared_pause_preempts_permission_prompt_stale +test_owned_pause_signal_registers_marker +test_owned_pause_absorbs_repeated_changed_stale +test_unowned_or_uncleared_pause_does_not_absorb test_busy_no_progress_suspects_system_permission_dialog test_terminal_stale_surfaced test_stale_terminal_status_overridden_by_active_run @@ -1966,7 +2226,7 @@ test_nonterminal_stale_provably_working_absorbed_then_escalated test_wedge_escalation_marks_demand_deep_inspection_after_threshold test_wedge_escalation_resets_when_pane_becomes_active test_nonterminal_stale_not_working_surfaced -test_terminal_run_step_declared_pause_absorbed_with_markers +test_terminal_run_step_overrides_declared_pause test_surface_nonterminal_stale_clears_pause_only_after_status_resumes test_nonterminal_stale_paused_absorbed_then_resurfaced test_failure_pause_stale_surfaced_not_absorbed @@ -1977,6 +2237,8 @@ test_secondmate_unpause_clears_pause_tracking test_nonterminal_stale_pause_transitions_reclassify_unchanged_hash test_nonterminal_paused_rechecks_authoritative_state test_paused_authoritative_working_preserves_wedge_timer +test_cpu_progress_suppresses_static_pane_wedge +test_near_zero_cpu_static_pane_still_wedge_escalates test_nonterminal_stale_repairs_missing_or_corrupt_timer test_triage_log_size_cap_accepts_spaced_wc_counts test_heartbeat_no_change_absorbed diff --git a/tools/agent-fleet/tests/test_quota_scheduler.py b/tools/agent-fleet/tests/test_quota_scheduler.py index 51a9d85732f..ec3f5f115c1 100644 --- a/tools/agent-fleet/tests/test_quota_scheduler.py +++ b/tools/agent-fleet/tests/test_quota_scheduler.py @@ -1161,7 +1161,7 @@ def test_concurrent_reservations_are_atomic_and_balanced( ) for index in range(12) ] - results = [process.communicate(timeout=20) for process in processes] + results = [process.communicate(timeout=60) for process in processes] failures = [ {"returncode": process.returncode, "stdout": stdout, "stderr": stderr} for process, (stdout, stderr) in zip(processes, results, strict=True)