Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .agents/skills/stuck-crewmate-recovery/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,11 @@ If the worktree or ownership cannot be reconciled safely, leave all state intact

Escalate in order:

1. Peek the pane.
1. Peek the pane, and read it as an ACTIVITY question, not a layout question.
A pane's last line is the composer prompt in every harness that has one, and it looks identical whether the worker is idle or thirteen minutes into a turn.
The activity indicator sits ABOVE that prompt, so a tail that cuts the pane off at its bottom line has cut off the only part that carries the answer.
Capture enough lines to see the region above the composer before concluding anything, and treat "the bottom line is a prompt" as no evidence at all.
A pane that still looks idle is a reason to ask `bin/fm-crew-state.sh <id>`, which reads the run-step and busy-signature sources rather than the pane's appearance; a worker running a suite or a long tool call is working, not wedged.
2. If the crewmate is waiting on a question its brief already answers, answer in one line via `FM_HOME=<this-firstmate-home> bin/fm-send.sh` from an active firstmate session unless `FM_HOME` is already set to the active firstmate home.
3. If the crewmate is confused or looping, interrupt with `FM_HOME=<this-firstmate-home> bin/fm-control.sh <task-id> interrupt`, then redirect with one corrective line through `fm-send`.
4. If the crewmate is genuinely wedged after redirection, relaunch it with `FM_HOME=<this-firstmate-home> bin/fm-control.sh <task-id> relaunch --note '<progress so far>'`, which stops the agent, carries the brief plus that note into a replacement in the same local copy, and restores the prior record if the replacement cannot start.
Expand Down
7 changes: 5 additions & 2 deletions bin/fm-claude-stop-autoarm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,10 @@ fm_primary_scope_matches "$FM_ROOT" "$STATE" || exit 0

# --- identity: only the lock-owning session's hooks may arm ------------------
# A prior session may have died after leaving its numeric harness pid in .lock.
# Use the shared liveness predicate to recognize only that stale-owner case.
# Recover ONLY from that stale-owner case, which the shared classifier is the
# single owner of. A holder that is running but whose name no pattern
# recognizes reports `unidentified`, not `stale`, and this hook stands down: an
# unrecognized command name is not evidence that a session ended.
# Defer the mutating claim until after the unchanged AFK and need gates, so an
# idle or away home remains byte-for-byte inert. Missing or malformed locks are
# uncertainty rather than stale-owner evidence and remain inert.
Expand All @@ -109,7 +112,7 @@ if ! fm_session_lock_owned_by_self "$STATE"; then
case "$LOCK_PID" in
''|*[!0-9]*) exit 0 ;;
esac
fm_harness_pid_alive "$LOCK_PID" && exit 0
fm_session_lock_state_permits_claim "$(fm_session_lock_holder_state "$STATE/.lock")" || exit 0
RECOVER_SESSION_LOCK=1
fi

Expand Down
8 changes: 8 additions & 0 deletions bin/fm-lint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,14 @@ while [ "$worker" -lt "$SHARD_COUNT" ]; do
worker=$((worker + 1))
done

# Concurrency telemetry ONLY - never a liveness decision.
#
# This is a name-pattern probe, so its count is a LOWER BOUND, not a census: it
# sees shellcheck processes whose kernel-side name is exactly `shellcheck`, and
# misses any that run behind a wrapper or shim under a different name. A zero
# here therefore means "none matched the name", not "none running". Nothing
# branches on this value - it is reported and discarded - which is the only
# reason a pattern probe is acceptable at this call site.
fm_lint_shellcheck_count() {
if command -v pgrep >/dev/null 2>&1; then
pgrep -x shellcheck 2>/dev/null | wc -l | tr -d '[:space:]'
Expand Down
36 changes: 28 additions & 8 deletions bin/fm-lock.sh
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,13 @@ if [ "${1:-}" = "status" ]; then
echo "lock: unreadable"
exit 0
}
if fm_harness_pid_alive "$old"; then echo "lock: held by live harness pid $old"; else echo "lock: stale (pid $old dead or not a harness)"; fi
case "$(fm_session_lock_holder_state "$LOCK")" in
live) echo "lock: held by live harness pid $old" ;;
unidentified) echo "lock: pid $old is RUNNING but not identifiable as a harness; treated as held, not stale" ;;
malformed) echo "lock: malformed" ;;
free) echo "lock: free" ;;
*) echo "lock: stale (pid $old is gone)" ;;
esac
exit 0
fi

Expand Down Expand Up @@ -61,10 +67,16 @@ if [ -f "$LOCK" ] && [ ! -L "$LOCK" ]; then
echo "lock acquired: harness pid $me"
exit 0
fi
if fm_harness_pid_alive "$old"; then
echo "error: another live firstmate session holds the lock (pid $old); operate read-only until resolved" >&2
exit 1
fi
case "$(fm_session_lock_holder_state "$LOCK")" in
live)
echo "error: another live firstmate session holds the lock (pid $old); operate read-only until resolved" >&2
exit 1
;;
unidentified)
echo "error: the session lock names pid $old, which is RUNNING but is not identifiable as a verified harness. It is not treated as stale, because an unrecognized name is not evidence that a session is gone - a wrapped or renamed session reports exactly this. Operate read-only; if that process is genuinely unrelated, stop it or clear $LOCK by hand." >&2
exit 1
;;
esac
fi

if ! fm_lock_try_acquire "$CLAIM_LOCK"; then
Expand All @@ -86,9 +98,17 @@ if [ -e "$LOCK" ] || [ -L "$LOCK" ]; then
echo "error: session lock is unreadable; operate read-only until resolved" >&2
exit 1
}
if [ "$old" != "$me" ] && fm_harness_pid_alive "$old"; then
echo "error: another live firstmate session holds the lock (pid $old); operate read-only until resolved" >&2
exit 1
if [ "$old" != "$me" ]; then
case "$(fm_session_lock_holder_state "$LOCK")" in
live)
echo "error: another live firstmate session holds the lock (pid $old); operate read-only until resolved" >&2
exit 1
;;
unidentified)
echo "error: the session lock names pid $old, which is RUNNING but is not identifiable as a verified harness. It is not treated as stale, because an unrecognized name is not evidence that a session is gone - a wrapped or renamed session reports exactly this. Operate read-only; if that process is genuinely unrelated, stop it or clear $LOCK by hand." >&2
exit 1
;;
esac
fi
fi
if ! { printf '%s\n' "$me" > "$LOCK"; } 2>/dev/null; then
Expand Down
146 changes: 146 additions & 0 deletions bin/fm-session-lock-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -174,3 +174,149 @@ $pids
EOF
return 1
}

# --- lock-holder classification ---------------------------------------------
#
# fm_harness_pid_alive above answers "is this pid a process I can RECOGNIZE as a
# harness?". That is a name-pattern question, and a name pattern can only ever
# produce evidence FOR a match. Its no-match result is ambiguous: the pid may be
# dead, or it may be a perfectly live session this table cannot name - launched
# through a generated wrapper script, renamed by its installer, or running a
# harness that has not been added to FM_HARNESS_RE yet. Treating that ambiguity
# as "dead" is what lets one session reclaim a live session's lock and put two
# writers on the same home.
#
# The reclaim decision therefore does not use the name pattern as its authority.
# It asks four questions in order of strength - is the pid gone, is it a
# recognized harness, is it too young to have written this lock, is it our own
# lineage - and only the strongest available answer decides. When none of them
# settles the question, the holder is left alone.

# Elapsed seconds for <pid>, or return 1 when it cannot be read.
#
# ps etime is used rather than lstart because its [[dd-]hh:]mm:ss form is
# locale-invariant on both macOS and procps, while lstart's date string is not:
# an identity written under one locale and re-read under another (ko_KR, say)
# mismatches and would reject a live holder. Elapsed time answers the only
# question asked of it below - "did this process exist before that file was
# written?" - without any date parsing at all.
fm_process_elapsed_seconds() { # <pid>
local pid=$1 raw days=0 hours=0 mins=0 secs=0 rest
case "$pid" in ''|*[!0-9]*) return 1 ;; esac
raw=$(LC_ALL=C ps -p "$pid" -o etime= 2>/dev/null) || return 1
raw=${raw//[[:space:]]/}
[ -n "$raw" ] || return 1
case "$raw" in
*-*) days=${raw%%-*}; rest=${raw#*-} ;;
*) rest=$raw ;;
esac
case "$rest" in
*:*:*) hours=${rest%%:*}; rest=${rest#*:}; mins=${rest%%:*}; secs=${rest##*:} ;;
*:*) mins=${rest%%:*}; secs=${rest##*:} ;;
*) return 1 ;;
esac
# Strip leading zeros so 08 is not read as an invalid octal literal.
days=$((10#${days:-0})); hours=$((10#${hours:-0}))
mins=$((10#${mins:-0})); secs=$((10#${secs:-0}))
printf '%s\n' "$(( days * 86400 + hours * 3600 + mins * 60 + secs ))"
}

# True when <pid> is too YOUNG to have written the file <path>: it started after
# that file was last written, so whatever wrote the file was some other process
# and this pid is a recycled number.
#
# This is the signal that keeps an unrecognized-but-live holder from wedging the
# home forever. A recycled pid is necessarily younger than the lock, because the
# process that wrote the lock had to die before its number could be reissued.
# Unreadable inputs return 1, which keeps the holder protected rather than
# reclaimed.
fm_pid_started_after_file() { # <pid> <path>
local pid=$1 path=$2 elapsed now mtime lock_age
elapsed=$(fm_process_elapsed_seconds "$pid") || return 1
if [ "$(uname)" = Darwin ]; then
mtime=$(stat -f %m "$path" 2>/dev/null) || return 1
else
mtime=$(stat -c %Y "$path" 2>/dev/null) || return 1
fi
case "$mtime" in ''|*[!0-9]*) return 1 ;; esac
now=$(date +%s 2>/dev/null) || return 1
lock_age=$(( now - mtime ))
# Both clocks are whole seconds, so require a clear margin rather than a bare
# inequality: a process that started in the same second the lock was written
# stays protected.
[ "$lock_age" -gt "$(( elapsed + 1 ))" ]
}

# True when <pid> appears anywhere in THIS process's parent chain.
#
# Unlike fm_harness_ancestry_pids this walk does not stop at the first
# non-harness hop, because the question is plain lineage, not harness identity:
# a lock naming an inner shell of our own session must be recognized as ours.
# A competing session can never satisfy this - if it were our ancestor it would
# be the session that launched us.
fm_pid_is_own_ancestor() { # <pid>
local target=$1 walk=$$ hops=0
case "$target" in ''|*[!0-9]*) return 1 ;; esac
while [ "$hops" -lt 32 ]; do
[ "$walk" != "$target" ] || return 0
walk=$(ps -o ppid= -p "$walk" 2>/dev/null | tr -d '[:space:]')
case "$walk" in ''|*[!0-9]*) return 1 ;; esac
[ "$walk" -gt 1 ] || return 1
hops=$((hops + 1))
done
return 1
}

# Classify the holder recorded in lock file <path>. Prints exactly one of:
#
# free no lock file, or it records no pid at all
# malformed the lock file exists but does not record a plain pid
# live the pid is alive AND identifiable as a verified harness
# unidentified the pid is ALIVE but no name pattern recognizes it, and it
# belongs to no session we can account for; it may be a live
# session this table cannot name, so it is NOT reclaimable
# self-inner the pid is ALIVE and unrecognized, but it is THIS process's own
# ancestor - an inner shell of our own session that wrote the
# lock under its own pid. Reclaimable: re-pointing the lock at the
# real harness pid is a correction, not a takeover.
# stale the pid is gone, or it is a recycled number too young to have
# written this lock; safe to reclaim
#
# `unidentified` is the whole point of this function: it is the case the old
# single boolean folded into "dead", and it is reported separately so callers
# refuse and say why, instead of silently taking a live session's home.
fm_session_lock_holder_state() { # <lock-path>
local lock=$1 pid
[ -f "$lock" ] || { printf 'free\n'; return 0; }
pid=$(cat "$lock" 2>/dev/null) || { printf 'malformed\n'; return 0; }
pid=${pid//[[:space:]]/}
[ -n "$pid" ] || { printf 'free\n'; return 0; }
case "$pid" in *[!0-9]*) printf 'malformed\n'; return 0 ;; esac
if ! kill -0 "$pid" 2>/dev/null; then
printf 'stale\n'
return 0
fi
if fm_harness_pid_alive "$pid"; then
printf 'live\n'
return 0
fi
# Alive, but unrecognized. Only strictly stronger evidence than the name may
# downgrade this to something reclaimable.
if fm_pid_started_after_file "$pid" "$lock"; then
printf 'stale\n'
elif fm_pid_is_own_ancestor "$pid"; then
printf 'self-inner\n'
else
printf 'unidentified\n'
fi
}

# True when the classifier's verdict permits this session to claim the lock.
# Reclaiming our own session's inner pid is a correction; taking a lock from a
# process we cannot account for is not.
fm_session_lock_state_permits_claim() { # <state>
case "$1" in
stale|self-inner) return 0 ;;
*) return 1 ;;
esac
}
3 changes: 2 additions & 1 deletion bin/fm-test-run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,8 @@ family_for_basename() {
printf '%s\n' pure-contract-unit
;;
fm-daemon.test.sh|fm-guard-stale-banner.test.sh|fm-pi-watch-extension.test.sh|\
fm-session-lock-ancestry.test.sh|fm-cursor-primary.test.sh|\
fm-session-lock-ancestry.test.sh|fm-session-lock-holder-state.test.sh|\
fm-cursor-primary.test.sh|\
fm-supervision-events.test.sh|fm-turnend-guard.test.sh|fm-wake-daemon-lifecycle-e2e.test.sh|\
fm-wake-drain-unread-status.test.sh|\
fm-wake-queue.test.sh|fm-watch-arm.test.sh|fm-watch-checkpoint.test.sh|fm-watch-triage.test.sh|\
Expand Down
5 changes: 4 additions & 1 deletion bin/fm-turnend-guard-cursor.sh
Original file line number Diff line number Diff line change
Expand Up @@ -238,10 +238,13 @@ current_session_still_ours() {
# Only the lock-owning session may arm or wake. A prior session that died
# leaving its numeric harness pid behind is the one recoverable
# case, delegated to bin/fm-lock.sh so acquisition keeps its single owner.
# A holder that is running but unrecognized reports `unidentified` rather than
# `stale`, and this park stands down instead of claiming: a name the table
# cannot match is not evidence that the session ended.
if ! fm_session_lock_owned_by_self "$STATE"; then
LOCK_PID=$(cat "$STATE/.lock" 2>/dev/null || true)
case "$LOCK_PID" in ''|*[!0-9]*) exit 0 ;; esac
fm_harness_pid_alive "$LOCK_PID" && exit 0
fm_session_lock_state_permits_claim "$(fm_session_lock_holder_state "$STATE/.lock")" || exit 0
"$SCRIPT_DIR/fm-lock.sh" >/dev/null 2>&1 || exit 0
fm_session_lock_owned_by_self "$STATE" || exit 0
fi
Expand Down
3 changes: 2 additions & 1 deletion docs/watcher-continuity.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ Pi same-process session replacement follows the generation-owner contract in `.p
Cursor's `.cursor/hooks.json` `stop` hook (`bin/fm-turnend-guard-cursor.sh`) owns routine tokenless re-arm for a Cursor primary by parking that awaited hook on `bin/fm-watch-arm.sh` and returning an actionable close as one follow-up; [`turnend-guard.md`](turnend-guard.md#harness-integrations) owns its loop bounds and supersession baton.
Claude's `.claude/settings.json` Stop `asyncRewake` hook (`bin/fm-claude-stop-autoarm.sh`) owns routine tokenless re-arm.
The hook fires on every Stop, and an eligible primary with supervision need admits one home-scoped owner that foregrounds `bin/fm-watch-arm.sh` inside the hook-owned process tree.
A numeric session-lock owner that fails the shared `fm_harness_pid_alive` predicate is reclaimed through `bin/fm-lock.sh` before auto-arm state changes, while a live owner, absent lock, or malformed lock keeps the competing hook inert.
A numeric session-lock owner is reclaimed through `bin/fm-lock.sh` before auto-arm state changes only when the shared `fm_session_lock_holder_state` classifier permits a claim, while any other verdict keeps the competing hook inert; `bin/fm-session-lock-lib.sh` owns that classification in full.
The reinforcement worth repeating here is the one this hook can get wrong silently: a holder whose command name matches nothing is not thereby a holder that is gone, so an unrecognized name never licenses the claim.
The stale-owner claim occurs only after the existing AFK and supervision-need gates pass.
After each non-actionable arm close, the hook rechecks the identity-matched watcher lock and fresh beacon before retrying a bounded number of times.
A cycle-end failure is benign when that live-watcher predicate is true, and the hook suppresses the arm output and continues silently.
Expand Down
Loading
Loading