feat(bin): session-independent fleet liveness guardian - #2427
Open
Nikita-Guzenko wants to merge 1 commit into
Open
feat(bin): session-independent fleet liveness guardian#2427Nikita-Guzenko wants to merge 1 commit into
Nikita-Guzenko wants to merge 1 commit into
Conversation
Add bin/fm-liveness-guardian.sh, a systemd-user-timer supervision backstop that runs below every session and keeps each firstmate home's watcher alive so tasks keep being handled when a director's session stops re-arming. Per guarded home (main + local secondmates, discovered from the main home's kind=secondmate meta records) it classifies supervision with the fleet's own model-aware predicates and acts only on a genuine lapse: - healthy/idle: no-op (never double-arms). - session-less lapsed home: re-arm a real watcher via a self-surviving systemd-run --user transient unit (never a fire-and-forget shell &), which restores the beacon and captures wakes durably. - dead local secondmate: relaunch the director through the sanctioned fm-spawn --secondmate path, rate-limited, escalate on a crash loop. - live-but-lapsed session: never re-arm (that would mask a wedge and defeat the confirmation clock) and never relaunch a live session; escalate a likely wedge only after it stays lapsed past a threshold longer than any legitimate turn. - remote secondmate: reported and skipped. It exports XDG_RUNTIME_DIR/DBUS_SESSION_BUS_ADDRESS before any systemctl --user call, uses absolute paths, runs cleanly under env -i, never touches the shared no-mistakes daemon, and logs one machine-scannable line per real event (silent when healthy). It deliberately does not put directors into unattended daemon mode, which is a fleet-behavior change left to the captain. Ships systemd .service/.timer templates under dist/systemd/, a colocated test exercising every classification under env -i against disposable mock homes only, and docs/liveness-guardian.md plus scripts.md and watcher-continuity.md pointers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
bin/fm-liveness-guardian.sh, a session-independent fleet supervision backstop that runs from a systemd user timer, below every session, and keeps each firstmate home's watcher alive so tasks keep being handled even when no interactive session is in the loop.The problem it closes
Each home's watcher (
bin/fm-watch.sh) is kept armed by a harness-owned continuity mechanism that depends on a live session in that home (Claude's Stop auto-arm, Cursor's stop park, the Pi/OpenCode extensions, a persistent tracked arm). When a director's session stops taking turns while its armed watcher dies, the harness has no Stop to re-arm on: the beacon (state/.last-watcher-beat) goes stale far pastFM_GUARD_GRACE, and incoming wakes (new Telegram-routed tasks, crew status) are never surfaced until a human notices. Observed live on 2026-08-15 with several homes' beacons stale for thousands of seconds while their panes stayed open. Nothing below the session repaired it; this guardian does.Behavior per home
Guarded homes: the main home plus every local secondmate, discovered from the main home's
kind=secondmatemeta records (recordedhome=), withdata/secondmates.mdas the fallback. Remote secondmates are reported and skipped. It classifies with the fleet's own model-aware predicates and acts only on a genuine lapse:idlehealthydead(secondmate) /dead(main)fm-spawn --secondmate(rate-limited); main: re-arm to capture wakes durably + escalate a human to start a sessionlapsedFM_GUARDIAN_WEDGE_ESCALATE_SECSremote-skipDesign rationale — the self-surviving arm, and what it deliberately does not do
systemd-run --usertransient unit — never a fire-and-forget shell&, which thebin/fm-watch-arm.shheader documents as the exact mistake that silently killed supervision for ~30 minutes. The timer service is a short oneshot; the transient watcher unit the user manager owns independently, so it outlives the pass.state/.last-watcher-beat; a fresh beacon is produced only by a real watcher. A healthy home is left completely alone, and the watcher singleton lock plusfm-watch-arm.sh's own attach converge any concurrent arm on one watcher.XDG_RUNTIME_DIRandDBUS_SESSION_BUS_ADDRESSbefore anysystemctl --usercall, uses absolute paths, setsFM_HOMEper home, never touches the shared no-mistakes daemon, and rate-limits relaunches/escalations so a crash-looping home is escalated instead of hammered. Logs one machine-scannable line per real event to~/.local/state/fm-liveness-guardian.log(silent when healthy).What is tested
Colocated
tests/fm-liveness-guardian.test.sh, 9 cases viabin/fm-test-run.sh, run underenv -iagainst disposable mock homes only — never the livealn/voice/main homes:systemd-run --user, never a shell&(asserted through--list);env -ienvironment;bin/fm-lint.sh(shellcheck 0.11.0) andbin/fm-doc-audience-check.shboth clean.Install — deferred to firstmate, after merge
The guardian is inert until installed as a systemd user timer. Templates live in
dist/systemd/; the full install and live-verify steps are indocs/liveness-guardian.md. Firstmate installs and enables it on the live box after merge; it is deliberately not enabled from this branch and this PR does not touch any live home.loginctl enable-lingeris required so user units run with no active login session — exactly when the guardian matters.