Skip to content

feat(bin): session-independent fleet liveness guardian - #2427

Open
Nikita-Guzenko wants to merge 1 commit into
kunchenguid:mainfrom
Nikita-Guzenko:fm/fleet-liveness-guardian
Open

feat(bin): session-independent fleet liveness guardian#2427
Nikita-Guzenko wants to merge 1 commit into
kunchenguid:mainfrom
Nikita-Guzenko:fm/fleet-liveness-guardian

Conversation

@Nikita-Guzenko

Copy link
Copy Markdown

What

Adds bin/fm-liveness-guardian.sh, a session-independent fleet supervision backstop that runs from a systemd user timer, below every session, and keeps each firstmate home's watcher alive so tasks keep being handled even when no interactive session is in the loop.

The problem it closes

Each home's watcher (bin/fm-watch.sh) is kept armed by a harness-owned continuity mechanism that depends on a live session in that home (Claude's Stop auto-arm, Cursor's stop park, the Pi/OpenCode extensions, a persistent tracked arm). When a director's session stops taking turns while its armed watcher dies, the harness has no Stop to re-arm on: the beacon (state/.last-watcher-beat) goes stale far past FM_GUARD_GRACE, and incoming wakes (new Telegram-routed tasks, crew status) are never surfaced until a human notices. Observed live on 2026-08-15 with several homes' beacons stale for thousands of seconds while their panes stayed open. Nothing below the session repaired it; this guardian does.

Behavior per home

Guarded homes: the main home plus every local secondmate, discovered from the main home's kind=secondmate meta records (recorded home=), with data/secondmates.md as the fallback. Remote secondmates are reported and skipped. It classifies with the fleet's own model-aware predicates and acts only on a genuine lapse:

Home state Class Action
No in-flight work / relay poll / event source idle none
Needs supervision, beacon fresh for its model healthy none (never double-arms)
Needs supervision, beacon stale, session-less dead (secondmate) / dead (main) secondmate: relaunch via sanctioned fm-spawn --secondmate (rate-limited); main: re-arm to capture wakes durably + escalate a human to start a session
Needs supervision, beacon stale, session alive lapsed never re-arm and never relaunch a live session; start a confirmation clock and escalate a likely wedge only after it stays lapsed past FM_GUARDIAN_WEDGE_ESCALATE_SECS
Remote secondmate (another host) remote-skip none

Design rationale — the self-surviving arm, and what it deliberately does not do

  • Arming survives the call. Re-arm launches a real watcher through a self-surviving systemd-run --user transient unit — never a fire-and-forget shell &, which the bin/fm-watch-arm.sh header documents as the exact mistake that silently killed supervision for ~30 minutes. The timer service is a short oneshot; the transient watcher unit the user manager owns independently, so it outlives the pass.
  • No fake beacon, no double-arm. The guardian never touches state/.last-watcher-beat; a fresh beacon is produced only by a real watcher. A healthy home is left completely alone, and the watcher singleton lock plus fm-watch-arm.sh's own attach converge any concurrent arm on one watcher.
  • Handling-honest, not just beacon-honest. Re-arming restores the beacon and captures wakes durably, but a surfaced wake is handled by the home's own live session. So a dead local director is relaunched (restoring a handler), and a home that cannot be safely restored is escalated rather than masked behind a fresh beacon. Critically, a live-but-lapsed session is never re-armed: a guardian-owned watcher would freshen that beacon and mask a wedged session (and defeat the confirmation clock), and re-arm gives an asleep session no handler anyway. Wakes are not lost meanwhile — their signal/status files persist and a watcher scans them once the session resumes or is relaunched.
  • Not unattended daemon mode. Making wakes self-handle with zero session is the away-mode sub-supervisor daemon (AGENTS.md section 8), which is a fleet-behavior change (batched escalations, changed token tradeoff, pane injection). This guardian deliberately leaves that choice to the captain rather than imposing it, and escalates a genuine wedge instead of silently adopting it.
  • Safe from a bare timer env. Exports XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS before any systemctl --user call, uses absolute paths, sets FM_HOME per home, never touches the shared no-mistakes daemon, and rate-limits relaunches/escalations so a crash-looping home is escalated instead of hammered. Logs one machine-scannable line per real event to ~/.local/state/fm-liveness-guardian.log (silent when healthy).

What is tested

Colocated tests/fm-liveness-guardian.test.sh, 9 cases via bin/fm-test-run.sh, run under env -i against disposable mock homes only — never the live aln/voice/main homes:

  • healthy and idle homes are no-ops (no arm, no beacon re-touch);
  • adversarial: a session-less home with a beacon stale past grace is re-armed to a fresh beacon with no session in the loop, and a second pass sees it healthy and does not double-arm;
  • a live-but-lapsed session is never re-armed and is escalated only after it persists past the wedge threshold (no false wedge on a long turn);
  • a dead local secondmate is relaunched once through the sanctioned seam, then rate-limited;
  • a remote secondmate is reported and skipped;
  • the default arm transport is a self-surviving systemd-run --user, never a shell & (asserted through --list);
  • it runs cleanly under a bare env -i environment;
  • a dead main home is re-armed and escalated, never relaunched (the guardian never spawns a competing firstmate).

bin/fm-lint.sh (shellcheck 0.11.0) and bin/fm-doc-audience-check.sh both clean.

Note: driven and validated outside the no-mistakes pipeline (the gate is structurally unavailable from the linked worktree — the repo is registered under the primary checkout), so this ships direct-PR with CI + review as the gate.

Install — deferred to firstmate, after merge

The guardian is inert until installed as a systemd user timer. Templates live in dist/systemd/; the full install and live-verify steps are in docs/liveness-guardian.md. Firstmate installs and enables it on the live box after merge; it is deliberately not enabled from this branch and this PR does not touch any live home. loginctl enable-linger is required so user units run with no active login session — exactly when the guardian matters.

Add bin/fm-liveness-guardian.sh, a systemd-user-timer supervision backstop
that runs below every session and keeps each firstmate home's watcher alive so
tasks keep being handled when a director's session stops re-arming.

Per guarded home (main + local secondmates, discovered from the main home's
kind=secondmate meta records) it classifies supervision with the fleet's own
model-aware predicates and acts only on a genuine lapse:

- healthy/idle: no-op (never double-arms).
- session-less lapsed home: re-arm a real watcher via a self-surviving
  systemd-run --user transient unit (never a fire-and-forget shell &), which
  restores the beacon and captures wakes durably.
- dead local secondmate: relaunch the director through the sanctioned
  fm-spawn --secondmate path, rate-limited, escalate on a crash loop.
- live-but-lapsed session: never re-arm (that would mask a wedge and defeat the
  confirmation clock) and never relaunch a live session; escalate a likely wedge
  only after it stays lapsed past a threshold longer than any legitimate turn.
- remote secondmate: reported and skipped.

It exports XDG_RUNTIME_DIR/DBUS_SESSION_BUS_ADDRESS before any systemctl --user
call, uses absolute paths, runs cleanly under env -i, never touches the shared
no-mistakes daemon, and logs one machine-scannable line per real event (silent
when healthy). It deliberately does not put directors into unattended daemon
mode, which is a fleet-behavior change left to the captain.

Ships systemd .service/.timer templates under dist/systemd/, a colocated test
exercising every classification under env -i against disposable mock homes only,
and docs/liveness-guardian.md plus scripts.md and watcher-continuity.md pointers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant