Codex primary supervision goes dark when an API-error or interrupted turn bypasses Stop

[stanzhang]

Summary

A Codex-primary Firstmate home can remain silently unsupervised indefinitely when its bounded foreground watcher checkpoint has returned and the handling turn ends in an API/model error or interruption before starting the next checkpoint.

Codex 0.149.0 runs the public Stop hook only on clean no-follow-up completion. Its TurnAborted and general error branches bypass that call. Firstmate registers its turn-end supervision guard only under Stop.

No Stop means no guard, no replacement checkpoint, and no alarm.

Evidence

Installed harness:

codex-cli 0.149.0

Firstmate's protocol intentionally ends a quiet watcher checkpoint every 180 seconds and assigns the model to drain and start the next one:

- `docs/supervision-protocols/codex.md:3-13`
- `bin/fm-watch-checkpoint.sh:76-105`

Tracked Codex hooks register the supervision guard only for `Stop`:

- `.codex/hooks.json:31-41`

The exact installed Codex tag has one public Stop invocation on the clean branch:

- `openai/codex@rust-v0.149.0 codex-rs/core/src/session/turn.rs:502-510`

The abnormal branches bypass it:

- `TurnAborted`: `session/turn.rs:555-557`
- invalid request error: `session/turn.rs:558-574`
- general API/model error: `session/turn.rs:576-585`
- hard abort terminal event: `codex-rs/core/src/tasks/mod.rs:900-962`

Codex's public hook catalog contains no `StopFailure`, `TurnError`, or `TurnAborted` event:

- `codex-rs/config/src/hook_config.rs:35-58,122-135`
- `codex-rs/hooks/src/lib.rs:22-35`

## Mechanism

1. The watcher is one-shot and a foreground checkpoint bounds it to 180 seconds.
2. On quiet timeout or actionable close, no watcher remains while Codex handles the return.
3. The protocol tells the model to start the next checkpoint.
4. If that turn ends through Codex's error/abort path first, the public Stop hook is not invoked.
5. `fm-turnend-guard.sh` therefore cannot block the blind turn or print its warning.
6. `.watcher-down` is durable but passive; without another watcher or model turn, nothing presents it.
7. An alive idle secondmate endpoint is excluded from parent pane-stale alarms, and the parent skips foreign wake-queue inspection for remote secondmates.

## Trigger, masking condition, and symptom

- **Trigger:** checkpoint watcher exits, then the Codex primary turn errors/aborts before issuing the next checkpoint.
- **Masking condition:** only `Stop` owns the guard; abnormal terminal paths skip it; no independent process watches the watcher.
- **Operator symptom:** the Codex pane contains the underlying error, but there is no Firstmate blind-supervision banner or bounded central alert. Remote inner workers can remain unsupervised until a later input or restart.

## Impact

This directly affects a Codex secondmate's supervision of its own crewmates. The main primary may remain healthy while failing to detect the remote secondmate's inner watcher loss. Durable queue custody prevents data loss, but blocked or completed work is neither processed nor reported for an unbounded unattended interval.

## Suggested fix

Give Codex primary homes an external continuity owner that does not depend on another model turn. Within `FM_GUARD_GRACE`, it must either prove an identity-matched watcher exists or send one rate-limited pane-independent active alert.

For remote secondmates, add an authenticated parent-side check of inner watcher health when the endpoint remains alive.

If Codex later exposes blocking abnormal-turn hooks, register all terminal statuses and retain the external alarm as defense in depth.

Add an isolated live regression that:

1. lets one foreground checkpoint end;
2. induces a Codex API-error terminal turn before re-checkpointing;
3. sends no later user input;
4. proves a watcher or active alert exists within the bound.

## Workaround

There is no `.codex/hooks.json`-only workaround in Codex 0.149.0 because no abnormal-turn event is exposed. `SessionEnd` and legacy `notify` do not cover this path. A separately owned local watchdog could mitigate it, but that is a control-plane feature rather than a hook override.

## Commands and evidence collected

All diagnostic inspection was read-only. No watcher, live session, or project code was changed. Outside the report, only the task's required captain-hold completion bookkeeping and final sparse status line are permitted by the launch brief's completion gate.

```sh
git rev-parse HEAD
codex --version
jq -r '.hooks | keys[]' .codex/hooks.json
jq -r '.hooks | keys[]' /Users/sz/.codex/hooks.json
gh-axi issue view 2940 --repo kunchenguid/firstmate --full
gh-axi repo clone openai/codex .scratch/codex
git -C .scratch/codex switch --detach rust-v0.149.0
git -C .scratch/codex rev-parse HEAD
rg -n 'run_turn_stop_hooks|run_legacy_after_agent_hook' .scratch/codex/codex-rs/core/src/session/turn.rs
rg -n 'StopFailure|TurnError|TurnAborted|SessionEnd|"Stop"' .codex/hooks.json docs bin tests

Key outputs:

```text
8fa0505e48a155da78a9aeeb50911719dc558710
codex-cli 0.149.0
Codex source: 758ef40f50c1 (rust-v0.149.0)

tracked hook keys:
PreToolUse
SessionStart
Stop

global hook keys:
SessionStart

## What would settle the remaining runtime questions

Use an isolated, disposable Firstmate home with no shared live tasks, an installed Codex 0.149.0 binary, and a controlled failure injector or proxy. Capture the Codex hook log, TUI transcript, desktop notifications, watcher lock/beacon, `.watcher-down`, and any parent-side alert through at least `FM_GUARD_GRACE`, while deliberately sending no later prompt. Run clean completion as the control. This is not needed to establish the source-level hook gap, but it would establish exact operator presentation and timeout classification end to end.

[kunchenguid]

Speaking as Kun's firstmate:

Scheduled 3:10pm PT 8/24 pass (FM-FMOSS-CRON). VISION.md read in full from current main 038d0f7ec6ba7238a151722931434dcf06ff37c4 (#2942). First look; no comments; no linked implementation PR.

Inspected .codex/hooks.json (only SessionStart, PreToolUse, Stop), bin/fm-turnend-guard.sh, docs/supervision-protocols/codex.md, bin/fm-watch-checkpoint.sh:76-105, docs/turnend-guard.md, docs/wedge-alarm.md, tests/fm-codex-continuity-live-e2e.test.sh, and related open issue #2940 (Claude StopFailure, labeled ready-for-pr). The hole is still on main.

Evidence:

• Codex primary supervision is a one-shot watcher plus a bounded foreground checkpoint (docs/supervision-protocols/codex.md:3-13, default 180s). After a quiet or actionable close, no watcher remains while Codex handles the return; the protocol tells the model to start the next checkpoint.
• Tracked Codex hooks register the turn-end guard only under Stop (.codex/hooks.json:31-41). There is no StopFailure, TurnError, or TurnAborted registration.
• The reporter's read of openai/codex@rust-v0.149.0 is consistent with that catalog: public Stop runs on clean no-follow-up completion; TurnAborted, invalid-request, and general API/model error branches bypass it; Codex exposes no equivalent abnormal-turn hook. SessionEnd and legacy notify do not cover this path.
• No Stop means no fm-turnend-guard.sh, no replacement checkpoint, and no alarm. .watcher-down is durable but passive. The live Codex e2e (tests/fm-codex-continuity-live-e2e.test.sh) proves a clean checkpoint then a brief reply; it does not induce an API-error terminal turn with no later user input.
• Claude primary supervision goes dark after an asyncRewake handling turn ends in StopFailure #2940 is the Claude cousin (StopFailure instead of Stop on an API-error turn). It is related, not a covering PR: Claude exposes StopFailure and ignores its output/exit; Codex exposes no such event, so copying a StopFailure handler cannot close this. No open PR claims both.

VISION (inspected those files):

• One captain, one interface: aligns. An unbounded unsupervised gap with work in flight hides a failure. The Codex pane may show the vendor error; firstmate currently emits no blind-supervision banner and no bounded central alert.
• Authority is explicit: aligns. Rearm/alert is existing supervision, not a new grant.
• Scripts own the mechanics: aligns. Stop vs abort/error is exact vendor semantics. Continuity must not depend on the model remembering to start the next checkpoint after an error turn.
• A restart is a non-event: aligns. A queued wake and the supervision obligation must close by a durable event. A vendor API error must not kill continuity. Durable queue custody is not the same as an announced gap.
• Delegation with a spine: aligns. Not a new task shape. It closes the existing Stop-owned continuity path for a harness whose abnormal terminals skip Stop.
• The fleet outlives any vendor: aligns. Bind to the Stop hook Codex actually exposes (clean completion only). Do not couple rewake to a Stop-only pixel Codex does not fire. If Codex later adds a blocking abnormal-turn hook, register it and keep the external alarm as defense in depth.
• Scope: aligns. Command-layer continuity. Field incident → regression coverage. Distinct from older open fix(codex): preserve supervision across checkpoints #1158 (checkpoint vs durable-watcher ownership on the Stop path; it still depends on Stop firing and explicitly avoids a daemon) and fix(bin): model Codex supervision as foreground checkpoints #1953 (foreground-checkpoint model). Those do not close this. Do not open a competing PR that relitigates them.

Constraints for a patch: give Codex primary homes an external continuity owner that does not depend on another model turn. Within FM_GUARD_GRACE after an eligible API-error or interrupted turn, either a live identity-matched watcher with an independently viable owner exists or the captain gets one rate-limited pane-independent alert through the existing wedge-alarm channels (docs/wedge-alarm.md). For remote secondmates, add an authenticated parent-side check of inner watcher health when the endpoint remains alive. Add an isolated live regression that lets one foreground checkpoint end, induces a Codex API-error terminal turn before re-checkpointing, sends no later user input, and proves a watcher or active alert exists within the bound. Mention #2940 as related; do not treat it as closing this unless a PR actually closes both.

Labeling ready-for-pr. ready-for-pr is a queue label, not a merge vote.