feat(worker): record verified compartment chain tips from the monitor

[ruby-dlee]

The gap

PR #271 moved the compartment landing anchor to controller-owned state: compartment-chain-tip records the verified outbox chain tip on the worker record under the controller lock, and fm-worker-authority.py reads the tip ONLY from there, refusing when it is absent with "exits through surrender until its monitor records tips with compartment-chain-tip".

Nothing called it. On main, grep -c compartment-chain-tip bin/fm-secondmate-cloud-monitor.sh bin/fm-secondmate-cloud-monitor.py returns 0 and 0, so every secondmate compartment could only take the sanctioned-but-degraded surrender exit. This wires the missing call.

What this builds

bin/fm-secondmate-cloud-monitor.py process-mailbox now attests the tip whenever local verification ADVANCES it, and only then.

• After the proof, never before. The call sits immediately after verify_mailbox and after both stateful rewind checks, and before any delivery. The command attests without verifying (it never reads the mailbox), so the monitor must never report a tip it has not itself proved.
• Idempotent and cheap. The recorded (sequence, chain_digest) pair is durable in the monitor's own state (recorded_chain_tip), and an unchanged tip is skipped.
• Its own verb, not the message lane. Invoked directly with its own argv, not routed through message-put/message-collect, whose claim-exempt carve and PR feat(worker): claim-exempt secondmate message lane (R2/R3 PR 3) #266 static pin assert the message ops write no lifecycle state. A unit pins that no message-* invocation ever carries --chain-digest.
• The bash monitor sources the persisted compartment environment around the helper (the same idiom the collect, relay, and child-relay passes use) because every lifecycle command builds its controller environment from the allowlisted FM_AZURE_* identity. No provider call is made: this command only takes the controller lock and writes one field. Timeout is a dedicated 300s rather than the 900s blob-transfer budget.

Refusal semantics, and why

1. Monotonicity refusal (rewind, or same sequence with a different digest) FREEZES the lane through the same sticky .chain-break marker, exit 3, delivering nothing. The record is monotone by construction, so this can only mean the controller's attested chain and the chain this monitor just proved are not the same chain. That is the same harm class the marker already exists for, and no later pass can heal it: the monitor's own stateful tip check catches this when its state file survives, and this is the case where the state file did NOT survive and the controller record is the only remaining memory.

2. "released work cannot record a compartment chain tip" is end-of-life, but it is never taken at face value. command_compartment_chain_tip checks release_proof BEFORE its monotonicity block (bin/fm-worker-lifecycle.py, the release_proof is not None guard immediately preceding the verified_chain_tip comparison; lines 4072 and 4074-4085 as of the rebase onto e2683acc, deliberately named by symbol because feat(worker): authorized task home for compartment child requests #278 shifted them ~211 lines and this PR's own code comments strip line numbers for that reason), so a RELEASED worker answers a genuine rewind or fork with the benign string. Classifying on the string alone would let the one refusal class that must freeze arrive dressed as the one that must not, and the compartment would keep relaying a chain the controller's own record contradicts. The monitor therefore reads the held tip back out of the controller document and judges it: a contradiction FREEZES, an unreadable document falls to the retry class (closing on faith would silently downgrade the compartment to surrender for good), and only a non-contradicting held tip closes the lane durably and quietly.

   That read-back is deliberately STRICTLY STRONGER than the controller's own rule, because it exists precisely to cover a rule the controller applies too late, and a check that merely mirrored the controller would inherit its blind spot. Monotonicity alone never contradicts a held tip strictly BELOW the proved sequence, whatever its digest: an attacker who mints a chain LONGER than the held tip but diverging BENEATH it gets no contradiction, so the forgery would close benignly, render, and land its bundle declarations. chain_tip_forks therefore also requires the proved chain to REPRODUCE the held tip's digest at the held tip's own sequence, which is the identical check secondmate_verified_chain in bin/fm-worker-authority.py already performs before it will prove landing. A held sequence that cannot be looked up in the proved chain (not a positive integer, or past its end) cannot be reproduced and is never read as agreement.

   Repairing the ordering inside the controller (running monotonicity before the release gate, so a fork refuses as a fork whatever the worker's phase) is a follow-up owned by bin/fm-worker-lifecycle.py, deliberately not touched here because PR feat(worker): authorized task home for compartment child requests #278 owns that file right now.

3. Everything else warns and retries under exponential backoff (not assigned, wrong assignment generation, unreadable controller, failed invocation), recorded durably in chain_tip_error with fatal: false, attempts, and next_attempt_at. These are statements about who owns the worker right now, not about the chain, and are expected to be transient; freezing on them would wedge a healthy compartment. Because every attempt takes the controller lock, retries back off 30s doubling to a 1h cap keyed on the durable error, and an identical repeated refusal is recorded rather than reprinted. Without that, a 15s poll loop stuck on an ownership refusal would cost roughly 5,700 controller-lock acquisitions and 5,700 pane lines a day for one compartment. The backoff resets when the tip being reported changes, since that is a genuinely different call.

The split is by the controller's own refusal texts, so a refusal this monitor does not recognise falls into the conservative warn-and-retry class rather than being mistaken for a fork.

Docs

docs/azure-workers.md no longer implies surrender is the normal exit. It states that the monitor records tips so the ordinary release authority can prove landing, describes the advance-only recording and the durable skip, the not-the-message-lane boundary, all three refusal semantics including the released-branch readback and the named controller-side follow-up, and the backoff. It keeps the honest residue below.

Test evidence

tests/fm-secondmate-cloud-monitor.test.sh: 37 units on main, 50 now (13 added). Registration audit (the harness registers by bare word, so a typo would silently exit 0): 50 registered, 50 unique, 50 defined, 0 registered-but-undefined, 0 defined-but-unregistered, 0 duplicates.

Run as CI runs it (python3 tests/run-one.py, which owns the process reaper; every backgrounded monitor is a session leader stopped by process-group kill):

$ python3 tests/run-one.py bash tests/fm-secondmate-cloud-monitor.test.sh
...
ok - a verified advance records the tip on the controller worker record with the exact argv
ok - the recorded (sequence, digest) is exactly what the local verifier proved
ok - an unchanged tip is skipped and only a genuine advance records again
ok - a refused chain verification attests no tip at all, gapped or substituted
ok - a monotonicity refusal freezes the lane like a chain break and relays nothing
ok - an already-released refusal closes the tip lane quietly and never freezes the compartment
ok - an ownership refusal warns once, backs off the controller lock, and retries instead of freezing
ok - a released worker whose held tip contradicts this chain freezes, though the CLI calls it benign
ok - a held tip below the proved sequence that this chain does not reproduce freezes, not closes
ok - a released worker whose held tip cannot contradict this chain closes the lane quietly
ok - a released refusal with an unreadable controller retries instead of closing on faith
ok - the whole monitor pass records the verified tip the release authority reads
ok - the monitor's chain tip argv is accepted and honoured by the real lifecycle CLI
# fm-secondmate-cloud-monitor.test.sh: all assertions passed
rc=0        (50 "ok -" lines, 0 "not ok", no tracebacks)

Sibling suites, same runner: tests/fm-worker-authority-secondmate.test.sh rc=0, tests/fm-worker-lifecycle.test.sh rc=0. bin/fm-lint.sh rc=0 over 246 scripts.

How the units avoid mocking the callee:

• The recorded tip is compared against an INDEPENDENT re-derivation of the collected mailbox that shares no code with the monitor, and the chain itself is produced by the real bin/fm-secondmate-session.py against the store. Three statements of the tip must agree: the monitor's durable state, the controller record, and the re-derivation.
• The argv is asserted byte-exact against the flag set bin/fm-worker-lifecycle.py declares.
• Four units now drive the REAL bin/fm-worker-lifecycle.sh against real controller documents built through the real module's own environment()/empty_state(): the healthy record, the released-with-rewind freeze, the released-with-below-tip-fork freeze, and the released-with-genuinely-clean-tip close. Each released unit asserts the real CLI really did answer with the released string, so they pin the ordering hole itself rather than a fixture's imitation of it.
• The "clean" control is clean for the right reason: its held tip carries the digest the proved chain actually reproduces at that sequence, asserted against an independent re-derivation. An earlier revision planted a bogus digest at a lower sequence, which meant that unit was pinning a below-tip fork as acceptable; that is fixed and is now its own freezing unit.

Mutation proofs

All eight applied, proven red, reverted, and re-proven green. Mutations 1-4 were re-run against the post-review code, not just against the version they were written for.

#	Mutation	Result
1	Drop the record_chain_tip call from command_process_mailbox	RED, 14 ok then not ok - the chain tip argv is not the exact contract
2	Attest BEFORE verification (fold the chain over the entries' name digests, so values stay byte-identical on a healthy chain and only ordering moves)	RED, 17 ok then not ok - a tip was attested for a mailbox whose bodies were substituted
3	Drop the unchanged-tip skip	RED, 16 ok then not ok - an unchanged tip was re-recorded (3 invocations for one tip)
4	Leave the recording lane unwired in the bash monitor (--lifecycle-bin "")	RED, 24 ok then not ok - ... the chain tip recorded through the monitor
5	Trust the released string without reading the held tip back (the pre-review behaviour)	RED, 21 ok then not ok - a released worker holding a contradicting tip did not freeze
6	Drop the backoff gate	RED, 20 ok then not ok - a backed-off pass still took the controller lock (2 invocations)
7	Close the lane on an unreadable controller	RED, 23 ok then not ok - the tip lane closed without reading the held tip back
8	Drop the reproduction clause, leaving the controller's monotonicity rule alone	RED, 22 ok then not ok - a held tip below the proved sequence with a digest this chain does not reproduce was accepted

Mutation 2 was deliberately re-done after a first attempt went red for the wrong reason (it changed the reported digest, so it failed the argv unit rather than the ordering one). The version above keeps the recorded values identical on a healthy chain, which is why the substitution assertion had to be added: it is the only case that separates "attested then verified" from "verified then attested".

Test-harness fix carried here

wait_for re-runs its argv on every retry, so any $(...) or <(...) in the arguments is evaluated ONCE at call time and the loop then re-tests a frozen value or an exhausted fd. Two instances are fixed, and both are replaced with in-shell predicates (inbox_has, worker_tip_recorded) rather than bash -c, which would need export -f to see the suite's functions:

• grep -q 'FIRSTMATE ACCEPTED' <(inbox_messages) was pre-existing and byte-identical on base, but this change inserts a subshell, a . "$CLOUD_ENV", and a lifecycle subprocess into process_mailbox immediately before child_relay in the same loop iteration, so it sits in that racing window.
• test -n "$(recorded_worker_tip)" was introduced by this PR and had the same defect.

A sweep of every <( in tests/ found no other instance inside a retry loop (the remaining uses are single-shot cmp/comm/assert_no_grep arguments), and every wait_for in this suite is now literal-argument or predicate.

Honest limits

• compartment-chain-tip remains an unverified ATTESTATION, as PR feat(worker): secondmate release authority and compartment status (R2/R3 PR 6) #271 documented: it never reads the mailbox, so its whole value is caller trust, and there is still no privilege separation between a caller who can write under state/ and one who can execute bin/fm-worker-lifecycle.sh. This change makes the honest monitor call it; it does not make the command self-verifying.
• The same attacker-writable state file can SUPPRESS attestation even though it cannot forge one. Planting chain_tip_closed, or a recorded_chain_tip equal to the tip about to be proved, makes the monitor skip the call silently and downgrades that compartment to surrender for good. The failure direction is the safe one (no false landing proof), but the suppression path is real and is now stated in the doc.
• The controller-side ordering (release gate before monotonicity) is worked around here, not fixed. The monitor's readback reproduces the controller's rule from a document it reads separately, so a controller whose rule changes and whose released-path text does not would drift; the released marker string is pinned against the real CLI to catch the text half of that.
• The freeze is per-compartment and needs an operator to clear the sticky marker, by design. A monotonicity refusal caused by controller-side corruption rather than a forged chain therefore also stops the lane. That is the intended trade.
• The retry backoff caps at 1 hour and never gives up. A compartment permanently unable to attest will keep one lock acquisition per hour and one durable state write per attempt.
• next_attempt_at is absolute wall-clock time and is persisted, so a backwards clock jump extends the wait by the size of the jump. Informational: the ladder is disclosed and the state file names the deadline.
• A compartment whose chain has stopped advancing and whose ownership refusal persists can land its final tip up to an hour late. An operator who releases inside that window gets surrender rather than the ordinary authority. Informational and operator-visible, not a correctness gap.
• The lifecycle CLI is still a fixture for the lanes that would reach Azure. No test drives a real Azure provider, and none can.
• Recording happens on the process-mailbox pass only. A compartment whose monitor never ran, or whose tip lane closed or froze before any tip landed, still exits through surrender.
• One pre-existing unit in this suite ("the reclaim did not release the claim for the next dispatch attempt") is mtime sensitive and has flaked once under CPU contention.

Does a compartment release ordinarily now?

Yes, for the case this owns. After a monitor pass whose chain verified, the controller-owned worker record carries verified_chain_tip, which is exactly what secondmate_chain_extent requires and refuses without. Proven end to end against the REAL lifecycle CLI writing a REAL controller document, and the authority's own suite is green against that same shape. surrender returns to being the degraded exit rather than the only one.

Rebased onto #278

Rebased onto origin/main at e2683acc (PR #278, authorized task home for compartment child requests). The predicted docs/azure-workers.md conflict did not materialise: #278's edits are in the compartment-request section (~line 40) and this PR's are in the release section (~line 199), so the auto-merge is genuinely two disjoint regions rather than two stitched halves. mergeStateStatus is CLEAN.

Two stale claims that #278 retired from the doc also lived in prose in this PR's own file, and the rebase is what made them false, so they are corrected here rather than left to rot:

• bin/fm-secondmate-cloud-monitor.py's module docstring said a compartment child was "NOT YET POSSIBLE" pending "an assertable owner_kind on the spawn side and an authorized task-home parameter", the exact two capabilities feat(worker): authorized task home for compartment child requests #278 delivered. It now records that both landed, and that the lane still refuses here for a different and honest reason: this relay does not set FM_SPAWN_TASK_HOME, so the spawn's task home is the primary home, owner_kind derives as primary, and verify_request refuses the parent-paired request as before. Behavior is unchanged and the existing unit still pins that refusal.
• spawn_environment's docstring reasoned from the retired "FM_HOME moves for the spawn" premise. It now states feat(worker): authorized task home for compartment child requests #278's split (FM_HOME keeps the money-document job only and never moves; FM_SPAWN_TASK_HOME carries the requester's authority and identity), and notes that feat(worker): authorized task home for compartment child requests #278's answer is the same one this comment already reached: move the TASK home, never FM_HOME. The FM_AZURE_WORKER_STATE_DIR no-pin rationale is retained because the doc still says the same thing.

Actually wiring the compartment's own home through as the task home would admit compartment children for real. That is a behavior change with its own bounds to prove, so it is named as a follow-up in the code rather than smuggled into the chain-tip lane. A grep for the retired claims (blocking compartment children, Until an assertable, stays pinned to the local controller, NOT YET POSSIBLE, missing capability) across docs/ and bin/ returns nothing.

Deferred follow-up, now done

The released marker string is now added to the owner-pinned marker list in tests/fm-worker-lifecycle.test.sh, alongside the second fork marker which was also unpinned there. Editing command_compartment_chain_tip and running only its owning suite now gives a local signal instead of relying on the cross-suite CI run. Proven load-bearing: typing the string wrong fails that suite with AssertionError: released work cannot record a compartment chain tipX / not ok - elastic worker static contract failed, rc=1.

One follow-up remains: the controller-side ordering repair described under refusal class 2.