Skip to content

fix(azure): bind a validation cell result to the attempt that produced it - #285

Merged
ruby-dlee merged 9 commits into
mainfrom
fm/r4-respond-nonanswer
Aug 21, 2026
Merged

fix(azure): bind a validation cell result to the attempt that produced it#285
ruby-dlee merged 9 commits into
mainfrom
fm/r4-respond-nonanswer

Conversation

@ruby-dlee

@ruby-dlee ruby-dlee commented Aug 21, 2026

Copy link
Copy Markdown
Owner

What this changes

A validation cell's published result is now bound to the attempt that produced it, an unbound
control-plane view can no longer end an attempt that is still running, and a guest edit can no
longer brick a cell that is already in flight.

observe previously treated "the Run Command instanceView reports a terminal executionState
and its output carries no result marker" as proof the attempt had died. Nothing bound the view it
read to the attempt it had just created, and the marker is the guest's last action, so its
absence proves nothing about an attempt still working. failed-retained is a phase observe
itself refuses, so one premature read was unrecoverable.

  • The guest stamps attempt into result.json and appends attempt=<n> to its marker.
  • observe accepts any marker naming the attempt being observed (finditer, not the first
    match), refuses a stamped marker naming another attempt, and refuses two conflicting results
    claiming one attempt.
  • An unbound view is never accepted. The terminal decision waits behind a settling window
    (FM_AZURE_VALIDATION_MARKER_SETTLE_SECONDS, default 300, re-armed per attempt in
    create_run_command), so silence means "could not tell" and never authorizes the destructive
    action. An unreadable stamp counts as not-yet-settled, never as elapsed.
  • observe refuses a result byte-identical to an earlier attempt's as an explicit non-answer
    rather than a generic failure.
  • verify_result_identity refuses a result that declares no attempt, or another attempt's.
  • Compatibility, both directions. create_run_command now runs the guest the request was
    SEALED with, read from the copy submit stages beside the request and accepted only when its
    digest is the sealed digest. And a cell whose sealed guest predates the stamp still has its
    unstamped marker read, with its result held to the pre-stamp contract.

Why: R4 blocker 1 was misdiagnosed, and the correction is in this PR

docs/azure-requirements.md recorded blocker 1 as "respond does not answer a gate" plus "the
guest re-publishes the byte-identical previous result". The second is wrong under every reading
of the evidence. The first is not established rather than confirmed. R4's own rule is that a
built claim contradicting the evidence is corrected, so the document is corrected here.

Ground truth is $FM_HOME/state/azure-validation/azv-36b2726cbcf3.json:

09:46:27  needs-decision   no-mistakes ask-user gate owns the exact run
09:47:51  responding       exact ask-user run received a protected response
09:48:00  failed-retained  cell ended without an authenticated result marker
09:50:30  failed-retained  exact disposable compute removed; worktree retained

respond created Run Command respond-a2 at 09:47:51. observe declared the attempt dead
nine seconds later and the VM was deleted 2m39s in. Attempt 1 had taken 1h57m.

The byte-identical republish was never a republish, and this does not depend on which
hypothesis below holds. The result blob has one fixed name per cell (staging.result_blob is
control/result.tar.gz), overwritten by each attempt's upload, and attempt 2 never reached its
upload. Same sha256 prefix 330ddea31bfbbb05, same 3056-byte run.log, same run_id, same
needs-decision, same gate. collect never ran: no result in the state file and
state/azure-validation/results/ is empty, so the comparison was made by downloading the one
blob directly, twice.

Two hypotheses remain open and the evidence does not separate them. An earlier revision of
this PR asserted the first as fact; that was wrong and is corrected:

  • (a) the view was stale, describing something other than the nine-second-old attempt; or
  • (b) attempt 2's guest genuinely failed fast, after its auth-home write-back and before its
    marker. control_error carries exactly two lines, the auth-home pull and push warnings, and
    they are emitted from the guest's own sequential path, not a trap: the sealed guest at
    payloads/azv-36b2726cbcf3/guest.sh calls auth_home_pull at line 504 and auth_home_push at
    line 895, its only trap is cleanup_mounts EXIT, and its marker is at line 1102. A guest past
    895 that died before 1102 produces exactly that stderr.

On resource identity (b) is better supported: resources.run_commands records two distinct
resources, start-a1 and respond-a2, and respond rebinds run_command_id to respond-a2
inside create_run_command before the responding transition, so the 09:48:00 read
addressed a resource nine seconds old, which cannot inherit start-a1's stderr. Ruling (b) out
would risk sending the next operator away from a real respond-path bug. Separating them needs a
live cell. The fix is correct under both, because what is established under both is that the
response was delivered as a protected run-command parameter and observe then ended the attempt
on a view it had never bound to it.

The latent defect this exposed is worse than the reported one

Had that view carried attempt 1's marker rather than no marker, observe would have accepted
it, collect would have downloaded attempt 1's archive, matched its digest, and passed
verify_result_identity, because on a resumed attempt the VM, boot_id, run_id, heads and
every other verified field are identical and result.json carried no attempt number. A silent
false verdict is categorically worse than a generic failure.

Upgrading no-mistakes on the host is NOT the fix

The cell's version comes from the runtime.tar.gz handed to submit --runtime-bundle: nothing
in this repo builds that bundle, it is extracted only on a start boot, and the request is
digest-sealed. The staged payload bundle for azv-36b2726cbcf3 declares
no_mistakes_version: 1.48.0 across 110 files, read from the bundle itself rather than the state
file, which records only its digest. A host upgrade reaches a cell only by rebuilding the bundle
and submitting a NEW cell.

Also in this PR: R7 flipped to DONE

R7 read Status: HOLDS, through R8. It has no acceptance sentence, so DONE means "no owner login
outstanding for any live lane", which is now true. Re-checked 2026-08-21 against
bin/fm-credential-expiry.py report and the live roster:

  • Eight pi profiles usable to 2026-08-29: openai-codex, -2 through -5, -7, -8, -9.
    The numbering skips 6, and those eight names are exactly the slot keys in
    ~/.pi/agent/auth.json, which is where bin/fm-pi-refresh.py takes its slot list from
    (DEFAULT_SOURCE) rather than scanning the account directory, so the legacy accounts/pi/1
    through 6 directories are never selected.
  • R8's LaunchAgent com.firstmate.pi-auth-refresh is active, runs = 7, last exit code = 0.
  • The crosscheck roster reads its GLM primary plus three of those same pi profiles, and the GLM
    lane authenticates with an api key, not an owner login. Whether it returns verdicts is R6.
  • Nothing on a live path reads a claude credential, an accounts/codex/* profile, or
    accounts/pi/1 through 6.

Stale fact corrected in the same edit. R7 called the third claude profile "refreshable with
material declared valid to 2026-09-10". That figure was the wrong field: it is the REFRESH token
horizon read as readiness. Confirmed by token length only, never value: claude/1 holds a
108-character access token that expired 2026-08-17T20:31:18Z behind a refresh token good to
2026-09-10; claude/2 and claude/3 hold access and refresh tokens of length zero. Renewing
is an owner login: bin/fm-pi-refresh.py contains zero occurrences of "claude" and one accounts
path, and bin/fm-credential-expiry.py has no write, POST or urlopen path. None of it is needed,
which is what makes R7 met rather than blocked.

Re-review findings N-1 and N-2, fixed

N-1 (blocking): the legacy gate inferred "this guest cannot stamp" from "the marker did not
parse".
LEGACY_MARKER is MARKER minus the attempt group, so any malformation of the attempt
field satisfies "no stamped marker matched" and bought the weaker pre-stamp contract. The
composed shape re-opened exactly the silent false verdict this PR exists to close: observing
attempt 2, a stale attempt-1 marker truncated inside its own attempt field makes the stamped set
empty, legacy binds, expected_result_digest becomes attempt 1's own digest, the blob still
holds attempt 1's archive so the digest check passes, and the attempt check is skipped. The
marker is the guest's LAST line by design, which is exactly where an output cap lands.

Fixed by removing the inference rather than patching it. guest_stamps_attempt reads whether the
sealed guest can stamp, from the sealed bytes sealed_guest_text already holds, recorded at
create_run_command time and derived from the seal for a state that predates it. A stamping
guest whose marker is malformed now fails closed instead of being handed the weaker contract.

The ninth mutation was green because producer and consumer were pinned separately: test 2g
asserted result_binding == "legacy" for the legacy case, nothing asserted == "attempt" for the
normal one, and the strict section built its own state dict by hand. Both are fixed: the normal
path now asserts the binding observe wrote, and the strict section is driven by that
recorded value rather than a hand-built default.

A second defect surfaced while pinning the strict fallback. An underivable binding raised
KeyError, not ValidationError, so observe would have crashed rather than failing closed on
exactly the states we know least about. The fallback now catches it and takes the STRICT contract.

N-2: sealed_guest_text digested one read and returned a different read. A TOCTOU whose bytes
upload as the Run Command script and execute as root on the cell. Now read once, digested in
memory, and those exact bytes returned. The docstring claimed this before the code did; it now
describes what the code does. The symlink refusal is pinned, and a race cannot be observed by
waiting for it, so the second read is instrumented to differ: an implementation that reads twice
returns the instrumented bytes and the test goes red.

Separate findings, deliberately NOT fixed here

1. A 90-minute dead poll per gate. adjudicate_gates polls
control/gate-response-a<n>-<i>.txt via fetch_gate_response, and nothing writes that blob.
gate-response appears exactly three times repo-wide: the guest's read, the doc, and a
tests/fm-azure-validation.test.sh:240 assertion that the string "appears somewhere", which pins
nothing. So the loop can only time out, polling every 30s until
FM_AZURE_VALIDATION_GATE_WAIT_SECONDS (default 5400 = 90 minutes). The arithmetic corroborates
it: 117.2 minutes to the gate verdict minus a 90-minute dead poll leaves 27.2 minutes of actual
pipeline work.

Recommended remedy: delete the loop. The operator answer already travels as a protected
run-command parameter, and deleting it removes nothing this guard depends on, because the settle
branch only runs on a terminal executionState while a guest parked in the poll keeps the
command Running. One argument for doing it sooner: at the default wall_seconds=10800 the
command timeout of 12600s exceeds the 5400s poll, so the poll does not currently manufacture a
TimedOut unbound view, but it would if wall_seconds ever dropped below 3600. Mixing the
change into this PR would blur what this one is for.

2. Unrecoverable phase residual. observe still drives into failed-retained, a phase it
refuses, so a false negative outlasting the settling window is recoverable only through
replace. The settling window mitigates this and does not remove it.

3. R4 leg one is blocked: the selection variable never reaches the step. Proven by direct
observation. A live no-mistakes run was started with
FM_AZURE_RUNNER_REMOTE_CLASSES='lint=validation-standard' exported into the axi run process.
Reading the environment of that run's own live step processes (found by cwd in the gate worktree,
per bin/fm-nm-step-liveness.sh) shows the step's shells carry zero FM_* variables: no
FM_AZURE_RUNNER_REMOTE_CLASSES, no FM_HOME. The only FM_* names anywhere in the tree are
eight FM_TEST_* set by tests/run.sh itself, with no overlap with the invoking shell's
FM_AZURE_* set. So no-mistakes does not forward the invoking CLI's environment to the step
commands it spawns; they inherit the shared daemon's environment, which its LaunchAgent pins to
{HOME, PATH}.

Consequence: bin/fm-azure-runner-dispatch.sh reads ${FM_AZURE_RUNNER_REMOTE_CLASSES:-} as
empty and runs exec "$@" locally. The recipe in docs/azure-runner.md cannot select a class
remote. The only environment that would reach the step is a machine-global launchctl setenv
plus a daemon restart, which is both a global flip and a restart of the daemon other pipelines
depend on. Leg one waits on a per-run scoping mechanism, not on the sealed suite becoming
Linux-clean: the class it needs is lint, already Linux-clean, while #282 unblocks
test=behavior-heavy. The run was aborted once this was established rather than spending further
capacity to watch a fallback that is now predictable.

4. R4 leg one carries no proof of where the work ran. The step's exit status and the
no-mistakes run record contain nothing distinguishing a real Azure execution from the local
fallback. The proof exists only in two artifacts outside no-mistakes: the
azure-runner: invocation=azr-<id> ... line print_logs_and_summary writes to stderr, and
$FM_HOME/state/azure-runner/azr-<id>.json carrying resources.vm_instance_id and
expected_boot_id. The proof of where the work ran should live in the same artifact as the
verdict.

5. Two records contradict the live credential reading.
$FM_HOME/config/crew-dispatch.json (2026-08-15) lists five of the eight openai-codex* slots as
"AWAITING RE-LOGIN (refresh tokens burned)"; the live report shows all eight usable to
2026-08-29. Separately, the live crosscheck-reviewer.json names the Fireworks GLM deployment
(accounts/pi/fireworks-glm) while bin/fm-crosscheck.py's comments describe an Azure AI Foundry
resource. The first is state that lost to a later reading. The second belongs inside R6, which
#281 rewrites, so it is untouched here and flagged for reconciliation there.

Verification

  • Run through tests/run.sh, the authoritative entry point, with the operator environment
    deliberately present (25 FM_AZURE_* names sourced from fleet.env). Earlier rounds of this PR
    were verified through python3 tests/run-one.py, which is the reaper and bypasses the ambient
    seal
    test(harness): seal the suite against the operator environment and gate its host-coupled units #282 added, so those runs said nothing about the sealed tree. Re-verified: 18 ok, 0 not ok.
  • tests/fm-crosscheck.test.sh (it reads docs/azure-requirements.md): green through the same
    entry point.
  • shellcheck --norc -x bin/fm-azure-validation-guest.sh tests/fm-azure-validation.test.sh: clean.
  • gate_answer_binding_contract drives the real guest emission region for attempts 1 and 2
    and asserts the two results differ only by attempt; runs the real observe against the
    unbound view, the settled unbound view, another attempt's marker, a republished digest, a
    genuine new result, two markers in one output, two conflicting markers for one attempt, and an
    unstamped legacy marker; runs the real verify_result_identity; and drives the real
    create_run_command to prove a sealed cell resumes on a guest edit and that each attempt
    re-arms its own settle window.
  • Thirteen mutations, each turns the suite red, all re-run through tests/run.sh. M1 guest
    marker drops attempt=; M2 the result-attempt check; M3 observe believes an unbound view at
    once; M4 the republish fence; M5 the per-attempt settle re-arm; M6 sealed_guest_text ignores
    the staged copy; M7 the legacy-marker fallback; M8 first-marker-wins restored; N9 force
    binding = "legacy" for every observation
    ; N10 drop the symlink refusal; N11 restore the
    double read
    ; N12 ungate the legacy branch from the sealed guest; N13 underivable binding
    falls back weak
    .

An earlier revision of this PR also carried a byte-equality fence inside collect. It was
unreachable, because observe fences the same condition first and the pre-existing
digest != expected_result_digest check catches the stale-blob case, and deleting it left the
suite green. It has been removed rather than left in as an unpinned control.

Scope

Rebased onto main at 1fc3bff6, including PR #288 Azure guest parser repair, PR #289 exact outer-schema prompt binding, and PR #290 one-shot Pi protocol correction with bounded diagnostics. The sealed validation, Azure crosscheck, and full crosscheck suites were rerun at exact head b0708482090b6e14ee9b0b6393e8644027bc9cfa after the rebase.

@ruby-dlee
ruby-dlee force-pushed the fm/r4-respond-nonanswer branch from c3f1a68 to b070848 Compare August 21, 2026 18:35
@ruby-dlee
ruby-dlee merged commit 9252690 into main Aug 21, 2026
13 checks passed
@ruby-dlee
ruby-dlee deleted the fm/r4-respond-nonanswer branch August 21, 2026 19:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant