fix(azure): bind a validation cell result to the attempt that produced it - #285
Merged
Conversation
ruby-dlee
force-pushed
the
fm/r4-respond-nonanswer
branch
2 times, most recently
from
August 21, 2026 15:10
da9bb77 to
01fe9b5
Compare
ruby-dlee
force-pushed
the
fm/r4-respond-nonanswer
branch
from
August 21, 2026 16:14
01fe9b5 to
b987aa9
Compare
ruby-dlee
force-pushed
the
fm/r4-respond-nonanswer
branch
from
August 21, 2026 17:15
b987aa9 to
c3f1a68
Compare
added 9 commits
August 21, 2026 14:29
ruby-dlee
force-pushed
the
fm/r4-respond-nonanswer
branch
from
August 21, 2026 18:35
c3f1a68 to
b070848
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
A validation cell's published result is now bound to the attempt that produced it, an unbound
control-plane view can no longer end an attempt that is still running, and a guest edit can no
longer brick a cell that is already in flight.
observepreviously treated "the Run Command instanceView reports a terminalexecutionStateand its output carries no result marker" as proof the attempt had died. Nothing bound the view it
read to the attempt it had just created, and the marker is the guest's last action, so its
absence proves nothing about an attempt still working.
failed-retainedis a phaseobserveitself refuses, so one premature read was unrecoverable.
attemptintoresult.jsonand appendsattempt=<n>to its marker.observeaccepts any marker naming the attempt being observed (finditer, not the firstmatch), refuses a stamped marker naming another attempt, and refuses two conflicting results
claiming one attempt.
(
FM_AZURE_VALIDATION_MARKER_SETTLE_SECONDS, default 300, re-armed per attempt increate_run_command), so silence means "could not tell" and never authorizes the destructiveaction. An unreadable stamp counts as not-yet-settled, never as elapsed.
observerefuses a result byte-identical to an earlier attempt's as an explicit non-answerrather than a generic failure.
verify_result_identityrefuses a result that declares no attempt, or another attempt's.create_run_commandnow runs the guest the request wasSEALED with, read from the copy
submitstages beside the request and accepted only when itsdigest is the sealed digest. And a cell whose sealed guest predates the stamp still has its
unstamped marker read, with its result held to the pre-stamp contract.
Why: R4 blocker 1 was misdiagnosed, and the correction is in this PR
docs/azure-requirements.mdrecorded blocker 1 as "responddoes not answer a gate" plus "theguest re-publishes the byte-identical previous result". The second is wrong under every reading
of the evidence. The first is not established rather than confirmed. R4's own rule is that a
built claim contradicting the evidence is corrected, so the document is corrected here.
Ground truth is
$FM_HOME/state/azure-validation/azv-36b2726cbcf3.json:respondcreated Run Commandrespond-a2at 09:47:51.observedeclared the attempt deadnine seconds later and the VM was deleted 2m39s in. Attempt 1 had taken 1h57m.
The byte-identical republish was never a republish, and this does not depend on which
hypothesis below holds. The result blob has one fixed name per cell (
staging.result_blobiscontrol/result.tar.gz), overwritten by each attempt's upload, and attempt 2 never reached itsupload. Same sha256 prefix
330ddea31bfbbb05, same 3056-byterun.log, samerun_id, sameneeds-decision, same gate.collectnever ran: noresultin the state file andstate/azure-validation/results/is empty, so the comparison was made by downloading the oneblob directly, twice.
Two hypotheses remain open and the evidence does not separate them. An earlier revision of
this PR asserted the first as fact; that was wrong and is corrected:
marker.
control_errorcarries exactly two lines, the auth-home pull and push warnings, andthey are emitted from the guest's own sequential path, not a trap: the sealed guest at
payloads/azv-36b2726cbcf3/guest.shcallsauth_home_pullat line 504 andauth_home_pushatline 895, its only trap is
cleanup_mounts EXIT, and its marker is at line 1102. A guest past895 that died before 1102 produces exactly that stderr.
On resource identity (b) is better supported:
resources.run_commandsrecords two distinctresources,
start-a1andrespond-a2, andrespondrebindsrun_command_idtorespond-a2inside
create_run_commandbefore therespondingtransition, so the 09:48:00 readaddressed a resource nine seconds old, which cannot inherit
start-a1's stderr. Ruling (b) outwould risk sending the next operator away from a real respond-path bug. Separating them needs a
live cell. The fix is correct under both, because what is established under both is that the
response was delivered as a protected run-command parameter and
observethen ended the attempton a view it had never bound to it.
The latent defect this exposed is worse than the reported one
Had that view carried attempt 1's marker rather than no marker,
observewould have acceptedit,
collectwould have downloaded attempt 1's archive, matched its digest, and passedverify_result_identity, because on a resumed attempt the VM,boot_id,run_id, heads andevery other verified field are identical and
result.jsoncarried no attempt number. A silentfalse verdict is categorically worse than a generic failure.
Upgrading no-mistakes on the host is NOT the fix
The cell's version comes from the
runtime.tar.gzhanded tosubmit --runtime-bundle: nothingin this repo builds that bundle, it is extracted only on a
startboot, and the request isdigest-sealed. The staged payload bundle for
azv-36b2726cbcf3declaresno_mistakes_version: 1.48.0across 110 files, read from the bundle itself rather than the statefile, which records only its digest. A host upgrade reaches a cell only by rebuilding the bundle
and submitting a NEW cell.
Also in this PR: R7 flipped to DONE
R7 read
Status: HOLDS, through R8.It has no acceptance sentence, so DONE means "no owner loginoutstanding for any live lane", which is now true. Re-checked 2026-08-21 against
bin/fm-credential-expiry.py reportand the live roster:openai-codex,-2through-5,-7,-8,-9.The numbering skips 6, and those eight names are exactly the slot keys in
~/.pi/agent/auth.json, which is wherebin/fm-pi-refresh.pytakes its slot list from(
DEFAULT_SOURCE) rather than scanning the account directory, so the legacyaccounts/pi/1through
6directories are never selected.com.firstmate.pi-auth-refreshisactive,runs = 7,last exit code = 0.lane authenticates with an api key, not an owner login. Whether it returns verdicts is R6.
accounts/codex/*profile, oraccounts/pi/1through6.Stale fact corrected in the same edit. R7 called the third claude profile "refreshable with
material declared valid to 2026-09-10". That figure was the wrong field: it is the REFRESH token
horizon read as readiness. Confirmed by token length only, never value:
claude/1holds a108-character access token that expired 2026-08-17T20:31:18Z behind a refresh token good to
2026-09-10;
claude/2andclaude/3hold access and refresh tokens of length zero. Renewingis an owner login:
bin/fm-pi-refresh.pycontains zero occurrences of "claude" and one accountspath, and
bin/fm-credential-expiry.pyhas no write, POST or urlopen path. None of it is needed,which is what makes R7 met rather than blocked.
Re-review findings N-1 and N-2, fixed
N-1 (blocking): the legacy gate inferred "this guest cannot stamp" from "the marker did not
parse".
LEGACY_MARKERisMARKERminus the attempt group, so any malformation of the attemptfield satisfies "no stamped marker matched" and bought the weaker pre-stamp contract. The
composed shape re-opened exactly the silent false verdict this PR exists to close: observing
attempt 2, a stale attempt-1 marker truncated inside its own attempt field makes the stamped set
empty, legacy binds,
expected_result_digestbecomes attempt 1's own digest, the blob stillholds attempt 1's archive so the digest check passes, and the attempt check is skipped. The
marker is the guest's LAST line by design, which is exactly where an output cap lands.
Fixed by removing the inference rather than patching it.
guest_stamps_attemptreads whether thesealed guest can stamp, from the sealed bytes
sealed_guest_textalready holds, recorded atcreate_run_commandtime and derived from the seal for a state that predates it. A stampingguest whose marker is malformed now fails closed instead of being handed the weaker contract.
The ninth mutation was green because producer and consumer were pinned separately: test 2g
asserted
result_binding == "legacy"for the legacy case, nothing asserted== "attempt"for thenormal one, and the strict section built its own state dict by hand. Both are fixed: the normal
path now asserts the binding
observewrote, and the strict section is driven by thatrecorded value rather than a hand-built default.
A second defect surfaced while pinning the strict fallback. An underivable binding raised
KeyError, notValidationError, soobservewould have crashed rather than failing closed onexactly the states we know least about. The fallback now catches it and takes the STRICT contract.
N-2:
sealed_guest_textdigested one read and returned a different read. A TOCTOU whose bytesupload as the Run Command script and execute as root on the cell. Now read once, digested in
memory, and those exact bytes returned. The docstring claimed this before the code did; it now
describes what the code does. The symlink refusal is pinned, and a race cannot be observed by
waiting for it, so the second read is instrumented to differ: an implementation that reads twice
returns the instrumented bytes and the test goes red.
Separate findings, deliberately NOT fixed here
1. A 90-minute dead poll per gate.
adjudicate_gatespollscontrol/gate-response-a<n>-<i>.txtviafetch_gate_response, and nothing writes that blob.gate-responseappears exactly three times repo-wide: the guest's read, the doc, and atests/fm-azure-validation.test.sh:240assertion that the string "appears somewhere", which pinsnothing. So the loop can only time out, polling every 30s until
FM_AZURE_VALIDATION_GATE_WAIT_SECONDS(default 5400 = 90 minutes). The arithmetic corroboratesit: 117.2 minutes to the gate verdict minus a 90-minute dead poll leaves 27.2 minutes of actual
pipeline work.
Recommended remedy: delete the loop. The operator answer already travels as a protected
run-command parameter, and deleting it removes nothing this guard depends on, because the settle
branch only runs on a terminal
executionStatewhile a guest parked in the poll keeps thecommand
Running. One argument for doing it sooner: at the defaultwall_seconds=10800thecommand timeout of 12600s exceeds the 5400s poll, so the poll does not currently manufacture a
TimedOutunbound view, but it would ifwall_secondsever dropped below 3600. Mixing thechange into this PR would blur what this one is for.
2. Unrecoverable phase residual.
observestill drives intofailed-retained, a phase itrefuses, so a false negative outlasting the settling window is recoverable only through
replace. The settling window mitigates this and does not remove it.3. R4 leg one is blocked: the selection variable never reaches the step. Proven by direct
observation. A live no-mistakes run was started with
FM_AZURE_RUNNER_REMOTE_CLASSES='lint=validation-standard'exported into theaxi runprocess.Reading the environment of that run's own live step processes (found by cwd in the gate worktree,
per
bin/fm-nm-step-liveness.sh) shows the step's shells carry zeroFM_*variables: noFM_AZURE_RUNNER_REMOTE_CLASSES, noFM_HOME. The onlyFM_*names anywhere in the tree areeight
FM_TEST_*set bytests/run.shitself, with no overlap with the invoking shell'sFM_AZURE_*set. So no-mistakes does not forward the invoking CLI's environment to the stepcommands it spawns; they inherit the shared daemon's environment, which its LaunchAgent pins to
{HOME, PATH}.Consequence:
bin/fm-azure-runner-dispatch.shreads${FM_AZURE_RUNNER_REMOTE_CLASSES:-}asempty and runs
exec "$@"locally. The recipe indocs/azure-runner.mdcannot select a classremote. The only environment that would reach the step is a machine-global
launchctl setenvplus a daemon restart, which is both a global flip and a restart of the daemon other pipelines
depend on. Leg one waits on a per-run scoping mechanism, not on the sealed suite becoming
Linux-clean: the class it needs is
lint, already Linux-clean, while #282 unblockstest=behavior-heavy. The run was aborted once this was established rather than spending furthercapacity to watch a fallback that is now predictable.
4. R4 leg one carries no proof of where the work ran. The step's exit status and the
no-mistakes run record contain nothing distinguishing a real Azure execution from the local
fallback. The proof exists only in two artifacts outside no-mistakes: the
azure-runner: invocation=azr-<id> ...lineprint_logs_and_summarywrites to stderr, and$FM_HOME/state/azure-runner/azr-<id>.jsoncarryingresources.vm_instance_idandexpected_boot_id. The proof of where the work ran should live in the same artifact as theverdict.
5. Two records contradict the live credential reading.
$FM_HOME/config/crew-dispatch.json(2026-08-15) lists five of the eightopenai-codex*slots as"AWAITING RE-LOGIN (refresh tokens burned)"; the live report shows all eight usable to
2026-08-29. Separately, the live
crosscheck-reviewer.jsonnames the Fireworks GLM deployment(
accounts/pi/fireworks-glm) whilebin/fm-crosscheck.py's comments describe an Azure AI Foundryresource. The first is state that lost to a later reading. The second belongs inside R6, which
#281 rewrites, so it is untouched here and flagged for reconciliation there.
Verification
tests/run.sh, the authoritative entry point, with the operator environmentdeliberately present (25
FM_AZURE_*names sourced fromfleet.env). Earlier rounds of this PRwere verified through
python3 tests/run-one.py, which is the reaper and bypasses the ambientseal test(harness): seal the suite against the operator environment and gate its host-coupled units #282 added, so those runs said nothing about the sealed tree. Re-verified: 18 ok, 0 not ok.
tests/fm-crosscheck.test.sh(it readsdocs/azure-requirements.md): green through the sameentry point.
shellcheck --norc -x bin/fm-azure-validation-guest.sh tests/fm-azure-validation.test.sh: clean.gate_answer_binding_contractdrives the real guest emission region for attempts 1 and 2and asserts the two results differ only by attempt; runs the real
observeagainst theunbound view, the settled unbound view, another attempt's marker, a republished digest, a
genuine new result, two markers in one output, two conflicting markers for one attempt, and an
unstamped legacy marker; runs the real
verify_result_identity; and drives the realcreate_run_commandto prove a sealed cell resumes on a guest edit and that each attemptre-arms its own settle window.
tests/run.sh. M1 guestmarker drops
attempt=; M2 the result-attempt check; M3observebelieves an unbound view atonce; M4 the republish fence; M5 the per-attempt settle re-arm; M6
sealed_guest_textignoresthe staged copy; M7 the legacy-marker fallback; M8 first-marker-wins restored; N9 force
binding = "legacy"for every observation; N10 drop the symlink refusal; N11 restore thedouble read; N12 ungate the legacy branch from the sealed guest; N13 underivable binding
falls back weak.
An earlier revision of this PR also carried a byte-equality fence inside
collect. It wasunreachable, because
observefences the same condition first and the pre-existingdigest != expected_result_digestcheck catches the stale-blob case, and deleting it left thesuite green. It has been removed rather than left in as an unpinned control.
Scope
Rebased onto
mainat1fc3bff6, including PR #288 Azure guest parser repair, PR #289 exact outer-schema prompt binding, and PR #290 one-shot Pi protocol correction with bounded diagnostics. The sealed validation, Azure crosscheck, and full crosscheck suites were rerun at exact headb0708482090b6e14ee9b0b6393e8644027bc9cfaafter the rebase.