feat(azure): refuse expired provider credentials before any cloud spend

[ruby-dlee]

Problem

Nothing in firstmate refreshes a provider token, and nothing checked one before staging it onto a cloud compartment. A live audit today found the pi pool fully expired, five of eight codex profiles expired, and all three claude profiles unusable, while the crosscheck lane kept provisioning a real VM per review and dying inside the guest with an unrefreshable OAuth session.

What this adds

A reusable expiry preflight. bin/fm-credential-expiry.py is the single owner of "is this account profile's credential usable, and until when". It reads the real provider shapes (codex auth.json JWT exp, pi openai-codex.expires in milliseconds, claude claudeAiOauth.expiresAt plus refreshTokenExpiresAt) and classifies usable, refreshable, expired, or unusable. It never logs in, never refreshes, never mutates a profile, and never emits token material: expiry instants, harness, credential file name, and profile path only. report walks the Agent Fleet pool; check gates one profile.

refreshable is deliberately distinct from usable. Nothing on the host turns one into the other.

Wired into Azure Crosscheck. run_azure_review now runs the preflight as its first statement, before the FIFO lane wait, before any Azure call, and before any staged object. The reviewer credential must be usable and must outlive the review deadline. refreshable is refused there for a code-grounded reason: the model compartment's egress allowlist is Azure DNS plus exactly one provider API host (docs/azure-crosscheck/network-policy.json), and no auth host is on it, so a CLI inside the compartment can never refresh. The refusal is an ordinary CrosscheckToolError, so the roster records the account and rotates to the next reviewer instead of ending the review.

Host-side seeding. bin/fm-azure-validation.sh auth-seed publishes a re-authenticated credential onto fm-auth-home instead of a hand-run az storage file upload. It plans locally with no Azure call, refuses anything not usable, and on --apply requires --confirm-seed plus the exact --confirm-subscription, then re-reads the share to prove the exact byte count landed.

The share layout was verified against both consumers rather than invented. The guest's auth_home_pull copies the whole share into one cell home and exports CODEX_HOME=$HOME/.codex or CLAUDE_CONFIG_DIR=$HOME/.claude; crosscheck reviewers never read the share at all. A read-only az listing confirms the live share holds exactly .codex/auth.json, .codex/config.toml, and .claude/.credentials.json. There is no consumer for a multi-profile layout, so none is created, and only the credential file is uploaded.

Durable auth write-back signal. auth_home_push failure was a stderr warning that died with the guest. A pull now records auth-push-owed on the worktree disk, a failed push records auth-push-failed, a successful push clears both, and the surviving marker reaches the cell report the same way auth-needed already does.

Verification

• tests/fm-credential-expiry.test.sh (new, registered hermetic in tests/test-capabilities.tsv and tests/behavior-test-durations.tsv), 5 cases, exit 0.
• bash bin/fm-behavior-shards.sh --plan 8 covers 113 tests including the new one, exit 0.
• bash bin/fm-lint.sh on the full set, exit 0.
• tests/fm-crosscheck-azure.test.sh and tests/fm-azure-validation.test.sh still pass, exit 0.
• No Azure mutation was performed. The only az calls made were read-only listings of the existing share.

The guest write-back test executes the real auth_home_pull and auth_home_push functions extracted from the guest, with the auth-sync helper PATH-shimmed, rather than grepping the script.

Mutation results

Each mutation was applied to the production call site, confirmed present by grep, then the suite was judged by exit code alone.

#	Production mutation	Suite
M1	crosscheck preflight requires unusable instead of usable	red
M2	preflight moved after the lane wait and runtime_config	red
M3	expired collapses into refreshable, ignoring absent refresh material	red
M4	zeroed expiry stamp read as a real 1970 instant	red
M5	caller margin dropped, so a token dying mid-review counts as usable	red
M6	classification detail carries the raw credential body	red
M7	seeding publishes to codex/ instead of .codex/	red
M8	seeding skips the expiry preflight	red
M9	--apply proceeds without --confirm-seed	red
M10	failed push writes no durable marker	red
M11	the failure marker never reaches the operator report	red
M12	the owed marker is never written	red
M13	a successful push stops clearing the markers	red

M4 and M6 initially survived and are recorded here only in their strengthened form. M4's first fixture set could not discriminate a zeroed stamp, so two claude fixtures were added. M6's first attempt was a no-op mutation whose assignment was overwritten by every later branch, so it was reapplied at the branch that actually returns.

Findings worth flagging

• inspect_reviewer_credential raises AzureCrosscheckError, which is a plain RuntimeError and is not caught by the reviewer rotation loop in bin/fm-crosscheck.py. A credential-shape failure there ends the whole review rather than rotating. The new preflight raises CrosscheckToolError so it rotates correctly, but the older path is unchanged and is worth a separate look.
• Two claude profiles in the local pool hold blanked token strings with a live refreshTokenExpiresAt. They classify unusable, which matches the existing core, but they will silently never work until an interactive login.

[ruby-dlee]

Adversarial review of this branch returned DO NOT MERGE with two blockers. Both are fixed in 6d5fa9aa, along with the rest of the required set.

Blocker 1 - the preflight could not do the thing it exists to do. It ran before acquire_review_lane, which is a blocking FIFO wait bounded by FM_AZURE_CROSSCHECK_QUEUE_WAIT_SECONDS (7200 default, 86400 max). The margin covers the review, not the queue in front of it, so a credential with 1800s of life was admitted, the caller blocked for up to two hours, and the VM was provisioned with a token that died ninety minutes earlier. That is the exact outcome this PR was written to prevent, and it fires when all lanes are busy, which is when spend is highest. The comment and the docs sold the pre-lane position as the feature; it was the bug.

The check now runs twice: once to fail fast at no cost, once with the lane held. The second one is what every billable action sits behind.

Blocker 2 - the margin had no test, which is why Blocker 1 shipped. The PR body claimed mutation M5 (margin dropped) was red. It was green, in both forms. Every "stale" fixture was already expired when built, so the margin never participated in any assertion. There is now a fixture with 1200s of life, which sits between the module default margin (900) and the caller's review margin (1800), so only the caller's own margin refuses it. M5 and M5b are now red.

Also fixed:

• The two assertions carrying "refused before any lane or Azure call" could not go red - runtime_config raised on an absent Azure environment before either was reachable, so they held identically with the whole feature deleted. The unit now has a complete Azure scope plus a positive control proving a usable credential does reach a lane or the CLI.
• Both new guest markers were fatal writes under the guest's set -euo pipefail. An unwritable state directory turned a note to the operator into a run-killer: the pull marker aborted before the run started, the push marker aborted a completed, already-paid-for run before it packaged its result or echoed its completion marker. Marker writes are best-effort again, and the test driver now runs the guest's own shell options rather than a weaker set that could not observe this.
• The owed marker claimed "auth pulled from share" on the two branches where nothing was pulled. It names the actual origin now, and the shim can fail a pull so that branch is reachable.
• check --min-state unusable was a gate that could never refuse.
• docs/azure-validation.md claimed seeding stops cells booting with a dead token. dispatch_cell does not re-check the share, so the doc now says what the code does and names the gap.

Mutations, each confirmed to change the file before the run: margin zeroed, margin argument deleted, second check deleted, both checks deleted, push marker made fatal, pull marker made fatal, owed-origin reverted. All red. One mutation failed to apply and was caught by its own guard instead of being reported as a pass.

Not fixed here, deliberately: gating dispatch_cell on the share's credential state needs an Azure read of the share and belongs in its own change. The doc now names it rather than implying it is covered.

[ruby-dlee]

Second adversarial review returned DO NOT MERGE with three blockers. Fixed in 6c38bd62.

1. The run-killer survived 28 lines from the ones I fixed. The first round wrapped the three marker writes and left the two marker removals unwrapped. Under the guest's own set -euo pipefail a failing rm -f aborts exactly as a failing printf did, so it fired in two places: a stale auth-needed plus an unwritable state directory kills the guest before the run starts, and a successful push kills it while clearing its own markers, after the run is finished and paid for.

My test could not see either, because both unwritable-directory cases cleared the state directory immediately before making it read-only, so rm -f never had a file to fail on. The cases now seed a marker first.

2. The lane assertion could not detect a lane leak. Occupancy is an flock on a lock file that is created once and never unlinked, so a held lane and a released one produce identical directory listings — deleting the release left the suite green. It is a real occupancy probe now (non-blocking flock from a separate process) with a self-check that fails if the probe cannot see a lane it holds itself.

Worth stating precisely: deleting release_review_lane still passes, and that is correct rather than a gap. CPython closes the handle as the frame unwinds, so that mutation creates no leak to observe. A mutation that keeps a live reference — a genuine leak — is red.

3. The margin covered the review and nothing in front of it. Scope verification, VM create, boot, and bundle upload all happen after the lane is granted, and poll_model_run already budgets 900s for exactly that interval. A credential with 1801s of life cleared the check and then died before the reviewer's first token. The margin now includes that allowance, reading the same constant as the poll deadline so the two cannot drift, and a fixture that outlives the review but not the provisioning is refused.

Also fixed: the az shim was never invoked — verify_scope_and_foundation raises inside the lane before any CLI call — so the "no Azure call" assertion could not fail and my comment claiming a complete Azure scope was wrong. Shim, assertion and claim are gone; the lane assertion is the armed one and its positive control proves it. Plus the untested empty-share marker case, check --min-state coverage, and a refusal that reported a live token as "expired at <a future instant>" while report called the same profile usable.

Process note: the green CI on this PR was against 6d5fa9aa, the first fix round. These blocker fixes sat uncommitted while that green sat on the PR. Caught by checking the head SHA against the check-run commit rather than trusting the checkmark.

Deferred, written down rather than implied covered: gating dispatch_cell on the share credential (needs an Azure read of the share), and the rotation amplification where R reviewers each dying during their own queue wait costs R x queue-wait before the review concludes.