test(harness): seal the suite against the operator environment and gate its host-coupled units - #282
Conversation
|
Verification log for the reviewer. Every line below was executed; nothing is inferred. Linux, real container (Ubuntu 24.04 with
ADMIT-red, host-capability gates (broken production code):
ADMIT-red, structural guard: gating one extra unit without declaring it turns ADMIT-red, cloud seal (same harness, same operator shell, a recording refusing
Provider-reach audit, all eight suites whose call graph can resolve
|
|
Verification update after the Caveat on CI, stated up front: the 13/13 green at The enumeration CONVERGEDThe earlier one-at-a-time iteration found 19 units and had not converged, which is why the partial set was reverted rather than shipped. Replaced with a single deterministic pass: the suite invokes every case through one choke point, Result: exactly 33 units, all in the secondmate teardown/retirement family. The 14 the iteration never reached include Linux verification of the gate, network off, cell declaration set: 110 units passed, 33 loud skips, 0 failures, both parts complete (72 + 71 = 143). 110 + 33 = 143 exactly. Control: the same two files with network available pass all 143 units with zero skips. macOS coverage: before vs after, executed units per fileBaseline is the merge-base
Zero The baseline run also reached the real control plane once more - |
…declaration The first live validation cell (azv-36b2726cbcf3, 2026-08-20) parked because the sealed suite is not Linux-clean. Its retained shard responses name three genuine host capabilities the cell does not have: a real tmux server it can create windows in, passwordless sudo with systemd-run, and the /usr/bin/cpp binding bin/fm-account-directory.sh needs before it can validate a Claude quota-axi Keychain approval marker. Fifteen units across six test files are bound to those three capabilities in tests/host-capabilities.tsv. tests/host-capability-gate.sh turns a capability the environment says it cannot provide into a loud FM_HOST_CAPABILITY_SKIP line on stdout and stderr, naming the test, the unit, the capability, what the capability is, and why it was unavailable. The gate never probes the capability. A probe reads false on a broken host too, which is how this program already lost mutation coverage once, to a unit gated on `command -v pi`. It reads two facts instead: the platform, and an explicit by-name declaration the environment makes about itself in FM_TEST_HOST_CAPABILITIES_ABSENT. Everything else runs, and a missing capability then fails. The declaration is refused outright on Darwin, so macOS coverage cannot be switched off by an environment variable, and CI declares nothing, so its coverage is unchanged. Only the cell declares, in bin/fm-azure-validation-shard-bridge.py, with bin/fm-azure-validation.py pinning the same constant and refusing any shard command that differs. The three silent `command -v tmux || skip` probes are gone: past the gate tmux is required, and a host that claims the capability without it goes red. tests/fm-host-capability-gate.test.sh is the structural guard. It pins the registry against the real call sites, pins the set at an exact literal count, pins the required call shape, and proves the gate refuses an unknown capability name and a Darwin declaration instead of skipping.
…e real cloud
The sealed suite provisioned real, billable, untracked cloud compute. On
2026-08-20 a Standard_D4as_v6 worker VM (vm-fm7c799d-wkr-01) was running in
the pilot resource group tagged task-binding=cloud-noenv-c7, a hardcoded
fixture id that exists only in tests/fm-spawn-cloud.test.sh, carrying a
repository-generation that is not a commit that exists. There was no
controller queue entry, no worker record, and no state/<task>.meta for it.
Nothing tracked it and nothing would ever have released it.
The unit that made it, test_cloud_spawn_fails_closed_when_the_lifecycle_
refuses_the_request, deliberately omits FM_WORKER_PROVIDER_COMMAND and the
FM_AZURE_* identity in order to prove the lifecycle REFUSES such a request.
Run in a shell with the operator's fleet.env sourced, the ambient identity
satisfied the very check the unit asserts is missing, and
bin/fm-worker-lifecycle.py's provider default resolved the REAL Azure
adapter. The request was served rather than refused.
Reproduced here without making any real Azure call, by placing a recording,
refusing `az` first on PATH: the unit failed "expected exit 1, got 0" and the
shim captured `az account show --subscription <operator subscription>`. That
is the same reach that, against a real CLI, becomes `az vm create`.
The same ambient reach is what made tests/fm-worker-lifecycle.test.sh
non-deterministic. FM_HOME alone was sufficient, and which unit failed varied
between runs, the signature of a suite reading live operator state.
Fail closed on the provider, never fall through. tests/run.sh now:
1. drops every name declared in tests/ambient-seal.tsv before it admits a
single test (FM_HOME, FM_WORKER_PROVIDER_COMMAND, FM_SPAWN_CLOUD, and
everything under FM_AZURE_, AZURE_ and ARM_), printing one
FM_AMBIENT_SEAL line so a reduced environment is visible;
2. binds FM_WORKER_PROVIDER_COMMAND to tests/cloud-provider-refusal.py, so
a test that names no provider gets a loud refusal instead of Azure, and
a test that means to drive one names its own fixture and still wins;
3. puts tests/cloud-guard-bin/az on every admitted test's PATH, so even a
path that reconstructs the real adapter cannot reach the control plane;
4. runs tests/cloud-reach-check.sh after every admitted test on every lane,
because both refusals RECORD, and any recorded reach fails the run and
names what was reached. Containment is the refusal; this is the alarm.
Nothing under FM_TEST_ is sealed: that prefix is the runner's own contract
with the test, and CI's FM_PI_* stay untouched.
tests/fm-cloud-provider-seal.test.sh is the structural guard. It pins the
registry at an exact literal set and count, pins that run.sh wires all four
layers on BOTH admission lanes and checks a reach after every lane, proves
each refusal refuses with 97 and records the exact argv, proves a recorded
reach fails the run and is not left to leak into the next test, and asserts
as live facts about its own process that no operator identity is set, that
the bound provider is the refusing one, and that `az` resolves to the guard.
The refused-request unit now also asserts, for itself, that it reached NO
provider at all rather than merely being refused by one.
…to live The gating commit named eleven failing test files from the first live validation cell, gated six, and left five red with a note saying they needed diagnosis on their own merits. This diagnoses all five by measurement, fixes the one that is a test defect, and records the rest. The measurement is a local reproduction of the cell's own package closure: Ubuntu 24.04 with exactly bin/fm-azure-cell-image.sh's apt set, unprivileged, no build toolchain, running the whole sealed suite in the same four parallel shards the cell uses. It executed 121 test files and failed six, and every failing assertion matched the cell's own text. THE FIX. Two fm-session-start units forced a "MISSING: node" diagnostic by deleting a fakebin node. Bootstrap detects a tool with `command -v` against a real system base PATH, so that only works on a host where node lives outside /usr/bin. On macOS it does. On the Linux the cell runs, nodesource installs /usr/bin/node, the deletion changed nothing, no MISSING line appeared, and both units failed for a reason with nothing to do with what they assert. They now ask the host which bootstrap-required tool it does NOT already provide, and use that one. If the host provides every candidate they FAIL, loudly, naming the base PATH: a required condition that fails when unmet rather than a unit that quietly asserts nothing. The helper assigns into a caller variable rather than echoing, because `fail` exits, and an exit inside a command substitution kills only the subshell and would leave the caller holding an empty tool name. WHAT THE OTHER FOUR ARE. fm-pi-watch-extension fails under four-way parallel load and PASSES on its own in the same container; fm-watcher-lock failed in the cell with exit 124, a timeout, next to Killed lines in the same shard log. Both are capacity, not capability, and a skip in either would hide a real regression. fm-teardown-a and fm-teardown-b share one root cause, now found by instrumenting run_secondmate_remote_probe: the probe never runs, because secondmate_remote_identity fails first. Two independent environment facts each produce it, and both were reproduced - no interface inventory at any of the four absolute paths secondmate_network_remote_identity accepts, and no outbound reach to the origin remote's host. With iproute2 AND network both files pass all 143 units; with iproute2 and --network none the identical refusal returns, and the cell's repository-command egress is deny-all by design, so no package fixes it there. That one is deliberately NOT gated here. Iterating found at least nineteen affected units, all in the secondmate teardown/retirement family, and the enumeration had not converged when it was stopped. A partial skip set is worse than none: it looks principled and covers less than it claims. It also needs a decision the test suite does not own, since permitting that probe a narrow egress path would keep the coverage rather than skip it. Both options and the measured unit list are recorded in docs/azure-requirements.md R4. One correction to the record while here: the tmux units PASSED in the Linux reproduction, and CI requires tmux on ubuntu-latest and runs them there today. real-tmux-server is a capability THIS CELL lacks, not one Linux lacks, which is exactly why the gate keys on an explicit by-name declaration rather than a blanket platform rule. A platform rule would have silently removed coverage that CI and an ordinary container both provide.
…red origin egress fm-teardown-a and fm-teardown-b were the last Linux blocker. Instrumenting run_secondmate_remote_probe showed the probe never runs at all: secondmate_remote_identity fails first, because it needs outbound DNS resolution and network reach to the origin remote's host before it can prove a surviving upstream. With network both files pass all 143 units; with --network none the identical refusal returns. The Azure validation cell's repository-command egress is deny-all BY DESIGN, so this is a genuine and permanent capability absence there, and no package fixes it. That is origin-egress, the fourth declared capability. ENUMERATED TO CONVERGENCE, not by iteration. The suite invokes every case through a single choke point, run_partitioned_test, so running each case in a subshell there reports EVERY failure in one pass instead of stopping at the first. Both files ran to completion with the network off - 143 of 143 cases - and the answer is exactly 33 units, all in the secondmate teardown/retirement family. That number is the point. An earlier one-at-a-time iteration found 19 and had not converged, and shipping those 19 would have been a skip set that looks principled and covers less than it claims. The 14 it never reached include test_secondmate_retirement_rejects_effective_ssh_redirects, test_secondmate_retirement_rejects_linked_worktree_graphs, test_secondmate_retirement_rejects_loopback_and_stale_tracking_authority, test_nested_secondmate_cleanup_requires_child_home_lock and test_normal_secondmate_retires_proven_detached_head. WHAT THIS COSTS, stated plainly because a skip must never read as coverage: those 33 units are SKIPPED in the cell, not preserved by another route. The cell does not verify secondmate teardown or retirement authority at all - not the landed-work refusals, not the registry locking, not the network-authority pinning, not the child quiescence ordering. macOS and CI still run every one of them, and CI is where that coverage lives for any change to bin/fm-teardown.sh. docs/azure-requirements.md R4 says this in those words. Deny-all egress in that cell is a security property, not an oversight. Permitting the upstream-authority probe a narrow egress path would give the cell this coverage back, and that is recorded in R4 as the alternative an owner can choose later. It is deliberately not done here: trading an isolation guarantee for a green check is an owner-level decision about the cell's security posture, not a test suite's call. The pinned skip-set literal moves 15 to 48 in the same diff as the set, which is what the structural guard exists to force.
5c5b7fb to
ef457a6
Compare
|
Rebased onto CI: 13/13 green, pinned to the head SHA via The one conflict was semantic, not textual
Kept the explicit unset — it states the unit's premise locally and holds even if the suite-wide seal is ever weakened. Deliberately did not keep the provider pin. Under the seal an unbound provider is the refusing one, so any request reaching a provider is recorded and TMPDIR versus #283's --task-home-out channelThe seal names six things — Ambient-leak proof re-driven post-rebaseOperator shell,
Note macOS units, before vs after, on the NEW baseline13 files, both runs shimmed. No file loses a unit. Full local sweep on the rebased tip122 files, 2575 units, zero az invocations, zero skips. Two failures, both investigated rather than waved off:
|
Commit 4 widened CELL_HOST_CAPABILITY_DECLARATION to four capabilities and left two descriptions at three. Both are places a reader goes to learn what a validation cell skips, and one of them asserted the opposite of the constant three lines beneath it. bin/fm-azure-validation-shard-bridge.py said the fm-teardown refusal "is NOT in this declaration, because its unit set has not been enumerated to convergence". It IS in the declaration - it is origin-egress - and the set WAS enumerated to convergence. docs/azure-validation.md enumerated three capabilities and never mentioned the fourth. Both now name origin-egress and, more importantly, say what it costs: the runner unit that executes these shards sets PrivateNetwork=yes, RestrictAddressFamilies=AF_UNIX and IPAddressDeny=any, so the secondmate upstream-authority probe can never reach the origin host, and 33 units - the whole secondmate teardown/retirement family - are SKIPPED in the cell rather than covered by some other route. macOS and CI are where that coverage lives. This is not cosmetic. Skipping 33 units is only acceptable while the skip stays legible, and a comment contradicting the constant below it is precisely how a known gap turns back into an assumed one. No CI check reads this prose: the guard compares the bridge and validation constants to each other and validates names against the registry, but not descriptions. Documentation only. The declaration constant is unchanged and still byte-identical on both sides.
|
Correction to my own earlier comment on this PR. I posted a "403 -> 440" macOS unit-count figure without stating what it counted. It is a thirteen-file targeted set, not the whole suite, and it is only meaningful under a stated definition of "unit". A reader comparing it against any other counting method will get a different absolute number and be right to. The PR body now carries the definition, the exact file list, the command, and the two SHAs, so it can be regenerated. The substantive claim is unchanged and is the part that matters: no file loses a unit (eleven identical, two gain), zero macOS Separately, the body now records the three-arm discriminating mutation that settled dropping main's provider pin, rather than leaving that as my reasoning. |
Open item filed here: fm-wake-queue fails deterministically on macOS mainRecorded on this PR rather than as an issue because the repository has issues disabled ( Not introduced by this PR. It reproduces identically on the base this PR branched from, on its head, and on merged Read this first: the failure message points the wrong wayOffered as a LEAD, not a diagnosis. The count is higher than expected, not lower: 8 expected, 14 observed. The message reads EvidenceByte-identical every run, from
Reproduce: Why CI never caught itEvery job in Please do not close this after one green runThe failure is deterministic, but the way it gets checked can look green when it is not. The assertion prints That is not hypothetical. While investigating, a single scripted run reported Scope, honestlyNine identical runs across three commits is strong evidence that the break is real and deterministic. It is weak evidence about how widely it reproduces: one Mac, Darwin 25.6.0, unverified on a second machine. "macOS" here means "this macOS host" until someone checks another. |
1 participant
Four commits, separable concerns. Read them in order.
Why
docs/azure-requirements.mdR4 records that no Azure validation cell can reach a green teststep, so none can reach
close, so R4's acceptance is unreachable. The first live cell(
azv-36b2726cbcf3) parked on a red test gate.While fixing that, a second defect surfaced that is worse than a red gate: the sealed suite
provisioned real, billable, untracked Azure VMs.
The gate is a DECLARATION, not a platform rule, and that is deliberate
The obvious design is "gate on the platform -
unameis a fact, a capability probe is not".That rule is right about probes and wrong about platforms, and the evidence is in this PR:
ubuntu-latest, which requirestmuxand runs them on everyPR today, and they passed in a Linux container built from the cell's own package set.
real-tmux-serveris a capability that cell lacks, not one Linux lacks. A blanketplatform gate would have silently deleted coverage that CI runs right now.
So the gate reads two facts and never probes: the platform, AND an explicit by-name declaration
the environment makes about itself in
FM_TEST_HOST_CAPABILITIES_ABSENT. Everything notdeclared RUNS, so a merely broken host goes red instead of quiet. The declaration is refused
outright on Darwin, so macOS coverage cannot be switched off by an environment variable. An
unknown capability name exits 97 rather than skipping. CI declares nothing, so its coverage is
unchanged. Only the cell declares, and its string is byte-pinned on both the emitting and the
validating side.
That is the durable lesson: prefer a required condition that fails when unmet; when a skip is
genuinely needed, key it on a declaration the environment must make about itself, never on a
probe of the thing you are guarding.
1.
skip the sealed suite's host-coupled units loudly, by declarationAdds
tests/host-capabilities.tsv,tests/host-capability-gate.shand a structural guard, andgates fifteen units on three capabilities the cell lacks.
2.
seal the suite against the operator environment and the real cloudA billable VM (
vm-fm7c799d-wkr-01) was found running taggedcloud-noenv-c7- a fixture idthat exists only in
tests/fm-spawn-cloud.test.sh- with a repository-generation that is not acommit that exists, and no controller record at all. The unit deliberately omits
FM_WORKER_PROVIDER_COMMANDand theFM_AZURE_*identity to prove the lifecycle REFUSES such arequest; with the operator's
fleet.envsourced, the ambient identity satisfied the very checkthe unit asserts is missing, and
bin/fm-worker-lifecycle.py's provider default resolved theREAL Azure adapter.
tests/run.shnow fails closed in four layers, all pinned bytests/fm-cloud-provider-seal.test.sh: drop the ambient names intests/ambient-seal.tsv; binda refusing fixture provider; put a refusing
azon PATH; and RECORD every refusal, failing therun after any test that reached a cloud seam.
3.
stop fm-session-start asserting on where node happens to liveTwo units forced a
MISSING: nodediagnostic by deleting a fakebinnode, which does nothingon a host with
/usr/bin/node. They now pick the first bootstrap-required tool the host doesnot already provide, and FAIL loudly if it provides all of them.
4.
gate the secondmate upstream-authority family on declared origin egressfm-teardown-a/brefuse withsecondmate home upstream probe cleanup is unverified.Instrumenting
run_secondmate_remote_probeshowed the probe never runs:secondmate_remote_identityfails first, because it needs outbound DNS resolution and reach to the origin remote's host. With
network the files pass all 143 units; with
--network nonethe identical refusal returns. Thecell's repository-command egress is deny-all by design, so this is a genuine and permanent
capability absence there. That is
origin-egress.Enumerated to convergence, in one deterministic pass. The suite invokes every case through a
single choke point, so running each case in a subshell there reports every failure in one run
instead of stopping at the first. Both files ran to completion with the network off - 143 of 143
cases - giving exactly 33 units, all in the secondmate teardown/retirement family. An earlier
one-at-a-time iteration found only 19 and had not converged; shipping those would have been a skip
set that looks principled and covers less than it claims.
What this costs, stated plainly. Those 33 units are SKIPPED in the cell, not preserved by
another route. The cell does not verify secondmate teardown or retirement authority at all - not
the landed-work refusals, not the registry locking, not the network-authority pinning, not the
child quiescence ordering. macOS and CI still run every one of them, and CI is where that coverage
lives for any change to
bin/fm-teardown.sh. R4 says this in those words.Deny-all egress is a security property, not an oversight. Permitting the probe a narrow egress path
would give the cell this coverage back, and R4 records that as the alternative an owner can choose
later. It is deliberately not done here: trading an isolation guarantee for a green check is an
owner-level decision, not a test suite's call.
Why the conflict resolution dropped main's provider pin
On rebase, main had independently fixed the same
cloud-noenv-c7defect per-unit: explicitunsetof tenFM_AZURE_*names PLUS a pin ofFM_WORKER_PROVIDER_COMMANDto the case fixture.The unset is kept. The pin is deliberately not, and that is a behavioral difference, not a taste
one, so it was settled by a discriminating mutation rather than an argument.
The mutation makes
environment()probe the provider BEFORE validating identity, then stillrefuse - a reach that a correct seal must catch. Three arms, run under the sealed runner:
Under the seal an unbound provider IS the refusing one, so any request reaching a provider is
recorded and
fm_assert_no_cloud_reachcatches it. Pinning a working fixture makes a reachunrecorded and silently neuters that assertion. The unit must prove it reached NO provider, not
that the one it reached was safe.
The pin defended a scenario that cannot occur: all 122 test files source
tests/test-entry.sh,which re-execs through
run.shwhen the runner token is absent;test-seal.py admitadditionallyrequires a uid-owned mode-safe token with matching JSON and a live runner ancestor; the unit keeps
its explicit unsets; and
require_uuidruns beforeprovider_argvresolves. Dropping the pin isstrictly stronger.
Verification performed
Every claim was executed. Full log in the review thread.
invariants. CI declares nothing, so all gated units run and pass there.
declaration.
SKIPS loudly behind the declaration on Linux.
names it.
az account show --subscription <the operator's real subscription>; with it intact, sameharness and same operator shell, it passes and reaches nothing. A recording, refusing
azshimcaptured every attempt, so nothing was provisioned.
macOS executed-unit counts, with a reproducible definition
Scope matters here, so state it rather than quote a bare number. This is a THIRTEEN-FILE targeted
set, not the whole suite, and "unit" means one line matching
^ok -on the stdout ofbash tests/run.sh <file>, one file per invocation, with a refusingazshim first on PATH.Measured at base
af4f50f3versus headef457a68(the head has since moved by one docs-onlycommit, which cannot change these):
The substantive claim: no file loses a unit. Eleven identical, two gain -
fm-azure-runner15 -> 17 andfm-worker-lifecycle2 -> 37 (the ambient failures that truncatedthem are fixed). Totals over this set are 403 -> 440. Zero
FM_HOST_CAPABILITY_SKIPlines onmacOS in either run, and zero az reaches in either run.
Those totals are scope-dependent and are NOT a whole-suite figure; a full macOS sweep at head is a
different measurement (122 files, 2575 units, two failures - one a load artifact that passes
unloaded, one pre-existing on
af4f50f3itself).Size
Over the 1000-added-line guideline. The concerns are separable but interlocking: the gate
registry, the seal that stops the suite reaching real cloud, and the Linux-clean completion all
move the same runner and the same registries, and splitting them would leave
maintransientlyin a state where the suite is gated but still reaches Azure.