fix(bridge-cutover): accept the pinned post-cutover state so worker-state steps unblock - #9
Merged
Conversation
…d machine The sealed-adoption quiet point accepts the live registry only at the two pre-cutover hashes, but the documented order applies the runtime switch (step 4) and boundary (step 5) before worker-state verification (6b) and finalize (6c), so their transitive bundle re-validation refused the legitimately advanced machine - the 2026-07-19 live NO-GO - and armed the rollback-attribution trap for the first published identity bundle. Keep the strict adoption plan byte-identical and add a pinned fallback: when it refuses, validate_bundle accepts exactly one further state - main cutover fully applied with the post-install boundary marked, a fully sealed adoption journal, intact retained sealed trees, and every adoption-managed live path at its exact cutover-new identity from the cutover manifest - reported as the new runtime-switched phase. Anything else refuses exactly as before. Regression tests pin the accepted set (initial, sealed, exact candidate) and run worker-state 6a/6b/6c and rollback attribution in the exact live order; pre-fix they reproduce the literal live refusal.
…nto routing test lab
… path on strict refusal
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Captain-issued work order (2026-07-19, workorder-cutover-workerstate-registry-fix.md): the Bridge account-routing cutover is half-applied and parked because bridge_worker_state_transaction.py --verify-provisioned (6b) and --finalize (6c) transitively re-run the sealed-adoption quiet point, which accepts the live registry only at the two pre-cutover hashes, while step 4 already legitimately rewrote it to the candidate registry - the 2026-07-19 live NO-GO. Both reverse edges are closed by design (rollback refused past the post-install boundary; re-prepare impossible post-adoption), so the tooling must accept the advanced state. The same refusal arms an unrecoverable trap: once a first provider login publishes an identity bundle, worker-state rollback() must call _candidate_api(), the exact refusing loader. Deliberate design decisions: (1) The work order's candidate fix (tolerate the registry as a third accepted state, mirroring the current-link tolerance) was verified INSUFFICIENT during investigation - after the registry digest, the adoption plan's observe() also refuses the post-cutover current-link targets and front door, and validate_bundle's phase branch then rejects the applied-plus-boundary main cutover. So the fix is a pinned post-cutover assessment instead: the strict adoption plan stays byte-identical for all adoption-time callers, and only when it refuses does validate_bundle check the one further accepted state - main cutover fully applied AND post-install boundary marked AND fully sealed adoption journal AND intact retained sealed release trees AND every adoption-managed live path (both current links, front door, registry) at its exact cutover-new identity pinned from the exact reconstructed cutover manifest - reported as a new runtime-switched phase with runtime_switch_ready remaining false. Any other combination re-raises the original strict refusal with the cutover-side probe diagnosis preserved as a suffix, and a full-apply WITHOUT the marked boundary intentionally keeps refusing (pinned to the documented 4-then-5-then-6 order). (2) Regression tests pin the accepted-registry set explicitly as the work order demands: tampered post-cutover registry refuses with unknown SHA-256, candidate registry before adoption refuses, partially reverted post-cutover state refuses, and the accept path requires the boundary. (3) Worker-state tests run 6a/6b/6c and rollback identity-drift attribution in the exact live order (adoption, step 4, boundary, then worker state) - pre-fix these reproduce the literal live NO-GO error string; the rollback test proves the armed trap is defused. (4) A Transaction order doc paragraph records the pinned post-cutover acceptance and why. (5) Prior gate rounds on this branch already applied and committed: review fixes preserving the cutover probe diagnosis, refusing malformed pins without crashing, and asserting the registry pin at the assessment site (515cc78), plus a test-lab opt-in for tests/fm-bootstrap.test.sh agent-fleet preflight cases (9db99c5) - that bootstrap test failure was machine-state-caused (the live cutover installed the real ~/.local/bin front door, which the physical-path resolution found despite the sandbox PATH) and is unrelated to the bridge diff; do not re-litigate those settled rounds. A prior run (01KXYYSQT5V84ACPQFQVQV1C3N) validated this exact content through review and test fix rounds and died only from an agent-process crash at the end of the test step. Full bridge-cutover python lane, fm-bootstrap.test.sh, and the key post-cutover regression tests all pass locally at this HEAD. Constraint honored: no absolute agent-fleet releases paths appear anywhere in this work (tilde/relative only); the change alters what a security-relevant validator accepts, which is why it ships through this full pipeline with pinning tests rather than as a hot patch.
What Changed
post_cutover_plantobridge_sealed_adoption.py: a non-mutating assessment that accepts a sealed adoption superseded by the fully applied main cutover, requiring a caller-supplied pin per adoption-managed path (registry file pin, current-link targets, front door), a fully sealed journal, the quiet-point contract, and intact retained sealed release trees._validate_disabled_registry/_validate_quiet_pointnow take an optional post-cutover registry digest and return it; the strictplanpath is unchanged for adoption-time callers. Malformed or missing pins raiseAdoptionErrorrather than crashing, and the standalone adoption CLI prints a bundle-path hint (CLI-only, never embedded in the refusal) on an unknown-live-registry refusal.prepare_bridge_cutover.validate_bundle: adoptionplanrefusals now route through the new_superseded_adoption_phase, which probes the main cutover for full application plus a marked post-install boundary, reconstructs pins from the cutover manifest via_post_cutover_pins, and reports the newruntime-switchedphase. Any other combination re-raises the original refusal with the cutover-side probe diagnosis appended as a suffix; a full apply without the marked boundary still refuses.docs/bridge-cutover-sealed-runtimes.md. Two unrelated harness adjustments ride along: thefm-bootstrapagent-fleet preflight cases opt into the account-routing test lab, and the agent-fleet lease timing bounds widen from 5s to 8s to absorb full-sweep scheduling load. The full bridge python lane (216 tests) and both shell suites pass.Risk Assessment
✅ Low: This round is a tightly scoped relocation of one operator hint from the library validator to the adoption CLI's error handler, restoring the library refusal to its byte-identical pre-hint form and adding two tests that pin the hint's presence on the CLI path and absence from the refusal text, with no change to what the validator accepts.
Testing
Beyond the already-green baseline sweep, I ran the full bridge-cutover python lane (216 tests, OK) plus fm-bootstrap.test.sh and fm-account-routing.test.sh standalone, then wrote a manual operator harness that assembles a genuine post-cutover machine state and drives the real CLIs at both the base commit and this branch. The harness reproduces the exact live 2026-07-19 NO-GO on base tooling and shows it cleared here: the strict adoption plan still refuses identically, validate now reports the pinned
runtime-switchedphase with runtime_switch_ready false, worker-state 6a/6b/6c complete, rollback attributes identity drift instead of dying, and all four pinned negative states still refuse with the cutover probe diagnosis preserved. One harness-only hiccup (base tools staged under an uncanonicalized /var symlink caused a wrong-reason refusal) was fixed and re-run before capturing evidence. Everything passes; no actionable findings.Evidence: Full end-to-end CLI transcript: live NO-GO reproduced on base tooling and cleared on this branch
Evidence: Reproduction harness (builds the post-cutover machine state, drives base vs HEAD CLIs)
Evidence: Before/after: the live 6b NO-GO and the pinned post-cutover acceptance
Evidence: Pinned refusal matrix: every other post-cutover combination still refuses, probe diagnosis preserved
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 2 issues found → auto-fixed (2) ✅
tools/bridge-cutover/bridge_sealed_adoption.py:1486- In post_cutover_plan the per-operation pin loop only rejectspin is None, while the registry pin above it (line 1462) is fully guarded withisinstance(registry_pin, Mapping). A pin value that is present but not a Mapping (e.g. a bare string) flows into_observe_post_cutover_operation, which immediately callspin.get("kind")at line 1415/1433 and raises AttributeError. That is not an AdoptionError, so_superseded_adoption_phase(prepare_bridge_cutover.py:5704, which catches onlyadoption_driver.AdoptionError) does not convert it, and the AttributeError escapesvalidate_bundleuncaught instead of producing a clean refusal. The in-repo caller_post_cutover_pinsalways builds dicts so this is not reachable today, butpost_cutover_planis a module-level API and commit 515cc78 explicitly hardened this function to refuse malformed pins without crashing — the loop was left out of that fix. Apply the sameisinstance(pin, Mapping)guard (or hoist a shared pin validator) so any malformed pin raises AdoptionError.tools/bridge-cutover/bridge_sealed_adoption.py:1597- The pinned post-cutover assessment is only reachable through prepare_bridge_cutover.validate_bundle. The sealed-adoption CLI's default action is still strictplan(main(), line 1597), andpost_cutover_planis not exposed as a subcommand nor referenced from the refusal text. An operator on the switched machine who runs bridge_sealed_adoption.py directly — the natural first move when diagnosing the parked cutover — still gets exactly the 2026-07-19 message,refused: live registry has unknown SHA-256: <digest>, with no indication that a legitimate post-cutover state exists and is accepted elsewhere. Consider adding a--post-cutover-planaction (requiring the cutover manifest to derive pins) or amending the strict refusal to name the runtime-switched phase. Flagging rather than fixing because exposing a second, more permissive assessment on the adoption CLI is a deliberate product/security surface decision.🔧 Fix: guard non-Mapping post-cutover pins; hint bundle path on strict refusal
1 warning still open:
tools/bridge-cutover/bridge_sealed_adoption.py:698- The new operator hint is appended inside the library validator_validate_disabled_registry, gated only onpost_cutover_sha256 is None. That gate suppresses it forpost_cutover_plan, but NOT for the strictplan()call thatvalidate_bundleitself makes (prepare_bridge_cutover.py:5453), which always passes None._superseded_adoption_phasethen embeds that refusal verbatim asinvalid = f"sealed-adoption state is invalid: {refusal}", so a failedprepare_bridge_cutover.py validateon a post-cutover machine now emits: 'sealed-adoption state is invalid: live registry has unknown SHA-256: <d>; if the main cutover has fully applied, validate through the bundle (prepare_bridge_cutover.py validate), which accepts the pinned post-cutover state (post-cutover probe: <real diagnosis>)'. The message advises running the exact command that just failed and claims the pinned state is accepted immediately before the probe explains why it was refused - muddying precisely the diagnosis commit 515cc78 was written to preserve. The same text reaches worker-state 6b/6c operators through_candidate_api'sWorkerStateError(f"cannot load exact candidate API/registry: {exc}")wrapper. Suggested fix: move the hint out of the library validator and append it only at the adoption CLI boundary, inmain()'sexcept (AdoptionError, OSError)printer (~line 1610). That still points the standalone-CLI operator at the working path (the stated goal) while keeping composite messages fromvalidate_bundleand worker-state clean. Marked ask-user rather than auto-fix because the round-1 instruction named_validate_disabled_registryor the plan-level wrapper as the location, andmain()is a third option you did not specify.🔧 Fix: move post-cutover hint from validator to adoption CLI
✅ Re-checked - no issues remain.
🔧 **Test** - 1 issue found → auto-fixed (2) ✅
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"🔧 Fix: no fix needed; account-routing timing flake re-verified green
1 error still open:
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"🔧 Fix: widen agent-fleet lease timing bounds to absorb sweep load
✅ Re-checked - no issues remain.
command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"bash tests/bridge-cutover-python.test.sh— full bridge python lane, 216 tests, OK (skipped=2), including the newtest_validate_accepts_post_cutover_state_only_with_boundary_marked,test_validate_refuses_tampered_live_registry_after_runtime_switch,test_validate_refuses_candidate_registry_before_adoption,test_validate_refuses_partially_reverted_state_after_runtime_switch,test_post_cutover_plan_refuses_registry_pin_without_sha256,test_verify_and_finalize_pass_after_runtime_switch,test_rollback_attributes_identity_drift_after_runtime_switchbash tests/fm-bootstrap.test.sh— 23/23 ok, exit 0 (confirms the settled test-lab opt-in for the agent-fleet preflight cases)bash tests/fm-account-routing.test.sh— standalone, green includingok - Agent Fleet lease mutations are bounded and ambiguous outcomes retain ownership state(the widened 8s bound from 22d0711)Manual end-to-end operator reproduction:python3.11 live_nogo_repro.py— builds a real post-cutover machine state (adoption applied,driver.execute(..., "forward"),mark_post_install_irreversible_boundary), then runs base-commit and HEAD CLIs as subprocesses against the same on-disk statepython3.11 <base>/bridge_sealed_adoption.py <adoption manifest>andpython3.11 <base>/prepare_bridge_cutover.py --driver ... validate <bundle>at base commit 6b67114 — reproduce the literal live NO-GO refusal (exit 2)python3.11 tools/bridge-cutover/prepare_bridge_cutover.py --driver tools/bridge-cutover/bridge_cutover_transaction.py validate <bundle>at HEAD — exit 0,cutover_phase: runtime-switched,runtime_switch_ready: falsepython3.11 tools/bridge-cutover/bridge_worker_state_transaction.py <worker-state manifest> --begin | --verify-provisioned | --finalizeat HEAD on the switched machine — exit 0 for all three (6a/6b/6c)python3.11 tools/bridge-cutover/bridge_worker_state_transaction.py <worker-state manifest> --rollbackwith a drifted codex identity bundle — exit 0,phase: rolled_back, snapshot restoredPinned-refusal matrix viavalidate: tampered post-cutover registry, full apply without the marked boundary, partially reverted current-link, and candidate registry before adoption — all exit 2 with the post-cutover probe diagnosis preservedgit diff <base>..<head> | grep -E '/(Users|home|opt|usr|var)/[^ ]*(agent-fleet|releases)'— no absolute agent-fleet releases paths introduced (intent constraint)docs/bridge-cutover-sealed-runtimes.md:173- The doc now names theruntime-switchedphase but does not state thatruntime_switch_readystays false in it (prepare_bridge_cutover.py:5527). I deliberately did not add this: the guide documents cutover semantics, not validate_bundle's output fields, and no existing doc owns that field — adding it would open a new surface rather than fix a stale fact. Flagging in case you want the flag's contract pinned in prose anyway.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.