Skip to content

feat(bridge-cutover): refresh runtime-switched bundle identity in place - #15

Merged
ruby-dlee merged 5 commits into
mainfrom
fm/bridge-prepare-inplace-refresh
Jul 22, 2026
Merged

feat(bridge-cutover): refresh runtime-switched bundle identity in place#15
ruby-dlee merged 5 commits into
mainfrom
fm/bridge-prepare-inplace-refresh

Conversation

@ruby-dlee

@ruby-dlee ruby-dlee commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Intent

The developer wanted an automated regression check in the firstmate repository to prevent the synthetic agent-fleet dataclass fixture from drifting from the real Registry, Settings, ProviderConfig, and Profile schemas, while keeping real-package imports out of synthetic pipeline execution paths. They required isolated worktree development, default-branch protection, and shipment through the no-mistakes pipeline. They then requested a worker-state cutover fix so validation accepts the exact fully applied post-cutover runtime state without weakening strict sealed-adoption checks, with pinned refusal tests for tampering and partial states, no provider logins, and tilde-form paths in process command lines. After merging PR #9, they asked for a self-contained handoff document in Downloads so Firstmate could finish the cutover, including the required 6b → browser-free identity adoption → 6c ordering and stopping before step 8.

What Changed

  • Add a refresh command that updates an applied runtime-switched bundle’s activation plan and worker-state manifest to the current provider identity while preserving sealed journals and strict validation.
  • Serialize refreshes with worker-state operations, refuse active transaction artifacts, atomically replace identity files, recover partial replacements on retry, and reload manifests after lock acquisition.
  • Document the refresh preconditions and recovery behavior, with regression coverage for eligibility, refusal paths, identity enforcement, partial updates, and locking.

Risk Assessment

✅ Low: Captain, the updated implementation now serializes against worker-state begin, refuses every existing rollback anchor before mutation, and remains retryable after an interrupted artifact pair update.

Testing

The configured full baseline had already succeeded; focused regressions and disposable end-to-end CLI, concurrency, recovery, and refusal checks also passed, with three reviewer-visible transcripts captured and no worktree residue.

Evidence: Bridge refresh CLI transcript 

### Applied bundle validates before reinstall identity drift
$ python3 tools/bridge-cutover/prepare_bridge_cutover.py --driver tools/bridge-cutover/bridge_cutover_transaction.py validate /private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/output/bundle/bundle.json
exit=0
{
  "adoption_manifest_fingerprint": "bcff86b5cc70b40146c2a1a8094f98ff1edc56f9da48b1216d84c5d3a256d19e",
  "agent_fleet_contract_version": 2,
  "agent_fleet_version": "0.2.0",
  "bundle": "/private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/output/bundle/bundle.json",
  "cutover_phase": "runtime-switched",
  "cutover_ready": false,
  "enabled": 0,
  "external_reserves": 2,
  "fleet_managed_workers": 6,
  "initial_registry_sha256": "fae41cb4a2227453332c68a03294c08b3c5291e751f3bc4e1f56fdc00325f799",
  "manifest_fingerprint": "c1a7bbc152d8556316b8511fe93894fce8d3cee1ddffac745410f69566387f22",
  "new_registry_sha256": "17b43a9aacf8dca74f9bc5f647b82c847e47b5691b5b4c74cd016c427b4b9016",
  "old_registry_sha256": "f165db8c0f420d5fabe58797d597a04865d4ef253203137ad551a99e97ef3661",
  "profiles": 8,
  "projects_per_provider": 1,
  "quota_binary": "/private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/quota-axi/releases/0.1.7-new/bin/quota-axi",
  "quota_node_binary": "/private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/quota-axi/releases/0.1.7-new/runtime/node",
  "rollback_contract_version": 1,
  "rollback_version": "0.1.5",
  "runtime_switch_ready": false,
  "transaction_id": "bridge-cutover-fixture",
  "valid": true,
  "worker_state_phase": "not-started",
  "worker_state_ready": false
}

### Strict validation refuses stale provider identity
$ python3 tools/bridge-cutover/prepare_bridge_cutover.py --driver tools/bridge-cutover/bridge_cutover_transaction.py validate /private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/output/bundle/bundle.json
exit=2
refused: bundle activation plan is not exact or generated-command-free

### Operator refreshes the applied bundle in place
$ python3 tools/bridge-cutover/prepare_bridge_cutover.py --driver tools/bridge-cutover/bridge_cutover_transaction.py refresh /private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/output/bundle/bundle.json
exit=0
{
  "adoption_manifest_fingerprint": "bcff86b5cc70b40146c2a1a8094f98ff1edc56f9da48b1216d84c5d3a256d19e",
  "agent_fleet_contract_version": 2,
  "agent_fleet_version": "0.2.0",
  "bundle": "/private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/output/bundle/bundle.json",
  "cutover_phase": "runtime-switched",
  "cutover_ready": false,
  "enabled": 0,
  "external_reserves": 2,
  "fleet_managed_workers": 6,
  "initial_registry_sha256": "fae41cb4a2227453332c68a03294c08b3c5291e751f3bc4e1f56fdc00325f799",
  "manifest_fingerprint": "c1a7bbc152d8556316b8511fe93894fce8d3cee1ddffac745410f69566387f22",
  "new_registry_sha256": "17b43a9aacf8dca74f9bc5f647b82c847e47b5691b5b4c74cd016c427b4b9016",
  "old_registry_sha256": "f165db8c0f420d5fabe58797d597a04865d4ef253203137ad551a99e97ef3661",
  "profiles": 8,
  "projects_per_provider": 1,
  "quota_binary": "/private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/quota-axi/releases/0.1.7-new/bin/quota-axi",
  "quota_node_binary": "/private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/quota-axi/releases/0.1.7-new/runtime/node",
  "refreshed": true,
  "rollback_contract_version": 1,
  "rollback_version": "0.1.5",
  "runtime_switch_ready": false,
  "transaction_id": "bridge-cutover-fixture",
  "valid": true,
  "worker_state_phase": "not-started",
  "worker_state_ready": false
}

### Fresh strict validation succeeds after refresh
$ python3 tools/bridge-cutover/prepare_bridge_cutover.py --driver tools/bridge-cutover/bridge_cutover_transaction.py validate /private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/output/bundle/bundle.json
exit=0
{
  "adoption_manifest_fingerprint": "bcff86b5cc70b40146c2a1a8094f98ff1edc56f9da48b1216d84c5d3a256d19e",
  "agent_fleet_contract_version": 2,
  "agent_fleet_version": "0.2.0",
  "bundle": "/private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/output/bundle/bundle.json",
  "cutover_phase": "runtime-switched",
  "cutover_ready": false,
  "enabled": 0,
  "external_reserves": 2,
  "fleet_managed_workers": 6,
  "initial_registry_sha256": "fae41cb4a2227453332c68a03294c08b3c5291e751f3bc4e1f56fdc00325f799",
  "manifest_fingerprint": "c1a7bbc152d8556316b8511fe93894fce8d3cee1ddffac745410f69566387f22",
  "new_registry_sha256": "17b43a9aacf8dca74f9bc5f647b82c847e47b5691b5b4c74cd016c427b4b9016",
  "old_registry_sha256": "f165db8c0f420d5fabe58797d597a04865d4ef253203137ad551a99e97ef3661",
  "profiles": 8,
  "projects_per_provider": 1,
  "quota_binary": "/private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/quota-axi/releases/0.1.7-new/bin/quota-axi",
  "quota_node_binary": "/private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/quota-axi/releases/0.1.7-new/runtime/node",
  "rollback_contract_version": 1,
  "rollback_version": "0.1.5",
  "runtime_switch_ready": false,
  "transaction_id": "bridge-cutover-fixture",
  "valid": true,
  "worker_state_phase": "not-started",
  "worker_state_ready": false
}

### Preserved proofs and exact regenerated artifacts
{
  "bundle_restored_to_exact_reconstruction": true,
  "cutover_phase": "runtime-switched",
  "sealed_transaction_journals_byte_identical": true,
  "worker_state_restored_to_exact_reconstruction": true
}

### Refresh refuses an active worker-state transaction
$ python3 tools/bridge-cutover/prepare_bridge_cutover.py --driver tools/bridge-cutover/bridge_cutover_transaction.py refresh /private/var/folders/y_/bfdbj_vx20l9b9tw7crgkzwm0000gn/T/bridge-cutover-prep-test-59cqn72d/fixture/output/bundle/bundle.json
exit=2
refused: in-place refresh refuses while a worker-state transaction is bound to the current manifest fingerprint; refreshing would change the fingerprint and strand the transaction and its rollback; bring worker-state back to 'not-started' before refreshing
artifacts_unchanged_after_refusal=true
Evidence: Concurrent worker-lock handoff transcript 
### Concurrent refresh-to-worker handoff
{
  "journal_bound_to_refreshed_manifest": true,
  "journal_manifest_fingerprint": "9e43409bf1def517a3603d4c34c1fc7e9aa398b733ad6d4cbf3c5070a7ffe4ec",
  "journal_not_bound_to_stale_manifest": true,
  "refreshed_manifest_fingerprint": "9e43409bf1def517a3603d4c34c1fc7e9aa398b733ad6d4cbf3c5070a7ffe4ec",
  "stale_manifest_fingerprint": "ad984631cc56707ed0f7ab62a578386182801a486a855a7c2bfed5dca4a6eace",
  "worker_begin_blocked_while_refresh_lock_held": true,
  "worker_state_phase": "snapshotted"
}
Evidence: Bound worker-state refusal matrix 
### Bound worker-state refusal matrix
{
  "journal": {
    "bundle_and_manifest_unchanged": true,
    "exit": 2,
    "refusal_mentions_bound_transaction": true
  },
  "snapshot": {
    "bundle_and_manifest_unchanged": true,
    "exit": 2,
    "refusal_mentions_bound_transaction": true
  },
  "snapshot_staging": {
    "bundle_and_manifest_unchanged": true,
    "exit": 2,
    "refusal_mentions_bound_transaction": true
  }
}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed (2) ✅
  • 🚨 tools/bridge-cutover/prepare_bridge_cutover.py:5733 - Replacing the worker-state manifest changes its fingerprint without acquiring the worker-state lock or migrating its snapshot/journal. Any transaction already at snapshotted or later will then fail validation and rollback because both records remain bound to the old fingerprint, contradicting the documented restart behavior and potentially stranding rollback. Lock the worker-state transaction and atomically rebind/reset its recoverable state, or refuse before modifying either artifact.

🔧 Fix: Captain, refuse refresh during active worker-state transactions
2 errors still open:

  • 🚨 tools/bridge-cutover/prepare_bridge_cutover.py:5746 - The two artifact replacements are not recoverable as a unit. If the process exits after replacing bundle.json but before replacing the worker-state manifest, the old manifest's bundle_sha256 no longer matches; the next refresh now fails in load_manifest() before it can repair the partial update. Add a durable refresh journal or exact partial-state recovery path so retry completes this crash window safely.
  • 🚨 tools/bridge-cutover/prepare_bridge_cutover.py:5716 - The new phase check remains a TOCTOU guard because the preparation lock is unrelated to the worker-state lock. A worker begin() can load the old manifest, start after this reports not-started, and publish an old-fingerprint snapshot/journal while refresh replaces the manifest, recreating the stranded rollback state. Preventing this requires cross-process exclusion with worker-state begin; please clarify whether acquiring the existing worker-state lock solely for serialization is acceptable despite the no-locking constraint.

🔧 Fix: Captain, serialize and recover in-place worker-state refresh
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"
  • Harness-provided baseline: command -v tmux >/dev/null || { echo "tmux is required for e2e tests" >&2; exit 1; }; tmux -V; rc=0; for t in tests/*.test.sh; do echo "== $t =="; bash "$t" || rc=1; done; uv run --directory tools/agent-fleet --locked pytest || rc=1; uv run --directory tools/agent-fleet --locked python -m compileall -q src || rc=1; exit "$rc"
  • PYTHONDONTWRITEBYTECODE=1 python3 -m unittest tests.test_prepare_bridge_cutover.PrepareBridgeCutoverTests.test_refresh_updates_identity_and_revalidates_runtime_switched tests.test_prepare_bridge_cutover.PrepareBridgeCutoverTests.test_refresh_does_not_loosen_the_strict_identity_gate tests.test_prepare_bridge_cutover.PrepareBridgeCutoverTests.test_refresh_refuses_bundle_that_is_not_runtime_switched tests.test_prepare_bridge_cutover.PrepareBridgeCutoverTests.test_refresh_refuses_runtime_switch_ready_bundle tests.test_prepare_bridge_cutover.PrepareBridgeCutoverTests.test_refresh_refuses_when_boundary_not_marked tests.test_prepare_bridge_cutover.PrepareBridgeCutoverTests.test_refresh_refuses_active_worker_state_transaction tests.test_prepare_bridge_cutover.PrepareBridgeCutoverTests.test_refresh_repairs_partial_artifact_replacement tests.test_bridge_worker_state_transaction.WorkerStateTransactionTests.test_begin_reloads_manifest_after_lock_acquisition
  • Disposable end-to-end CLI sequence using prepare_bridge_cutover.py validate, refresh, and validate around simulated provider-identity drift; verified refreshed: true, valid: true, cutover_phase: runtime-switched, byte-identical sealed journals, and unchanged artifacts after active-transaction refusal.
  • Controlled concurrent lock handoff using PYTHONDONTWRITEBYTECODE=1 python3 -; verified worker startup blocked on the refresh lock and journaled the refreshed manifest fingerprint rather than its stale pre-lock fingerprint.
  • CLI refusal matrix for worker-state journal, snapshot, and snapshot-staging artifacts; each returned exit 2 and left the bundle and worker manifest unchanged.
  • git status --short --untracked-files=all confirmed testing left the worktree clean.
🔧 **Document** - 1 issue found → auto-fixed ✅
  • ⚠️ tools/bridge-cutover/prepare_bridge_cutover.py:5735 - The refusal message says to return worker-state to not-started, but the driver preserves its journal and provides no reset action. Correcting this user-facing message requires executable/test changes outside the documentation-only scope.

🔧 Fix: Clarify bridge refresh transaction precondition
✅ Re-checked - no issues remain.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

ruby-dlee added 5 commits July 21, 2026 19:25
@ruby-dlee
feat(bridge-cutover): refresh applied runtime-switched bundle identit…
5de199d 
…y in place

A byte-exact reinstall of the provider binary gives it a fresh inode/mtime and
re-stales the recorded activation plan and worker-state manifest against the
live binary, so validate_bundle's strict identity gates refuse even though the
machine is genuinely runtime-switched. A from-scratch rebuild cannot recover:
the runtime-switched proof is journal-bound to the applying bundle's own
transaction journals, which a rebuild's empty output_dir can neither borrow
byte-for-byte nor re-seal against the already-migrated live registry.

Add a `refresh` preparer mode that regenerates exactly those two identity
artifacts of the existing bundle in place, atomically, from the current
provider-binary identity while leaving the sealed cutover and adoption journals
untouched. It refuses unless the bundle is provably the applied runtime-switched
one (reusing the same journal-bound phase determination as validation) and gates
completion on a full strict validate_bundle that must still report
runtime-switched, so the strict gates pass only because the recorded identity
again equals the live binary - no proof is loosened.

Factor the shared bundle reconstruction into _reconstruct_bundle_state so the
strict validator and the refresh share one owner for every integrity check up to
the two identity gates. Tests prove the refresh realigns identity and revalidates
runtime-switched with journals preserved, that it does not loosen the strict gate,
and that it refuses a not-yet-applied or unmarked-boundary bundle without mutating it.
@ruby-dlee
no-mistakes(review): Captain, refuse refresh during active worker-sta…
f52911f 
…te transactions
@ruby-dlee
no-mistakes(review): Captain, serialize and recover in-place worker-s…
f24d258 
…tate refresh
@ruby-dlee
no-mistakes(document): Clarify bridge in-place refresh documentation
3eca359
@ruby-dlee
no-mistakes(document): Clarify bridge refresh transaction precondition
7bef99c
@ruby-dlee
ruby-dlee merged commit bf7c5ad into main Jul 22, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ruby-dlee