Skip to content

fix(crosscheck): recover transient Azure compartment failures - #331

Merged
ruby-dlee merged 4 commits into
mainfrom
codex/crosscheck-compartment-retry
Aug 25, 2026
Merged

fix(crosscheck): recover transient Azure compartment failures#331
ruby-dlee merged 4 commits into
mainfrom
codex/crosscheck-compartment-retry

Conversation

@ruby-dlee

Copy link
Copy Markdown
Owner

Intent

Harden operational Azure Crosscheck against the two observed transient infrastructure failures without weakening its security or review semantics. Add a bounded apt/dpkg lock wait for normal unattended-upgrade races, one bounded structured-verdict repair for zero, multiple, or malformed Pi verdict calls that still fails closed after the repair, and internally consistent failed-run telemetry that omits compartment-only phases when no completed Azure identity can bind them. Preserve exact-head binding, isolated Azure model/tool/verifier compartments, credentialless evidence verification, cleanup guarantees, and fail-closed behavior. Add executable regressions for both failure modes, keep the change lightweight, do not use Bugbot, and release promptly so consumers can retry PR kunchenguid#1514. The obsolete kunchenguid#1513 verifier azr-080976557725 already completed through the official runner with exit 0 and no manual cloud mutation.

What Changed

  • Add a bounded three-minute apt/dpkg lock wait and retry path to Azure guest bootstrap package operations.
  • Retry zero, multiple, or malformed Pi verdict submissions once in the same isolated reviewer compartment, aggregate both attempts’ telemetry, and fail closed if repair is exhausted.
  • Preserve compatible failed-run timing telemetry while omitting Azure-only phases that lack a completed compartment identity, with executable regressions and updated operational documentation.

Risk Assessment

✅ Low: The fixes close both previously identified retry-state and apt lock-race paths while preserving bounded retries, exact-head review isolation, telemetry consistency, and fail-closed behavior.

Testing

  • ⏭️ Test - skipped

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 2 issues found → auto-fixed ✅
  • 🚨 bin/fm-azure-runner-guest.sh:118 - The required bounded wait still has a check/use race: wait_for_apt_locks releases every apt lock before apt-get starts, and DPkg::Lock::Timeout only covers dpkg locking. An unattended-upgrade process can acquire /var/lib/apt/lists/lock or /var/cache/apt/archives/lock between lines 118 and 119, causing the same transient bootstrap failure. The intent requires hardening “normal unattended-upgrade races”; retry lock-specific apt failures within the same deadline at the apt invocation boundary.
  • ⚠️ bin/fm-crosscheck-pi-reviewer.py:218 - verdict_protocol_error survives an accepted auto_retry_start even though calls and other per-attempt state are cleared. A failed internal Pi attempt containing an invalid verdict call followed by Pi's valid retry is therefore treated as malformed and consumes the one external repair; if that repair fails, a valid final verdict is refused. Clear this flag alongside calls at the retry boundary.

🔧 Fix: Retry apt lock races and reset Pi retry state
✅ Re-checked - no issues remain.

⏭️ **Test** - skipped

Step was skipped.

✅ **Document** - passed

✅ No issues found.

⏭️ **Lint** - skipped

Step was skipped.

✅ **Push** - passed

✅ No issues found.

@ruby-dlee
ruby-dlee merged commit 5118dbe into main Aug 25, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant