fix(crosscheck): recover transient Azure compartment failures - #331
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Harden operational Azure Crosscheck against the two observed transient infrastructure failures without weakening its security or review semantics. Add a bounded apt/dpkg lock wait for normal unattended-upgrade races, one bounded structured-verdict repair for zero, multiple, or malformed Pi verdict calls that still fails closed after the repair, and internally consistent failed-run telemetry that omits compartment-only phases when no completed Azure identity can bind them. Preserve exact-head binding, isolated Azure model/tool/verifier compartments, credentialless evidence verification, cleanup guarantees, and fail-closed behavior. Add executable regressions for both failure modes, keep the change lightweight, do not use Bugbot, and release promptly so consumers can retry PR kunchenguid#1514. The obsolete kunchenguid#1513 verifier azr-080976557725 already completed through the official runner with exit 0 and no manual cloud mutation.
What Changed
Risk Assessment
✅ Low: The fixes close both previously identified retry-state and apt lock-race paths while preserving bounded retries, exact-head review isolation, telemetry consistency, and fail-closed behavior.
Testing
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 2 issues found → auto-fixed ✅
bin/fm-azure-runner-guest.sh:118- The required bounded wait still has a check/use race:wait_for_apt_locksreleases every apt lock beforeapt-getstarts, andDPkg::Lock::Timeoutonly covers dpkg locking. An unattended-upgrade process can acquire/var/lib/apt/lists/lockor/var/cache/apt/archives/lockbetween lines 118 and 119, causing the same transient bootstrap failure. The intent requires hardening “normal unattended-upgrade races”; retry lock-specific apt failures within the same deadline at the apt invocation boundary.bin/fm-crosscheck-pi-reviewer.py:218-verdict_protocol_errorsurvives an acceptedauto_retry_starteven thoughcallsand other per-attempt state are cleared. A failed internal Pi attempt containing an invalid verdict call followed by Pi's valid retry is therefore treated as malformed and consumes the one external repair; if that repair fails, a valid final verdict is refused. Clear this flag alongsidecallsat the retry boundary.🔧 Fix: Retry apt lock races and reset Pi retry state
✅ Re-checked - no issues remain.
⏭️ **Test** - skipped
Step was skipped.
✅ **Document** - passed
✅ No issues found.
⏭️ **Lint** - skipped
Step was skipped.
✅ **Push** - passed
✅ No issues found.