fix(validator): persist last sealed bundle across restarts

[echobt]

Summary

• Validators persist the last independently verified sealed bundle on disk (/var/lib/base/last-sealed.bundle) so a master/gateway restart that serves burn (sealed: false) still Matches and submits on-chain weights.
• Fetch/verify of a sealed latest still does not fall back to that file.
• Compose volume base-validator-lkg keeps the file across container replace (validator host only).

Test plan

• cargo test -p validator --lib epoch_loop
• local pre-push cargo test --workspace --lib
• After merge: CI green, staging pins, tag v3.3.26, deploy-prod
• Prod validator logs validator_lkg_saved after first sealed Match, then validator_match on later gateway burn

Summary by CodeRabbit

• New Features
  • Validators now persist the last verified sealed bundle for reliable recovery.
  • During an unsealed burn, validators can re-compare and submit the persisted sealed bundle.
  • Added configurable storage with environment-based or custom paths, including disabled mode.
• Deployment
  • Added persistent storage for validator recovery data across local, staging, and production environments.
• Documentation
  • Clarified sealed-bundle fallback behavior and updated completeness guidance.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: efb63db5-8d41-4bbd-8748-412d55f53dec

📥 Commits

Reviewing files that changed from the base of the PR and between e7d9667 and 1def15c.

📒 Files selected for processing (11)

• crates/validator-verify/src/recompute.rs
• crates/validator/src/epoch_loop.rs
• crates/validator/src/lib.rs
• crates/validator/src/lkg.rs
• deploy/compose/env-local.yml
• deploy/compose/env-prod.yml
• deploy/compose/env-staging.yml
• deploy/compose/role-validator.yml
• docker-compose.yml
• docs/BUNDLE_SPEC.md
• docs/COMPLETENESS.md

---

📝 Walkthrough

Walkthrough

The validator now persists verified sealed bundles and reloads them during unsealed gateway burns. The coordination loop centralizes comparison outcomes and submission. Container deployments mount persistent storage, and public APIs and specifications document the behavior.

Changes

Sealed bundle LKG fallback

Layer / File(s)	Summary
LKG storage and persistence wiring crates/validator/src/lkg.rs, crates/validator/src/lib.rs, deploy/compose/*.yml, docker-compose.yml	Added SealedBundleLkg with configurable paths, atomic save/load behavior, tests, public exports, and persistent volume mounts.
Coordination and burn fallback crates/validator/src/epoch_loop.rs, crates/validator-verify/src/recompute.rs	The coordination loop saves verified sealed bundles, reloads them during unsealed burns, maps fetch errors to NoSubmission, and submits matching outcomes. Tests cover persistence across restart.
Specification and completeness updates docs/BUNDLE_SPEC.md, docs/COMPLETENESS.md	Documented persisted sealed-bundle use during gateway burns and the BASE_VALIDATOR_LKG_PATH setting.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Gateway
  participant CoordinationLoop
  participant SealedBundleLkg
  participant Submitter
  Gateway->>CoordinationLoop: Return sealed or unsealed latest bundle
  CoordinationLoop->>SealedBundleLkg: Save verified sealed bundle
  CoordinationLoop->>SealedBundleLkg: Load persisted bundle during unsealed burn
  CoordinationLoop->>Submitter: Submit matching outcome

Loading

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/validator-lkg-set-weights

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}