test(ika-upgrade-test): out-of-process cross-binary upgrade harness

CLAUDE.md

@@ -6,6 +6,10 @@ Decentralized MPC signing network built on Sui. dWallets provide zero-trust mult
 
 Act as a critical intellectual sparring partner, not a yes-man. Evaluate every idea on its merits—the user is a collaborator who can be wrong, not an authority to defer to. Question assumptions, point out flaws, logical errors, unstated premises, and potential bugs immediately and directly. Be skeptical by default; each claim must prove itself. No opening praise or "you're right" unless genuinely warranted after scrutiny. Prioritize truth over harmony. Be ruthless with constructive criticism.
 
+## No Time Estimates
+
+NEVER estimate time, effort, or duration for any work — no "weeks", "days", "hours", "engineer-quarter", "this is quick", "this will take a while", no calendar/sprint-based sequencing. You have no reliable concept of wall-clock effort, and time estimates produce bad downstream planning decisions. Express plans as **ordering and dependencies only** ("first / then / after X / in parallel", "P0 before P1"), never as durations. Applies to plans, design docs, PR descriptions, and chat. If asked "how long will this take", say you don't estimate time and give the dependency-ordered steps instead.
+
 ## Build Commands
 
 ```bash
@@ -246,7 +250,7 @@ Other gotchas:
 
 - **Release mode required**: Crypto operations are extremely slow in debug mode
 - **Forked from Sui**: Much code structure mirrors Sui Network patterns
-- **Sui dependency pinned**: Uses `mainnet-v1.51.5` tag for all Sui dependencies
+- **Sui dependency pinned**: Uses `mainnet-v1.70.2` tag for all Sui dependencies
 - **WASM excluded**: `sdk/ika-wasm` is excluded from workspace (separate build)
 - **Mysticeti consensus**: Uses Sui's Mysticeti for MPC message routing
 - **NOA checkpoints not live**: The NOA checkpoint system (`crates/ika-core/src/noa_checkpoints/`) is under active development and not yet deployed. No backward compatibility constraints on serialization formats or type names

Cargo.toml

@@ -25,6 +25,7 @@ members = [
     "crates/ika-network",
     "crates/ika-archival",
     "crates/ika-test-cluster",
+    "crates/ika-upgrade-test",
 ]
 [workspace.package]
 # This version string will be inherited by ika-core, ika-node, ika-tools, ika-sdk, and ika crates.
@@ -343,6 +344,7 @@ ika-move-contracts = { path = "crates/ika-move-contracts" }
 ika-network = { path = "crates/ika-network" }
 ika-archival = { path = "crates/ika-archival" }
 ika-test-cluster = { path = "crates/ika-test-cluster" }
+ika-upgrade-test = { path = "crates/ika-upgrade-test" }
 
 [patch.crates-io]
 crypto-bigint = { git = "https://github.com/ycscaly/crypto-bigint.git", rev = "8aabcee553b25f9f619722391de33edbe2f7412c" }

PR-1721-action-plan.md

@@ -0,0 +1,44 @@
+# PR #1721 — Review Action Plan
+
+Combined from: `docs/off-chain-metadata-v2-review.md` (feature walkthrough),
+the GitHub PR #1721 review (`ycscaly` — naming), `PR-1721-review.md` (Cursor),
+and `pr_1721_code_review.md`. Decisions agreed with the user.
+
+Branch `feat/off-chain-metadata-v2`. Both Cursor reviews are already
+merge-ready/Approve — everything below is polish/follow-up, not a blocker.
+
+---
+
+## ✅ Will do — in order
+
+| # | Item | Why | Notes | Status |
+|---|------|-----|-------|--------|
+| 1 | **Naming: `class_groups` → `mpc_data` / `ValidatorMpcData`** on the assembly path | The bundle is class-groups **+ per-curve PVSS keys + proofs** since #1707; the name lies. `ValidatorMpcData` is already the convention elsewhere. | Source-only (BCS is positional → no wire-shape impact). Sites: `install_mpc_data_source` (`sui_connector/mod.rs:181`), `OffChainCommitteeClassGroupsSource` trait, assembly-path sites. Follow-up sites (out of diff): `MPCDataV1.class_groups_public_key_and_proof` field + `VersionedMPCData::class_groups_public_key_and_proof()` accessor. **Do NOT** rename `Committee.class_groups_public_keys_and_proofs` (genuinely class-groups, beside `*_pvss_*`). | ✅ `<naming>` |
+| 2 | **Fix stale "consensus-voted" comments** in `mpc_manager` / `dwallet_mpc_service` | Comments describe the vote path that was removed in the unification. Misleads the next reader. | Trivial; my own debt. | ✅ `<comments>` |
+| 3 | **EOP: reject the EOP vote when the bundled handoff sig *verifiably* fails** | Makes the `EndOfPublishV2` bundle atomic ("observed together" ⇒ "processed together"). Safe now that `AttestationMismatch` ≈ 0. | **Nuance:** only when the sig *verifies-and-fails* (`AttestationMismatch`). While the sig is *buffered* (expected attestation not installed yet, can't verify), still count the vote — else epoch advance stalls. | ✅ `<eop-atomic>` |
+| 4 | **Fail-closed bootstrap on `Rejected`** | `Rejected` = every reachable peer served a wrong cert = possible eclipse / wrong prior-committee view. Halt loudly instead of limping. | The unification already half-does this (no cert ⇒ no key ⇒ can't really operate); this adds the explicit halt + actionable alert. | ✅ `<fail-closed>` |
+| 5 | **F6: escalate when off-chain assembly never converges** | Exactly the "assembly incomplete" we kept hitting — today it spins forever at `warn!` with no `error!`/metric. | Surface `EverythingExcluded` / permanent-incompleteness as `error!` + metric; keep transient (waiting-for-P2P) as `warn!`. | ✅ `<assembly-escalate>` (metric = follow-up) |
+| 6 | **F7: resolve departed prior-committee signers' pubkeys** | Under churn, a *valid* cert is `Rejected` on a joiner because it can't resolve the keys of signers who left after E-1. | Three layers: **(A, primary)** bootstrap chain-reads `validator_set.previous_committee` by object id (StakingPool persists after a validator leaves the active set) and merges it with the current active set into the verify provider — resolves every departed signer whose pool still exists; **(B, slack)** the handoff aggregator now collects *past* quorum (up to full committee), enriching the cert so a signer fully gone (StakingPool deleted) can be dropped while a quorum of the rest verifies; **(skip)** `verify_certified_handoff_attestation` skips an unresolvable signer instead of hard-failing. No P2P sig-sync needed — sigs are consensus-ordered and a joiner verifies any fetched cert independently. | ✅ `<f7-departed-signers>` |
+| 7 | **F5: epoch-consistency check in `refresh()`** | 2-line belt-and-suspenders: stops a lagging prev-epoch pubkey updater from installing the *next* committee's keys onto the live store. | `if system_inner.epoch != self.epoch_id { return Ok(()); }`. | ✅ `<f5-epoch-guard>` |
+| 8 | **F3-5: receiver-side relay buffer** | Closes the consensus-delivery race the joiner-retry can't: a validator whose `JoinerPubkeyProvider` lagged drops the relayed joiner announcement, and consensus dedup means it never re-sees it. Under load the window widens and a dropped joiner can diverge the next-committee assembly. | Buffer (bounded size + TTL) joiner announcements with a currently-absent/lagging provider; re-evaluate on provider install. Buffer on **no provider** or **`UnregisteredJoiner`**; drop genuinely-bad (`InvalidSignature`/`InconsistentEnvelope`). Can't bound by next-epoch membership (the provider that knows it is what's missing), so bounded by a hard cap + TTL + last-write-wins per joiner. Pure buffer/re-eval helpers unit-tested. | ✅ `<f3-5-relay-buffer>` |
+
+---
+
+## ❌ Won't do
+
+| Item | Why |
+|------|-----|
+| **BLS aggregate handoff cert** (docs F3-4) | Big rewrite of a working, well-tested Ed25519 path for a size/speed win that isn't hurting us. Risk > reward. |
+| **F4-1 deadline excludes slow joiners** | By design — the liveness backstop so one dead joiner can't wedge the epoch. Already logged. Correct trade-off. |
+
+---
+
+## ⏭️ Follow-up (after this plan)
+
+| Item | Why deferred |
+|------|--------------|
+| **F5/F6 nits** — refresh loop spins forever on dropped epoch store; `from_iter` silent overwrite on duplicate `AuthorityName`; base64 dedup cleanup; no RPC backoff; `CommitteeMembership` type for the chain channel; incomplete empty-blob entry publish | Each trivial + low-impact; batch later. |
+| **Churn green on CI** | Behaviors verified by 5 targeted tests (incl. `test_user_sessions_across_multiple_epochs`, a multi-reconfig mini-churn under load); CI just captures the full 10-cycle stress run this box can't sustain. |
+| **Restart-replay integration test** | Replay re-verify logic is already in + unit-tested; a dedicated integration test is nice-to-have. |
+| ~~**F7 deep-history catch-up**~~ — **closed (no code).** Analysis: a multi-epoch cert-chain walker isn't a real path. The prior committee's trust root is Sui (chain `previous_committee` + `committee_store`), not an older handoff cert, so a joiner anchors one hop on the chain-provided recent committee. Documented the why in `verify_joiner_bootstrap_cert`'s one-hop note. The only residual gap (a prior signer whose StakingPool was deleted) is single-hop, handled by the slack + skip layers. |
+| **Final review together — part by part** | On the *last* version of the PR, walk the whole thing with the user section by section as a final pass (replaces the F9–F13 solo walkthrough). **Last item.** |

crates/ika-core/src/authority.rs

@@ -853,6 +853,14 @@ impl AuthorityState {
         self.epoch_store.load()
     }
 
+    /// Returns the shared `AuthorityPerpetualTables` handle. Used by
+    /// producer-side broadcasters (e.g. mpc_data announcement) to
+    /// persist content-addressed blobs so peers can fetch them by
+    /// digest over the existing `GetMpcDataBlob` RPC.
+    pub fn perpetual_tables(&self) -> Arc<AuthorityPerpetualTables> {
+        self.perpetual_tables.clone()
+    }
+
     // Load the epoch store, should be used in tests only.
     pub fn epoch_store_for_testing(&self) -> Guard<Arc<AuthorityPerEpochStore>> {
         self.load_epoch_store_one_call_per_task()
@@ -1048,6 +1056,17 @@ impl AuthorityState {
             epoch_start_configuration,
             cur_epoch_store.get_chain_identifier(),
         )?;
+        // The new epoch store starts with `perpetual_tables_for_handoff`
+        // empty. Install ours so the per-epoch handoff record path
+        // persists freshly certified attestations into perpetual
+        // storage from this epoch onward (mirrors what
+        // `IkaNode::new` does for the genesis epoch store). Without
+        // this, every reconfig after the first drops handoff certs
+        // silently — the cert insert site logs "perpetual tables
+        // not installed; handoff cert not persisted" and joiners
+        // never see the cert that authenticated their place in the
+        // committee.
+        new_epoch_store.install_perpetual_tables_for_handoff(self.perpetual_tables.clone());
         self.epoch_store.store(new_epoch_store.clone());
         Ok(new_epoch_store)
     }