fix(prism): harden Lium install/429 and allow infra resubmit

[echobt]

Summary

• Split Lium pip install into its own SSH session and re-verify imports after host drops, so post-install disconnects no longer burn the whole eval.
• Back off and retry Lium HTTP 429s (and pause between rent candidates) instead of instantly exhausting the infra retry budget.
• After infra ChallengeInternal / blocked, miners can resubmit for 30 minutes (new POST /v1/submissions or /retry); cheat rejected stays closed.

Test plan

• cargo test -p prism-lium -p submission-gating -p prism-challenge --lib
• cargo clippy -p prism-lium -p submission-gating -p prism-challenge --all-targets -- -D warnings
• cargo run -p xtask -- loc-cap / external-docs-check
• Deploy prism-challenge to prod; confirm new install failures reconnect deps and 429s backoff
• Ops: reset_open for hotkeys already blocked outside the 30m window (e.g. the two miners from today)

Summary by CodeRabbit

• New Features

  • Infrastructure-related submission failures can be retried within a 30-minute recovery window.
  • Manual retries now allow up to three attempts.
  • Network requests automatically retry rate-limit responses with bounded backoff.

• Bug Fixes

  • Improved handling of dependency installation, verification, timeouts, and error reporting.
  • Extended SSH keepalive settings to improve connection reliability.

• Documentation

  • Updated submission retry and infrastructure recovery guidance.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b34c2f32-847f-4cfc-af98-9634f63d3cb4

📥 Commits

Reviewing files that changed from the base of the PR and between a6ad6e7 and 194a3ac.

📒 Files selected for processing (7)

• bins/prism-challenge/src/main.rs
• crates/prism-challenge/src/api.rs
• crates/prism-lium/src/client.rs
• crates/prism-lium/src/ssh.rs
• crates/submission-gating/src/lib.rs
• docs/PRISM.md
• docs/external-miner/prism.md

---

📝 Walkthrough

Walkthrough

The change adds a 30-minute infrastructure resubmission window, updates challenge retry handling, centralizes Lium REST requests with bounded 429 retries, moves dependency installation before evaluation, and increases SSH keepalive limits.

Changes

Submission retry window

Layer / File(s)	Summary
Infrastructure retry eligibility crates/submission-gating/src/lib.rs	The gating library identifies infrastructure errors and permits blocked rows within a 30-minute window. Tests cover valid, expired, and invalid cases.
Challenge submission and retry flow crates/prism-challenge/src/api.rs, bins/prism-challenge/src/main.rs, docs/PRISM.md, docs/external-miner/prism.md	The API accepts eligible infrastructure resubmissions, bypasses retry limits, resets gating state, and requeues submissions. Retry configuration and documentation describe the updated behavior.

Lium execution resilience

Layer / File(s)	Summary
Unified REST request handling crates/prism-lium/src/client.rs	REST operations use a shared helper with optional JSON bodies, empty-response support, sanitized errors, and bounded 429 retries.
Evaluation dependency preparation crates/prism-lium/src/client.rs	Pinned Python dependencies are installed and verified over SSH before evaluation.
Provisioning and SSH resilience crates/prism-lium/src/client.rs, crates/prism-lium/src/ssh.rs	Provisioning waits after rental rate-limit errors. Both SSH paths use longer keepalive settings.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SubmissionAPI
  participant GatingLibrary
  participant LiumClient
  participant SSHExecutor
  SubmissionAPI->>GatingLibrary: check infrastructure retry window
  GatingLibrary-->>SubmissionAPI: return eligibility
  LiumClient->>LiumClient: retry REST request after 429
  LiumClient->>SSHExecutor: install and verify dependencies
  LiumClient->>SSHExecutor: execute evaluation harness

Loading

Possibly related PRs

• BaseIntelligence/base#85: Both changes modify Prism challenge retry configuration constants.
• BaseIntelligence/base#88: Both changes address Prism retry handling for infrastructure failures.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/prism-lium-resubmit

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}