Skip to content

fix(prism): stop control-plane OOM restarts - #169

Merged
echobt merged 1 commit into
mainfrom
fix/prism-control-plane-oom
Aug 19, 2026
Merged

fix(prism): stop control-plane OOM restarts#169
echobt merged 1 commit into
mainfrom
fix/prism-control-plane-oom

Conversation

@echobt

@echobt echobt commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Slim GET /v1/submissions / reconcile list SQL so we no longer load tree_blob + full scripts + telemetry for hundreds of rows (this OOM-killed prism-challenge on the 16 GiB prod master and batched control_plane_restart requeues).
  • claim_next prefers queued rows that already have a pod_id so mid-flight resume is not pushed behind new submits (and workers reattach instead of a second lium rent).

Test plan

  • cargo test -p prism-store -p prism-orphan claim/reconcile cases
  • cargo clippy -p db -p prism-store -p prism-orphan -p prism-challenge --all-targets -- -D warnings
  • After merge: CI green on main, staging pins, prod tag v*.*.*
  • Prod: prism-challenge RSS stays well below host RAM; no batched control_plane_restart; no new lium rent for rows that already have a RUNNING pod_id

Summary by CodeRabbit

  • New Features

    • Improved recovery of interrupted evaluations by prioritizing submissions that can reconnect to existing running pods.
    • Orphan reconciliation now preserves live pods and requeues resumable evaluations without provisioning replacements.
  • Performance

    • Submission listings load faster by omitting large source files and telemetry data.
    • Champion views use slimmer data projections for more efficient retrieval.
  • Tests

    • Added coverage verifying resumed submissions are claimed before new submissions while retaining their existing pod assignments.

Listing submissions selected full tree_blob and scripts, which OOM-killed
prism-challenge on the 16 GiB master and requeued running jobs as
control_plane_restart. Claim now prefers queued rows that already have a
pod_id so resume does not wait behind new submits or rent a second Lium pod.
@echobt
echobt merged commit 56107ea into main Aug 19, 2026
1 of 3 checks passed
@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 436ac33d-8833-4682-9f27-34dca0f84623

📥 Commits

Reviewing files that changed from the base of the PR and between a9632b3 and b47828a.

📒 Files selected for processing (5)
  • crates/db/src/prism_store.rs
  • crates/prism-challenge/src/orchestrator.rs
  • crates/prism-orphan/src/lib.rs
  • crates/prism-store/src/store.rs
  • docs/PRISM.md

📝 Walkthrough

Walkthrough

Prism listing queries now use slim projections. Queue claiming prioritizes submissions with retained pod IDs. Orphan reconciliation and measurement preserve pod reattachment, with tests and documentation updated.

Changes

Prism resume and query behavior

Layer / File(s) Summary
Slim listing and champion projections
crates/db/src/prism_store.rs
Listing queries omit heavy source and review data. Champion queries retain architecture and review data while reducing other payloads.
Retained-pod queue priority
crates/db/src/prism_store.rs, crates/prism-store/src/store.rs
Database and in-memory claims select queued rows with existing pod_id values before new rows, while preserving FIFO order within each group. Tests verify the ordering.
Resume reattachment flow
crates/prism-challenge/src/orchestrator.rs, crates/prism-orphan/src/lib.rs, docs/PRISM.md
Comments, documentation, and orphan reconciliation tests describe and verify retained-pod reattachment.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OrphanReconciliation
  participant PrismStore
  participant PrismChallenge
  participant RetainedPod
  OrphanReconciliation->>PrismStore: claim_next queued row
  PrismStore-->>OrphanReconciliation: resumed row with pod_id
  OrphanReconciliation->>PrismChallenge: measure resumed submission
  PrismChallenge->>RetainedPod: reattach using pod_id
Loading

Possibly related PRs

✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/prism-control-plane-oom

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@echobt echobt mentioned this pull request Aug 19, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant