-
Notifications
You must be signed in to change notification settings - Fork 16
fix(prism): rank top-model / board by G2 lattice score #163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
cc87374
66062fb
cef49a4
f701a32
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -8,10 +8,12 @@ | |
| //! their arch. Idempotent on digest (simultaneous duplicates share the | ||
| //! first registration). | ||
| //! 2. **Arch best bpb** — lower-wins update feeding owner credit + the | ||
| //! public leaderboard (any trainer's result counts). | ||
| //! 3. **Top-model publish** — when the row's bpb is a new global best | ||
| //! (≤ best scored bpb ever AND < last published bpb), publish to GitHub | ||
| //! and journal the publication. No-op without a configured publisher. | ||
| //! public leaderboard (any trainer's result counts; audit / secondary). | ||
| //! 3. **Top-model publish** — when the row's **lattice score** is a new | ||
| //! global best (≥ best scored score ever AND > last published score), | ||
| //! publish to GitHub/HF and journal. Matches live board ranking | ||
| //! (G2 benchmark lattice under `scoring_version` 4). Never min-bpb alone. | ||
| //! No-op without a configured publisher. | ||
|
|
||
| use std::sync::Arc; | ||
|
|
||
|
|
@@ -24,16 +26,35 @@ use tracing::{info, warn}; | |
| use crate::publish::{TopModelPublisher, TopModelRequest, TOPMODEL_REPO_PATH}; | ||
|
|
||
| /// Run registry + top-model bookkeeping for one finalized row. | ||
| #[allow(clippy::too_many_lines)] | ||
| pub async fn post_score_hooks( | ||
| store: &Arc<dyn PrismStore>, | ||
| publisher: Option<&TopModelPublisher>, | ||
| row: &SubmissionState, | ||
| ) { | ||
| let (Some(bpb), Some(FinalScore::Score(v))) = (row.bpb, &row.final_score) else { | ||
| post_score_hooks_inner(store, publisher, row, false).await; | ||
| } | ||
|
|
||
| /// Operator force-publish (republish CLI): same path as the auto trigger but | ||
| /// skips the global-best / beats-published guards. | ||
| pub async fn force_publish_topmodel( | ||
| store: &Arc<dyn PrismStore>, | ||
| publisher: Option<&TopModelPublisher>, | ||
| row: &SubmissionState, | ||
| ) { | ||
| post_score_hooks_inner(store, publisher, row, true).await; | ||
| } | ||
|
|
||
| #[allow(clippy::too_many_lines)] | ||
| async fn post_score_hooks_inner( | ||
| store: &Arc<dyn PrismStore>, | ||
| publisher: Option<&TopModelPublisher>, | ||
| row: &SubmissionState, | ||
| force: bool, | ||
| ) { | ||
| let (Some(bpb), Some(FinalScore::Score(lattice))) = (row.bpb, &row.final_score) else { | ||
| return; | ||
| }; | ||
| if *v == 0 { | ||
| if *lattice == 0 { | ||
| return; // cheat / copy-gate zero never publishes nor sets arch best | ||
| } | ||
|
|
||
|
|
@@ -83,9 +104,9 @@ pub async fn post_score_hooks( | |
| } | ||
| } | ||
|
|
||
| // (3) Top-model publish on a new global best — GitHub (optional) + HF | ||
| // (optional). Both require a verified secure-receive receipt when | ||
| // `PRISM_TOPMODEL_REQUIRE_WEIGHTS=1` (default); source-only is opt-in. | ||
| // (3) Top-model publish on a new global-best **lattice score** — GitHub | ||
| // (optional) + HF (optional). Both require a verified secure-receive | ||
| // receipt when `PRISM_TOPMODEL_REQUIRE_WEIGHTS=1` (default). | ||
| // Recipe 2.0 / AutoModel only — legacy 1.x never becomes the published | ||
| // top-model champion (historical FE rows stay in Postgres). | ||
| if !row.weight_eligible() { | ||
|
|
@@ -95,12 +116,20 @@ pub async fn post_score_hooks( | |
| ); | ||
| return; | ||
| } | ||
| let last = store.last_publication_bpb().await.unwrap_or(None); | ||
| let global = store.best_scored_bpb().await.unwrap_or(None); | ||
| let is_global_best = global.is_some_and(|g| bpb <= g); | ||
| let beats_published = last.is_none_or(|l| bpb < l); | ||
| if !(is_global_best && beats_published) { | ||
| return; | ||
| if force { | ||
| info!( | ||
| submission_id = %row.id, | ||
| score = lattice, | ||
| "top-model: force republish (operator)" | ||
| ); | ||
| } else { | ||
| let last = store.last_publication_score().await.unwrap_or(None); | ||
| let global = store.best_scored_score().await.unwrap_or(None); | ||
| let is_global_best = global.is_some_and(|g| *lattice >= g); | ||
| let beats_published = last.is_none_or(|l| *lattice > l); | ||
| if !(is_global_best && beats_published) { | ||
| return; | ||
| } | ||
|
Comment on lines
+104
to
+111
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift Select and reserve one canonical champion. Line 128 compares only the lattice score. The live leaderboard uses submission ID as the tie-breaker. Equal-score submissions can publish in hook-completion order instead of leaderboard order. Concurrent hooks can also pass before either publication is journaled. Make the store atomically select and reserve the canonical champion by descending score and ascending submission ID before external publication. Publish only the reserved submission. 🤖 Prompt for AI Agents |
||
| } | ||
| let ckpt = match prism_artifacts::verify_parked(&row.id) { | ||
| Ok(receipt) => { | ||
|
|
@@ -139,7 +168,13 @@ pub async fn post_score_hooks( | |
| if let Some(publisher) = publisher { | ||
| match publisher.publish(&req).await { | ||
| Ok(sha) => { | ||
| info!(submission_id = %row.id, bpb, commit = %sha, "top model published to GitHub"); | ||
| info!( | ||
| submission_id = %row.id, | ||
| score = lattice, | ||
| bpb, | ||
| commit = %sha, | ||
| "top model published to GitHub" | ||
| ); | ||
| let rec = TopModelPublication { | ||
| submission_id: row.id.clone(), | ||
| arch_id: arch_id.clone(), | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -118,17 +118,25 @@ pub trait PrismStore: Send + Sync + std::fmt::Debug { | |
| /// Journal one top-model publication. | ||
| async fn record_publication(&self, p: &TopModelPublication) -> Result<(), StoreError>; | ||
|
|
||
| /// bpb of the most recent publication (idempotency guard; `None` = never | ||
| /// published). | ||
| /// bpb of the most recent publication (audit; prefer [`Self::last_publication_score`]). | ||
| async fn last_publication_bpb(&self) -> Result<Option<f64>, StoreError>; | ||
|
|
||
| /// Lattice score of the submission behind the most recent publication | ||
| /// (idempotency guard for G2 / score ranking; `None` = never published or | ||
| /// the published row has no positive score). | ||
| async fn last_publication_score(&self) -> Result<Option<u64>, StoreError>; | ||
|
|
||
| /// Most recent top-model publication row (`None` = never published). | ||
| async fn last_publication(&self) -> Result<Option<TopModelPublication>, StoreError>; | ||
|
|
||
| /// Best (lowest) bpb across all scored submissions ever (global top-model | ||
| /// trigger baseline). | ||
| /// Best (lowest) bpb across weight-eligible scored submissions (audit / | ||
| /// legacy). Top-model publish uses [`Self::best_scored_score`]. | ||
| async fn best_scored_bpb(&self) -> Result<Option<f64>, StoreError>; | ||
|
|
||
| /// Best (highest) lattice score across weight-eligible scored submissions | ||
| /// (global top-model / HF champion trigger — matches live board ranking). | ||
| async fn best_scored_score(&self) -> Result<Option<u64>, StoreError>; | ||
|
|
||
| /// Non-terminal rows beyond grace — for the stuck sweep. | ||
| async fn list_stuck(&self, grace_secs: u64) -> Result<Vec<SubmissionState>, StoreError>; | ||
|
|
||
|
|
@@ -596,6 +604,24 @@ impl PrismStore for MemoryPrismStore { | |
| .map(|p| p.bpb)) | ||
| } | ||
|
|
||
| async fn last_publication_score(&self) -> Result<Option<u64>, StoreError> { | ||
| let last = self.last_publication().await?; | ||
| let Some(p) = last else { | ||
| return Ok(None); | ||
| }; | ||
| let rows = self | ||
| .rows | ||
| .lock() | ||
| .map_err(|_| StoreError::Backend("poison".into()))?; | ||
| Ok(rows | ||
| .iter() | ||
| .find(|r| r.id == p.submission_id) | ||
| .and_then(|r| match r.final_score { | ||
| Some(FinalScore::Score(v)) if v > 0 => Some(v), | ||
| _ => None, | ||
| })) | ||
| } | ||
|
Comment on lines
+509
to
+522
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift Persist the lattice score with the publication record. Both implementations read the current Store the lattice-score snapshot in
📍 Affects 2 files
🤖 Prompt for AI Agents |
||
|
|
||
| async fn last_publication(&self) -> Result<Option<TopModelPublication>, StoreError> { | ||
| Ok(self | ||
| .publications | ||
|
|
@@ -617,6 +643,20 @@ impl PrismStore for MemoryPrismStore { | |
| .min_by(f64::total_cmp)) | ||
| } | ||
|
|
||
| async fn best_scored_score(&self) -> Result<Option<u64>, StoreError> { | ||
| Ok(self | ||
| .rows | ||
| .lock() | ||
| .map_err(|_| StoreError::Backend("poison".into()))? | ||
| .iter() | ||
| .filter(|r| r.weight_eligible()) | ||
| .filter_map(|r| match r.final_score { | ||
| Some(FinalScore::Score(v)) if v > 0 => Some(v), | ||
| _ => None, | ||
| }) | ||
| .max()) | ||
| } | ||
|
|
||
| async fn list_stuck(&self, grace_secs: u64) -> Result<Vec<SubmissionState>, StoreError> { | ||
| let cutoff = now_ms().saturating_sub(grace_secs.saturating_mul(1000)); | ||
| Ok(self | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Return the force-publication outcome to the CLI.
The force hook returns
(), and publisher failures are only logged. The CLI then returnsOk(())and can print an older journal row as if the requested submission published successfully. This gives operators a successful exit status when no configured publisher exists or every publication attempt fails.Return a typed publication outcome or error from the force path. Fail the CLI command unless the requested submission creates a current publication record.
crates/prism-registry/src/hooks.rs#L39-L45: return the result of the forced GitHub/HuggingFace publication and journal operation.bins/prism-challenge/src/main.rs#L308-L317: convert a failed or absent outcome intoErrand verify the resulting record belongs toid.📍 Affects 2 files
crates/prism-registry/src/hooks.rs#L39-L45(this comment)bins/prism-challenge/src/main.rs#L308-L317🤖 Prompt for AI Agents