POC for versioned leaderboards#472

Open

msaroufim wants to merge 4 commits intomainfrom

Member

msaroufim commented Mar 19, 2026

No description provided.

Mark Saroufim added 3 commits

March 13, 2026 14:26


          [POC] Add task versioning and backfill support for eval changes

709ec89

Vibe-coded proof of concept — NOT production ready, putting up for
discussion. When update-problems changes a leaderboard's task/eval,
old run scores become stale. This adds:

- task_version column on leaderboard + runs tables
- Auto-bump task_version when update_leaderboard() detects task changes
- Ranking queries filter by current task_version (old scores hidden)
- POST /admin/backfill endpoint to re-run top N submissions from
  previous versions against the new eval

Known gaps / TODOs for discussion:
- No Discord slash command yet (API-only)
- No backfill progress tracking (fire-and-forget via BackgroundSubmissionManager)
- kernelboard frontend doesn't surface task_version info
- popcorn-cli unaware of version changes
- No tests
- Needs actual deployment testing with a real eval change


          Fix duplicate backfill bug and deduplicate update SQL

8ea9e8a

- Add NOT EXISTS clause to get_top_submissions_for_backfill to skip
  users who already have a passing run at the current task_version.
  Without this, running backfill twice creates duplicate submissions.
- Collapse the two near-identical UPDATE branches in update_leaderboard
  into a single query using CASE WHEN for the version bump.


          Merge origin/main; add Discord backfill command, progress tracking, a…

c92bb09

…nd tests

- Resolve merge conflicts in api/main.py (keep both backfill + export-hf endpoints)
- Update backfill migration to depend on ban-user (latest main migration)
- Add /admin backfill Discord slash command with progress tracking via
  interaction.edit_original_response() updates
- Add 5 API tests for /admin/backfill endpoint (auth, validation, noop, candidates)
- Add 7 DB tests for task versioning (version bumps, rankings filtering,
  backfill candidate discovery, exclusion of current version)

msaroufim changed the title ~~Pr 467~~ POC for versioned leaderboards

github-actions bot commented Mar 19, 2026

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/libkernelbot
leaderboard_db.py					1024
utils.py
Project Total

_{This report was generated by python-coverage-comment-action}


          Update ARC runner config: minRunners=40 to keep all runners online

30c8d3e

Previously minRunners was 0 (scale-to-zero), meaning runners only
appeared on the GitHub runners tab when jobs were queued. Set to 40
so all 5 nodes × 8 GPUs stay online and ready.

Member Author

msaroufim commented Apr 3, 2026

@codex[agent] wdyt

Codex AI commented Apr 3, 2026

@msaroufim The model is not available for your account. This can happen if the model was disabled by your organization's policy or if your Copilot plan doesn't include access to it.

You can try again without specifying a model (just @copilot) to use the default, or choose a different model from the model picker.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bf08e6b4-c467-4ee4-bd4b-48baae7328ef

chatgpt-codex-connector bot commented Apr 3, 2026

To use Codex here, create an environment for this repo.

Codex AI commented Apr 3, 2026

@msaroufim The model is not available for your account. This can happen if the model was disabled by your organization's policy or if your Copilot plan doesn't include access to it.

You can try again without specifying a model (just @copilot) to use the default, or choose a different model from the model picker.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bf08e6b4-c467-4ee4-bd4b-48baae7328ef

4 similar comments

Codex AI commented Apr 3, 2026

@msaroufim The model is not available for your account. This can happen if the model was disabled by your organization's policy or if your Copilot plan doesn't include access to it.

You can try again without specifying a model (just @copilot) to use the default, or choose a different model from the model picker.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bf08e6b4-c467-4ee4-bd4b-48baae7328ef

Codex AI commented Apr 3, 2026

@msaroufim The model is not available for your account. This can happen if the model was disabled by your organization's policy or if your Copilot plan doesn't include access to it.

You can try again without specifying a model (just @copilot) to use the default, or choose a different model from the model picker.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bf08e6b4-c467-4ee4-bd4b-48baae7328ef

Codex AI commented Apr 3, 2026

@msaroufim The model is not available for your account. This can happen if the model was disabled by your organization's policy or if your Copilot plan doesn't include access to it.

You can try again without specifying a model (just @copilot) to use the default, or choose a different model from the model picker.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bf08e6b4-c467-4ee4-bd4b-48baae7328ef

Codex AI commented Apr 3, 2026

@msaroufim The model is not available for your account. This can happen if the model was disabled by your organization's policy or if your Copilot plan doesn't include access to it.

You can try again without specifying a model (just @copilot) to use the default, or choose a different model from the model picker.

If you want to contact GitHub about this error, please mention the following identifier so they can better serve you: bf08e6b4-c467-4ee4-bd4b-48baae7328ef

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet