You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bf4e638
Browse filesBrowse the repository at this point in the historyBrowse files
feat(evalboard): add a Scheduled tab with per-pipeline trends and fold Scribe into it (#255)
* feat(evalboard): read series and excluded from meta.json and drop series runs from the Ad-hoc list
meta.json may now carry `series` (the series key) and `excluded` (a reason string) next to adhoc, title and description. Both are optional and tolerated when absent or malformed.
- RunMeta and PerRun carry the two fields; the Ad-hoc listing drops any run with a `series`, and keeps listing date-named runs flagged adhoc that have none.
- The listing loads further batches while series runs crowd the newest ids, so the section still fills to its limit.
- ensureRunMeta can refetch a cached copy older than a given age, because `excluded` and a backfilled `series` are written after the first download; a failed refetch keeps the cached copy.
- ensureRunExperiment and readRunExperiment read the per-arm aggregates in experiment.json.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* feat(evalboard): add the Series tab
A series is a recurring pipeline whose runs are points on one trend. lib/series.ts is the code-side table (maestro v1 vs v2 on codex and delegate, n8n vs Flow, ETE weekly, Scribe); a key found only in meta.json still gets a page under "Other".
- /series lists the series by group with schedule, last run with a health pill, the latest pass rate per arm (and the gap for two arms) and a sparkline of the last 8 non-excluded runs. Each row streams in its own Suspense boundary.
- /series/<key> has the trend with a metric toggle (pass rate per arm; cost per task, error share and wall-clock at run level), the gap between arms for two-arm series, and the run list with excluded runs greyed out and their reason shown.
- Per-arm numbers of a two-arm run come from experiment.json, never run.json.
- Excluded runs stay in the run list and are left off every trend and out of "latest".
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* feat(evalboard): point the Ad-hoc list and Scribe at the Series tab and warm /series
- The front page's Ad-hoc list gets a one-line note pointing to the Series tab (internal edition only).
- The Scribe page links to /series/scribe.
- The warmer requests /series after /, so the series index is cached before a visitor opens it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(evalboard): keep a null arm pass rate null and cap Series run.json loads
experiment.json writes pass_rate null when no row of an arm produced a verdict. The parser treated that like a missing field and derived 0% from the counts, so such an arm plotted a false 0 on the trend. Only a file without the field now gets a derived rate.
Every row of the Series run list loaded its cost at once, and each cold load parses a multi-MB run.json. Run-level loads now go through a shared limit of 8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(evalboard): simplify the Series index and drop hand-copied schedules
The series table held each pipeline's schedule and cadence, copied by hand from the cron lines in coder_eval_uipath. Nothing kept the two in step, and a run that starts hours after its cron (the ETE runs start between 04:12 and 13:51 UTC) can land on another UTC day, so the health pill could report a missed run that was not missed. The table now holds no timing: the schedule column, the cadence, the health pill and the paused flag are gone. The last-run date stays, because it comes from the run id.
The index also loses its group header rows, the arm count under each name and the gap between arms (the detail page keeps its gap chart). A sparkline needs two points to draw, and a series with no runs moves to one line under the table.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(evalboard): rename the nav tabs, fold Scribe into Scheduled, link runs to their trend
"Trends" and "Series" sat side by side in the nav and read as the same thing. Trends is now "Task trends", which matches its page heading, and Series is now "Scheduled", with the same label on its page and in the front-page note. The /series URL and the meta.json `series` key stay.
The Scribe tab duplicated /series/scribe: a pass-rate trend and the canonical run list. The detail page of a canonical series now also lists its container's ad-hoc runs, which were reachable only from the Scribe tab. /scribe redirects (307, from next.config) to /series/scribe, and the tab leaves the nav. The tiles and the 30-day chart are gone; the trend covers them.
A run page now links to its series' trend when the run belongs to one: by its meta.json `series`, or as a canonical run of a container that one series owns (Scribe).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(deps): cap typer below 0.29
typer 0.29.0 vendors its own click and removes Option(click_type=...), so
`coder-eval` fails at import on any unlocked install (TypeError in the run
command's --stream option). The plugin scaffold and Action dogfood CI jobs
install from the checkout without the lockfile and pick it up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
0 commit comments