Skip to content

Commit bf4e638

Browse files
bai-uipathclaude
andauthored
feat(evalboard): add a Scheduled tab with per-pipeline trends and fold Scribe into it (#255)
* feat(evalboard): read series and excluded from meta.json and drop series runs from the Ad-hoc list meta.json may now carry `series` (the series key) and `excluded` (a reason string) next to adhoc, title and description. Both are optional and tolerated when absent or malformed. - RunMeta and PerRun carry the two fields; the Ad-hoc listing drops any run with a `series`, and keeps listing date-named runs flagged adhoc that have none. - The listing loads further batches while series runs crowd the newest ids, so the section still fills to its limit. - ensureRunMeta can refetch a cached copy older than a given age, because `excluded` and a backfilled `series` are written after the first download; a failed refetch keeps the cached copy. - ensureRunExperiment and readRunExperiment read the per-arm aggregates in experiment.json. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * feat(evalboard): add the Series tab A series is a recurring pipeline whose runs are points on one trend. lib/series.ts is the code-side table (maestro v1 vs v2 on codex and delegate, n8n vs Flow, ETE weekly, Scribe); a key found only in meta.json still gets a page under "Other". - /series lists the series by group with schedule, last run with a health pill, the latest pass rate per arm (and the gap for two arms) and a sparkline of the last 8 non-excluded runs. Each row streams in its own Suspense boundary. - /series/<key> has the trend with a metric toggle (pass rate per arm; cost per task, error share and wall-clock at run level), the gap between arms for two-arm series, and the run list with excluded runs greyed out and their reason shown. - Per-arm numbers of a two-arm run come from experiment.json, never run.json. - Excluded runs stay in the run list and are left off every trend and out of "latest". Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * feat(evalboard): point the Ad-hoc list and Scribe at the Series tab and warm /series - The front page's Ad-hoc list gets a one-line note pointing to the Series tab (internal edition only). - The Scribe page links to /series/scribe. - The warmer requests /series after /, so the series index is cached before a visitor opens it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(evalboard): keep a null arm pass rate null and cap Series run.json loads experiment.json writes pass_rate null when no row of an arm produced a verdict. The parser treated that like a missing field and derived 0% from the counts, so such an arm plotted a false 0 on the trend. Only a file without the field now gets a derived rate. Every row of the Series run list loaded its cost at once, and each cold load parses a multi-MB run.json. Run-level loads now go through a shared limit of 8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(evalboard): simplify the Series index and drop hand-copied schedules The series table held each pipeline's schedule and cadence, copied by hand from the cron lines in coder_eval_uipath. Nothing kept the two in step, and a run that starts hours after its cron (the ETE runs start between 04:12 and 13:51 UTC) can land on another UTC day, so the health pill could report a missed run that was not missed. The table now holds no timing: the schedule column, the cadence, the health pill and the paused flag are gone. The last-run date stays, because it comes from the run id. The index also loses its group header rows, the arm count under each name and the gap between arms (the detail page keeps its gap chart). A sparkline needs two points to draw, and a series with no runs moves to one line under the table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(evalboard): rename the nav tabs, fold Scribe into Scheduled, link runs to their trend "Trends" and "Series" sat side by side in the nav and read as the same thing. Trends is now "Task trends", which matches its page heading, and Series is now "Scheduled", with the same label on its page and in the front-page note. The /series URL and the meta.json `series` key stay. The Scribe tab duplicated /series/scribe: a pass-rate trend and the canonical run list. The detail page of a canonical series now also lists its container's ad-hoc runs, which were reachable only from the Scribe tab. /scribe redirects (307, from next.config) to /series/scribe, and the tab leaves the nav. The tiles and the 30-day chart are gone; the trend covers them. A run page now links to its series' trend when the run belongs to one: by its meta.json `series`, or as a canonical run of a container that one series owns (Scribe). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(deps): cap typer below 0.29 typer 0.29.0 vendors its own click and removes Option(click_type=...), so `coder-eval` fails at import on any unlocked install (TypeError in the run command's --stream option). The plugin scaffold and Action dogfood CI jobs install from the checkout without the lockfile and pick it up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent 68cd17f commit bf4e638

35 files changed

Lines changed: 3363 additions & 217 deletions

‎evalboard/README.md‎

Lines changed: 44 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -42,9 +42,16 @@ show up in the index — empty shells and the `latest` symlink are filtered out.
4242
- `/watchlist` — what needs attention, ranked over the recent-runs window: tasks
4343
and skills scored on failures, regressions and turn-budget pressure
4444
(`lib/watchlist.ts`).
45-
- `/scribe` — the Autopilot (aria/Composer) suite, run by `coder_eval_uipath`'s
46-
`UiPath.Autopilot.Eval.Manual` pipeline. This is the one surface that reads a
47-
**different blob container** (`aria-runs`, not `runs`) — see *Sources* below.
45+
- `/series` (nav: "Scheduled") — pipelines that run on a schedule outside the
46+
skills nightly, one trend each (`lib/series.ts`): a row per series with its last
47+
run, the latest pass rate per arm and a sparkline; series with no runs are
48+
listed under the table. `/series/<key>` has the trend (pass rate per arm; cost
49+
per task, error share and wall-clock at run level), the gap between arms for a
50+
two-arm series, and the run list. A run page links back to its series.
51+
- `/series/scribe` — the Autopilot (aria/Composer) suite, run by
52+
`coder_eval_uipath`'s `UiPath.Autopilot.Eval.Manual` pipeline. It reads a
53+
**different blob container** (`aria-runs`, not `runs`) — see *Sources* below —
54+
and also lists that container's ad-hoc runs. `/scribe` redirects here.
4855
- `/runs/latest` — shortcut that redirects to the newest run id.
4956
- `/runs/<run-id>` — run summary (pass rate, cost, duration) + one row per task.
5057
A "Download run (.zip)" button bundles the whole run folder.
@@ -115,7 +122,7 @@ dimension threaded through the data layer as a trailing
115122
| Source | Container | Surface |
116123
|--------|-----------|---------|
117124
| `skills` (default) | `runs` | Everything not listed below |
118-
| `scribe` | `aria-runs` | `/scribe` |
125+
| `scribe` | `aria-runs` | `/series/scribe` |
119126
| `gha` | `runs-gha` | none — direct links only |
120127

121128
Non-default sources are selected by a `?src=<id>` query param, which every
@@ -151,13 +158,42 @@ Two invariants worth preserving if you add a source:
151158
steady stream of ad-hoc uploads needs its own container, not a prefix in `runs`.
152159
- **Local mode is per-source too.** `listRunIds` resolves
153160
`runsDirFor(RUNS_DIR, source)` when `EVALBOARD_LOCAL_RUNS_DIR` is set, so
154-
`/scribe` reads `<local>-scribe`. Listing off the bare local dir instead —
161+
`/series/scribe` reads `<local>-scribe`. Listing off the bare local dir instead —
155162
which is what shipped first — returns the *default* source's ids for every
156163
source while the readers resolve under the sibling, so the listing and the
157164
reads disagree about which container they describe.
158165
`lib/__tests__/source-isolation.test.ts` pins both halves; it's the only test
159166
that exercises the reader layer, where the invariant above actually lives.
160167

168+
## Series
169+
170+
A series is a recurring pipeline whose runs are points on one trend. The table in
171+
`lib/series.ts` (like `lib/sources.ts`) lists each one with its label, arms and
172+
source. It holds no schedule: the crons live in `coder_eval_uipath`'s pipelines, and
173+
a copy here would drift. Two optional fields in a run's `meta.json` drive it, and
174+
every run without them behaves as before:
175+
176+
- `series` — the series key. The run is a point on that series' trend. The run
177+
keeps `adhoc: true`, so it stays out of the front page's chart and table.
178+
- `excluded` — a reason string. The run stays in its series' run list, greyed out
179+
with the reason (a run id inside it links to that run), and is left off every
180+
trend and out of "latest".
181+
182+
A key found only in `meta.json` gets a page labelled by the key.
183+
`scribe` has no `series` field: every date-shaped, non-adhoc run in its container
184+
belongs to it.
185+
186+
- Membership reads only `meta.json`, through a per-source index cached for five
187+
minutes. `meta.json` is edited after upload (`excluded`, a backfilled `series`),
188+
so the series index refetches a copy older than a minute for a run started in the
189+
last two days and older than an hour for any other.
190+
- Per-arm numbers of a variant series come from `experiment.json`, not `run.json`.
191+
Cost, cost per task, error share and wall-clock come from the run overview and
192+
are loaded only on the detail page, at most 8 runs at a time.
193+
- Each row of the index renders in its own `Suspense` boundary, so one slow series
194+
does not hold up the page.
195+
- The front page's Ad-hoc list drops runs that have a `series`.
196+
161197
## Conventions
162198

163199
- `/api/file?run=<id>&path=<relpath>[&src=<source>]` serves `.flow`, `.uipx`,
@@ -167,9 +203,9 @@ Two invariants worth preserving if you add a source:
167203
Files are gathered by `collectTaskFiles` / `collectRunFiles`, which reuse the
168204
`walkArtifacts` noise filter, and zipped by `lib/zip.ts` (a dependency-free
169205
DEFLATE writer).
170-
- In production the server requests its own `/` 15 s after it starts and then
171-
every 4 minutes (`lib/warm.ts`, started from `instrumentation.ts`), so the
172-
per-run cache is rebuilt by that request rather than by a visitor.
206+
- In production the server requests its own `/` and `/series` 15 s after it starts
207+
and then every 4 minutes (`lib/warm.ts`, started from `instrumentation.ts`), so
208+
the per-run cache is rebuilt by those requests rather than by a visitor.
173209
`EVALBOARD_WARM_INTERVAL_SECONDS` sets the interval; `0` turns it off.
174210
- Pass rows render green (`bg-green-50 text-green-700`), failures render red
175211
(`bg-red-50 text-red-700`), on a white background.

‎evalboard/app/layout.tsx‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,8 @@ export const metadata = {
1313
const NAV: readonly { href: string; label: string }[] = [
1414
{ href: "/path-to-ga", label: "Path to GA" },
1515
{ href: "/watchlist", label: "Watchlist" },
16-
{ href: "/trends", label: "Trends" },
17-
// The Autopilot/aria suite, read from its own blob container — see lib/sources.ts.
18-
{ href: "/scribe", label: "Scribe" },
16+
{ href: "/trends", label: "Task trends" },
17+
{ href: "/series", label: "Scheduled" },
1918
];
2019

2120
export default function RootLayout({ children }: { children: ReactNode }) {

‎evalboard/app/loading.tsx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ import {
77
} from "@/app/_components/skeleton";
88

99
// Fallback for the front page and, by nesting, for every route without a
10-
// closer one: /trends, /watchlist, /path-to-ga, /scribe and /runs/latest.
10+
// closer one: /trends, /watchlist, /path-to-ga, /series and /runs/latest.
1111
// Warm server time on those ranges from 3ms (/watchlist) to 2.3s
1212
// (/path-to-ga), and the slow ones are slow because they read run.json off the
1313
// Azure Files mount, which is worse in production than locally.

‎evalboard/app/page.tsx‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -502,6 +502,18 @@ export default async function Page({
502502
list, and trends above
503503
</span>
504504
</div>
505+
{isInternal && (
506+
<p className="text-xs text-gray-500">
507+
Runs from scheduled pipelines are on the{" "}
508+
<Link
509+
href="/series"
510+
className="text-studio-blue hover:underline"
511+
>
512+
Scheduled
513+
</Link>{" "}
514+
tab.
515+
</p>
516+
)}
505517
<TableScroll
506518
footer={
507519
adhocExpandable || adhocExpanded ? (

‎evalboard/app/runs/[id]/page.tsx‎

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import Link from "next/link";
12
import { Suspense } from "react";
23
import { notFound } from "next/navigation";
34
import {
@@ -12,7 +13,13 @@ import {
1213
readRunSummary,
1314
readRunTasks,
1415
} from "@/lib/runs";
15-
import { readRunReviewIndex, indexByTask, tagCountsForRun } from "@/lib/reviews";
16+
import {
17+
indexByTask,
18+
parseRunIdDate,
19+
readRunReviewIndex,
20+
tagCountsForRun,
21+
} from "@/lib/reviews";
22+
import { seriesForRun } from "@/lib/series";
1623
import { sourceById } from "@/lib/sources";
1724
import { scalarParam } from "@/app/_lib/source-param";
1825
import { fmtRunTime } from "@/lib/format";
@@ -78,6 +85,13 @@ export default async function RunPage({
7885
const tagCounts = reviewIndex ? tagCountsForRun(reviewIndex) : [];
7986
const reviewsByTask = reviewIndex ? indexByTask(reviewIndex) : undefined;
8087
const human = fmtRunTime(id);
88+
const series = isInternal
89+
? seriesForRun(source.id, {
90+
series: meta?.series ?? null,
91+
adhoc: meta?.adhoc ?? false,
92+
dateShaped: parseRunIdDate(id) != null,
93+
})
94+
: null;
8195

8296
return (
8397
<div className="space-y-5">
@@ -91,6 +105,14 @@ export default async function RunPage({
91105
Ad-hoc
92106
</span>
93107
)}
108+
{series && (
109+
<Link
110+
href={`/series/${series.key}`}
111+
className="text-xs text-studio-blue hover:underline"
112+
>
113+
{series.label} trend
114+
</Link>
115+
)}
94116
{/* Refresh re-pulls the run from blob storage — an
95117
internal-hosting surface (the public OSS edition has no
96118
blob backend). See lib/edition.ts.

‎evalboard/app/scribe/page.tsx‎

Lines changed: 0 additions & 159 deletions
This file was deleted.

0 commit comments

Comments
 (0)