ci(perf-repeat): post the shadow-check report as a sticky PR comment - #708
ci(perf-repeat): post the shadow-check report as a sticky PR comment#708mparrett wants to merge 2 commits into
Conversation
Repeat A/B (variance-reduced) — informational shadow checkPer-cycle logs are in the job summary. Per-snapshot evidence — raw samples, cycle order, provenance — is in the Informational only. This check never gates the PR — the workflow header says why. |
|
It self-tested, which is the nicest kind of evidence: the sticky comment is already rendered on the PR with the aggregate table, the would-gate verdict, the artifact link, and no leaked Shell holds under One nit that is worth fixing in both lanes rather than just this one. Two smaller ones, neither blocking: the report text originates from PR-branch code and is interpolated inside a ``` fence, so a head branch could break out of the fence — same exposure as the existing lane, and the reason the token is in a separate job, but the artifact content is still head-controlled. And a stale sticky comment persists if the Worth noting what this unblocks: #706's author had to hand-post benchmark results because this step did not exist yet. |
listComments defaults to 30 per page. Past that the marker is not found and the "sticky" upsert silently creates a duplicate on every run. Flagged on #708 for both lanes — this was inherited by the repeat comment job from perf-pr.yml, so fix them together. Co-authored-by: Cursor <cursoragent@cursor.com>
afb4814 to
3f22fe9
Compare
listComments defaults to 30 per page. Past that the marker is not found and the "sticky" upsert silently creates a duplicate on every run. Flagged on #708 for both lanes — this was inherited by the repeat comment job from perf-pr.yml, so fix them together. Co-authored-by: Cursor <cursoragent@cursor.com>
The repeat A/B lane writes a job summary and uploads its evidence, but has no comment step, so a 65-minute run leaves nothing on the PR. On #706 the results had to be pulled out of the run artifact by hand to be read at all -- which means in practice the shadow check's output is only seen by whoever remembers to go looking. Mirrors perf-pr.yml's existing comment job rather than inventing a second pattern: separate job so the elevated token is scoped away from the one that builds PR code, sticky upsert by marker, continue-on-error for fork PRs whose token is read-only. Two deliberate choices: - The marker is `perf-pr-repeat-report`, distinct from perf-pr.yml's. Both lanes run on the same PR during the shadow phase and comparing them is the point, so neither may overwrite the other. - Gated on bench success. The bench job exits nonzero when families are not comparable across cycles; posting a delta table from a measurement already known to be broken is worse than posting nothing, and the summary still carries it. The comment body is the same text as the job summary, plus a link to the run for the per-snapshot evidence artifact. actionlint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The self-test on this PR posted the whole report: 120 lines, 9.6 KB, with seven literal `::group::` markers. Those are Actions log-folding directives and render as text in a comment body, so the useful part -- the family table and the would-gate verdict -- sat under a hundred lines of per-cycle bench output. The comment now starts after the last `::endgroup::`, which is where the aggregate begins: 36 lines, 1.8 KB, opening on the provenance header. The job summary is unchanged and still carries the per-cycle logs, and the comment points at it. Falls back to the full report if no group markers are found, so a format change or an early driver death posts something rather than nothing. Measured against the report this PR's own previous run produced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3f22fe9 to
559b821
Compare
listComments defaults to 30 per page. Past that the marker is not found and the "sticky" upsert silently creates a duplicate on every run. Flagged on #708 for both lanes — this was inherited by the repeat comment job from perf-pr.yml, so fix them together. Co-authored-by: Cursor <cursoragent@cursor.com>
perf-pr-repeat.ymlwrites a job summary and uploads its evidence, but has no comment step, so a 65-minute run leaves nothing on the PR. On #706 I had to download the run artifact and readaggregate.jsonto find out what the lane had concluded. The shadow phase is supposed to be collecting false-positive and provenance data, and output nobody sees does not get compared.This adds the comment job, mirroring the one already in
perf-pr.ymlrather than inventing a second pattern: separate job so the elevated token is scoped away from the one that checks out and builds PR code, sticky upsert by marker so repeated pushes update in place,continue-on-errorbecause fork PRs get a read-only token and that should not redden the run.The comment carries the aggregate — provenance header, family table, would-gate verdict — and points at the job summary for the per-cycle logs. The first self-test posted the whole report instead: 120 lines with seven literal
::group::markers, which are Actions log-folding directives that do not render in a comment. Trimming to the text after the last::endgroup::takes it to 36 lines.Two choices worth surfacing
The marker is
perf-pr-repeat-report, deliberately notperf-pr-report. Both lanes run on the same PR during the shadow phase, and the header of this workflow says comparing them is the point, so neither may overwrite the other.The comment is gated on bench success. The bench job exits nonzero when families are not comparable across cycles. Posting a delta table built from a measurement already known to be broken seems worse than posting nothing, and the job summary still carries the report in that case. Easy to relax if you would rather see the integrity failure on the PR too.
Verification
actionlintclean, including its shellcheck pass — the file was clean before and still is. The change is inert unless theperf-repeatlabel is applied, and since.github/workflows/perf-pr-repeat.ymlis inside the workflow's ownpathsfilter and same-repo PRs run it from the PR branch, labeling this PR exercises the new job on itself.