Skip to content

ci: run the benchmark/ suite in a label-opt-in lane #693

Description

@mparrett

Summary

#655 adds the pinned-release third leg to benchmark/run.sh plus a per-workload regression check, with BASELINE_STRICT=1 to turn it into a non-zero exit. It touches .gitignore, benchmark/BASELINE_REF, and benchmark/run.sh — no workflow — so nothing calls it.

benchmark/ therefore still never executes in CI. No workflow references it, and no Go-benchmark lane can reach it: the directory holds .lg, .clj, and run.sh. The PR lanes (perf-pr, perf-wasm) run the pr-fast profile, a single pkg/vm job with a fixed family regex, and perf-timeline runs -full, covering pkg/vm and test. None of them execute a let-go program.

This issue is the lane that calls #655's check. The rationale for the pinned baseline itself lives there and is not repeated here.

Proposal

A workflow following the existing perf-*.yml shape:

  • Label-opt-in, matching the perf / perf-repeat / perf-wasm pattern, so unlabeled PRs cost nothing.
  • Runs benchmark/run.sh with BASELINE_STRICT=1 against the BASELINE_REF bench: compare against a pinned-release baseline #655 pins.
  • Prints provenance: machine, Go version, and the pinned lg version, so a delta is attributable to a lane and not to an unlabeled runner.
  • Informational before gating, as perf-pr-repeat ran during its shadow phase.

Why the lane is worth adding

Two regressions found in the past week live in workloads the suite already covers:

CI caught neither. Both were found by hand, two days and one week after their respective merges. #655 makes the reduce case visible to the harness; a lane is what puts it in front of someone without running the suite locally.

Open questions

  1. Obtaining and caching the pinned lg. go install at a tag is simplest; building from a tag in a second checkout is more faithful for the AOT leg. Either wants caching, since this binary never changes by design.
  2. Runtime budget. Seven workloads across three legs under hyperfine. perf-pr-repeat already carries a 90-minute cap for a smaller matrix.
  3. Whether startup belongs in the lane. It is milliseconds and noise-dominated, so it may fit an always-on check better than a delta table.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions