Skip to content

docs(agents): point batch runs at run-multi-model-benchmark.sh - #172

Merged
aarora79 merged 1 commit into
mainfrom
docs/agents-multi-model-benchmark
Sep 7, 2026
Merged

aarora79 merged 1 commit into
mainfrom
docs/agents-multi-model-benchmark

Conversation

@aarora79

@aarora79 aarora79 commented Sep 7, 2026

Copy link
Copy Markdown
Owner

Why

The repository map in AGENTS.md labelled run-e2e-benchmark.sh the "top-level end-to-end entry point" and never mentioned run-multi-model-benchmark.sh. An agent asked to benchmark several models therefore finds the one-model script, and either wraps it in a hand-rolled loop or lands on run-benchmark-batch.sh, which judges inline and serves nothing. That is the wrong tool for a self-hosted batch, and the mistake costs GPU-idle hours rather than failing loudly.

This happened while setting up the issue #157 re-run: two scripts were read before the right one, even though .claude/skills/benchmark/SKILL.md already says "do not hand-write a scoring watcher". That guidance only reaches someone who has opened the skill.

What changed

Repository map -- all three runners are now listed with what each is for:

run-multi-model-benchmark.sh # MANY models back to back (start here for a batch)
run-e2e-benchmark.sh         # ONE model, end to end
run-benchmark-batch.sh       # simple sequential loop; judges inline, serves nothing

Running a benchmark -- a new first entry states the rule before the reading list, and records what the multi-model script already handles, since each is something an agent would otherwise rebuild:

  • serves each model from its own registry, stopping the previous one, with the tensor-parallel size and parsers that model requires
  • self-detaches with setsid, so a session teardown cannot kill a multi-day run
  • commits each run-summary to the current branch
  • --judge-mode async scores the finished model while the next one generates

It also explains why the judge mode matters: the judge is a Bedrock call that uses no GPU, so async overlaps it for free, inline leaves the GPU idle for roughly 50 minutes per 21-task model, and skip leaves a serial tail after the batch. The run-benchmark-batch.sh line says what that script is for rather than leaving it unexplained.

Documentation only. No code paths change, and every path named was checked to exist.

AGENTS.md named run-e2e-benchmark.sh as the "top-level end-to-end entry
point" and did not mention run-multi-model-benchmark.sh at all, so an
agent asked to benchmark several models finds the one-model script and
either wraps it in a hand-rolled loop or lands on run-benchmark-batch.sh,
which judges inline and serves nothing.

Name all three in the repository map with what each is for, and add a
"Running a benchmark" entry that states the rule up front: more than one
model means the multi-model script. Record what it already does, since
each is a thing an agent would otherwise rebuild -- model serving from
its registry with the right tensor-parallel size, setsid self-detach,
per-model run-summary commits, and --judge-mode async, which overlaps the
GPU-free Bedrock judge with the next model's generation instead of
leaving the GPU idle for roughly 50 minutes per model.
@aarora79 aarora79 added the documentation Improvements or additions to documentation label Sep 7, 2026
@aarora79
aarora79 merged commit 2a323c8 into main Sep 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant