Repository navigation
docs(agents): point batch runs at run-multi-model-benchmark.sh - #172
Merged
Merged
Conversation
AGENTS.md named run-e2e-benchmark.sh as the "top-level end-to-end entry point" and did not mention run-multi-model-benchmark.sh at all, so an agent asked to benchmark several models finds the one-model script and either wraps it in a hand-rolled loop or lands on run-benchmark-batch.sh, which judges inline and serves nothing. Name all three in the repository map with what each is for, and add a "Running a benchmark" entry that states the rule up front: more than one model means the multi-model script. Record what it already does, since each is a thing an agent would otherwise rebuild -- model serving from its registry with the right tensor-parallel size, setsid self-detach, per-model run-summary commits, and --judge-mode async, which overlaps the GPU-free Bedrock judge with the next model's generation instead of leaving the GPU idle for roughly 50 minutes per model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The repository map in AGENTS.md labelled
run-e2e-benchmark.shthe "top-level end-to-end entry point" and never mentionedrun-multi-model-benchmark.sh. An agent asked to benchmark several models therefore finds the one-model script, and either wraps it in a hand-rolled loop or lands onrun-benchmark-batch.sh, which judges inline and serves nothing. That is the wrong tool for a self-hosted batch, and the mistake costs GPU-idle hours rather than failing loudly.This happened while setting up the issue #157 re-run: two scripts were read before the right one, even though
.claude/skills/benchmark/SKILL.mdalready says "do not hand-write a scoring watcher". That guidance only reaches someone who has opened the skill.What changed
Repository map -- all three runners are now listed with what each is for:
Running a benchmark -- a new first entry states the rule before the reading list, and records what the multi-model script already handles, since each is something an agent would otherwise rebuild:
setsid, so a session teardown cannot kill a multi-day runrun-summaryto the current branch--judge-mode asyncscores the finished model while the next one generatesIt also explains why the judge mode matters: the judge is a Bedrock call that uses no GPU, so
asyncoverlaps it for free,inlineleaves the GPU idle for roughly 50 minutes per 21-task model, andskipleaves a serial tail after the batch. Therun-benchmark-batch.shline says what that script is for rather than leaving it unexplained.Documentation only. No code paths change, and every path named was checked to exist.