Context
The README references a 4h33 autonomous run with 16,393 events and 39/39 passing tests,
but there's no easy way for others to reproduce a benchmark run on their own machine.
Proposed solution
- Add a
grinta benchmark command (or a script in benchmarks/) that:
- Clones a small, deterministic test repository
- Runs Grinta with a predefined task prompt
- Measures wall-clock time, token usage, event count, and test pass rate
- Outputs a structured JSON report
- Include a
benchmarks/README.md explaining how to run and interpret results
- Document expected baseline numbers for at least one model (e.g., GPT-4o, Claude Sonnet)
Why this matters
Reproducible benchmarks build trust and let contributors measure the impact of their changes.
Context
The README references a 4h33 autonomous run with 16,393 events and 39/39 passing tests,
but there's no easy way for others to reproduce a benchmark run on their own machine.
Proposed solution
grinta benchmarkcommand (or a script inbenchmarks/) that:benchmarks/README.mdexplaining how to run and interpret resultsWhy this matters
Reproducible benchmarks build trust and let contributors measure the impact of their changes.