Skip to content

Add benchmark reproduction command #92

Description

@josephsenior

Context

The README references a 4h33 autonomous run with 16,393 events and 39/39 passing tests,
but there's no easy way for others to reproduce a benchmark run on their own machine.

Proposed solution

  • Add a grinta benchmark command (or a script in benchmarks/) that:
    • Clones a small, deterministic test repository
    • Runs Grinta with a predefined task prompt
    • Measures wall-clock time, token usage, event count, and test pass rate
    • Outputs a structured JSON report
  • Include a benchmarks/README.md explaining how to run and interpret results
  • Document expected baseline numbers for at least one model (e.g., GPT-4o, Claude Sonnet)

Why this matters

Reproducible benchmarks build trust and let contributors measure the impact of their changes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions