Skip to content

Performance report against v0.1 release gates on the reference machine #101

Description

@burakdede

Parent epic: #8

Context

§5.5 states the performance release gates and adds a hard rule: "These are benchmark gates to measure before release. They must not be advertised until achieved." §20.10 requires versioned benchmark data and documented reference-machine details.

Spec references

§5.5 Performance release gates, §5.4 scale envelope, §20.10 Performance benchmarks, §24.x (do not optimize beyond measured bottlenecks).

Scope

In: the reference-machine definition, the consolidated benchmark suite, the published performance report, and the release gate enforcement.
Out: speculative optimization work (only measured bottlenecks justify follow-up issues).

Design notes

  • Reference machine per §5.5: 8 modern CPU cores, 16 GB RAM, local NVMe. Document the exact instance/hardware used so numbers are reproducible and comparable release over release.
  • Benchmarks are versioned with their corpus generator and metadata so a regression is attributable to a change rather than to a different corpus.
  • Any gate not met blocks the release or the corresponding claim is removed from documentation, never both ignored.

Acceptance criteria

  • Reference machine documented (hardware/instance, OS, Node version, storage) in benchmarks/README.md.
  • Suite measures all §5.5 gates: fingerprint over 2,500 files / 1 GB (p95 < 4 s), warm lexical search over 50,000 chunks (p95 < 250 ms), lore_context_for_task (p95 < 1.5 s), single-document incremental rebuild (< 2 s), progress emitted at least once per second during long operations, and cancellation leaving the previous build intact.
  • Results published as docs/compatibility/performance-v0.1.md with p50/p95, corpus description, and machine metadata.
  • Release gate: publishing is blocked when a gate is unmet unless the corresponding claim is removed from all documentation, enforced by a check linking claims to measurements.
  • Benchmark results stored per release for trend comparison; a significant regression fails CI.
  • Any bottleneck found is filed as its own issue with the measurement attached rather than fixed speculatively.

Tests / verification

  • Benchmark suite run on the reference machine for the release candidate.
  • Regression-detection test against stored baselines.
  • Claims/measurement linkage check.

Dependencies

Blocked by the Phase 1/3/5 benchmark issues (#40, #83).

Definition of Done

Every published performance number is measured on a documented machine, and no unmet gate is ever advertised.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:infraRepo, CI, tooling, releasetype:perfPerformance and benchmarks

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions