Skip to content

fix: Replace flaky performance tests with statistical benchmarking fr…#1127

Merged
braedonsaunders merged 1 commit into
mainfrom
claude/fix-flaky-performance-tests-4iwTV
Jan 30, 2026
Merged

fix: Replace flaky performance tests with statistical benchmarking fr…#1127
braedonsaunders merged 1 commit into
mainfrom
claude/fix-flaky-performance-tests-4iwTV

Conversation

@braedonsaunders

Copy link
Copy Markdown
Owner

…amework

Introduces a world-class performance testing approach that eliminates test flakiness caused by JIT compilation, system load, and CI environment variance.

Key changes:

  • Add BenchmarkRunner.ts: Statistical benchmark harness with warmup phases, outlier removal (IQR method), and percentile-based metrics (p50/p75/p95)
  • Add performanceTestHelpers.ts: Utilities for algorithmic complexity testing, cache effectiveness validation, and adaptive threshold management
  • Refactor flockingBehavior.perf.test.ts: Replace single performance.now() measurements with statistical sampling and environment-calibrated thresholds
  • Refactor gameLoop.test.ts: Use Vitest fake timers for deterministic timing tests instead of real setTimeout
  • Add @tests alias to vitest.config.ts for test utility imports
  • Document performance testing guidelines in docs/TESTING.md

The new approach:

  • Multiple iterations with warmup phases (JIT optimization)
  • Statistical metrics (median, p95) instead of single measurements
  • Environment calibration for CI vs local differences
  • Algorithmic complexity verification (O(n) vs O(n²)) instead of absolute times
  • Welch's t-test for cache effectiveness validation

https://claude.ai/code/session_01PPES1DBUDZVb8fzcoe9Rwh

…amework

Introduces a world-class performance testing approach that eliminates test
flakiness caused by JIT compilation, system load, and CI environment variance.

Key changes:
- Add BenchmarkRunner.ts: Statistical benchmark harness with warmup phases,
  outlier removal (IQR method), and percentile-based metrics (p50/p75/p95)
- Add performanceTestHelpers.ts: Utilities for algorithmic complexity testing,
  cache effectiveness validation, and adaptive threshold management
- Refactor flockingBehavior.perf.test.ts: Replace single performance.now()
  measurements with statistical sampling and environment-calibrated thresholds
- Refactor gameLoop.test.ts: Use Vitest fake timers for deterministic timing
  tests instead of real setTimeout
- Add @tests alias to vitest.config.ts for test utility imports
- Document performance testing guidelines in docs/TESTING.md

The new approach:
- Multiple iterations with warmup phases (JIT optimization)
- Statistical metrics (median, p95) instead of single measurements
- Environment calibration for CI vs local differences
- Algorithmic complexity verification (O(n) vs O(n²)) instead of absolute times
- Welch's t-test for cache effectiveness validation

https://claude.ai/code/session_01PPES1DBUDZVb8fzcoe9Rwh
@braedonsaunders braedonsaunders merged commit a890605 into main Jan 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants