Skip to content

Add quick performance benchmark tests - #329

Draft
DanRod1999 wants to merge 4 commits into
mainfrom
danrod1999-quick-perf-benchmarks
Draft

Add quick performance benchmark tests#329
DanRod1999 wants to merge 4 commits into
mainfrom
danrod1999-quick-perf-benchmarks

Conversation

@DanRod1999

Copy link
Copy Markdown
Contributor

Motivation

The existing performance regression tests require hub access, authentication, and downloading real iModels -- making them slow and hard to run locally for quick feedback. We need lightweight benchmarks that can run offline in seconds to catch performance regressions during development.

Approach

Adds a new pnpm test:quick-perf script in packages/performance-tests that generates synthetic iModels at runtime and benchmarks the transformer with minimal instrumentation overhead.

Key design decisions:

  • Runtime-generated iModels -- no auth, no hub, no .env required. Creates a StandaloneDb with 15 custom ECSchemas and 10k PhysicalObject elements.
  • BenchmarkTransformer subclass -- overrides processSchemas() and process() to capture phase-level wall-clock timing, plus counts elements/relationships/schemas via lightweight counter-only overrides (no per-call performance.now() to avoid measurement overhead).
  • ECReferenceTypesCache benchmark -- standalone test that directly times initAllSchemasInIModel() since those metrics are useful but don't belong as logging in the library itself.
  • Console output -- clean table with schemas exported, processSchemas time, elements exported, process time, and avg per element.

Sample output

  Quick Perf Benchmark Results
  ──────────────────────────────────────────────────
    Schemas exported:       15
    processSchemas():       792.18 ms
  ──────────────────────────────────────────────────
    Elements exported:      10005
    Relationships exported: 0
    process():              19004.90 ms
    Avg per element:        1.8995 ms
  ──────────────────────────────────────────────────

Notes

  • The .cache directory generated by IModelHost.startup() is now gitignored.
  • A VS Code launch config is included for debugging.
  • Avg per element (~1.9ms) is within ~7% of the existing TransformerPerf.test.ts (1.78ms), with the gap explained by the 15 additional schemas adding metadata overhead.

Closes #328

DanRod1999 and others added 4 commits July 14, 2026 13:36
Create lightweight perf benchmarks in the performance-tests package that:
- Generate synthetic iModels at runtime (no auth/hub/.env needed)
- Import 15 custom ECSchemas + insert 10k PhysicalObject elements
- Measure processSchemas() and process() with detailed breakdowns
- Use BenchmarkTransformer/BenchmarkImporter subclasses to capture
  per-method timing (export elements, import elements, per-schema cost)
- Include ECReferenceTypesCache initialization benchmark
- Output structured results to console

Run with: pnpm test:quick-perf (from packages/performance-tests)

Closes #328

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
IModelHost.startup() generates .bcv and .itwindb files in the cache
directory during test runs. These should not be tracked.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove per-call performance.now() from element/relationship/schema
overrides to eliminate measurement overhead. Keep only:
- processSchemas() and process() wall-clock timing
- Element/relationship/schema counts
- Avg per element derived from process() / element count

Remove BenchmarkImporter (no longer needed without per-call timing).
This brings process() times within ~7% of TransformerPerf.test.ts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create smaller quick perf tests in Perf Test Package

1 participant