Add quick performance benchmark tests - #329
Draft
DanRod1999 wants to merge 4 commits into
Draft
Conversation
Create lightweight perf benchmarks in the performance-tests package that: - Generate synthetic iModels at runtime (no auth/hub/.env needed) - Import 15 custom ECSchemas + insert 10k PhysicalObject elements - Measure processSchemas() and process() with detailed breakdowns - Use BenchmarkTransformer/BenchmarkImporter subclasses to capture per-method timing (export elements, import elements, per-schema cost) - Include ECReferenceTypesCache initialization benchmark - Output structured results to console Run with: pnpm test:quick-perf (from packages/performance-tests) Closes #328 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
IModelHost.startup() generates .bcv and .itwindb files in the cache directory during test runs. These should not be tracked. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove per-call performance.now() from element/relationship/schema overrides to eliminate measurement overhead. Keep only: - processSchemas() and process() wall-clock timing - Element/relationship/schema counts - Avg per element derived from process() / element count Remove BenchmarkImporter (no longer needed without per-call timing). This brings process() times within ~7% of TransformerPerf.test.ts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The existing performance regression tests require hub access, authentication, and downloading real iModels -- making them slow and hard to run locally for quick feedback. We need lightweight benchmarks that can run offline in seconds to catch performance regressions during development.
Approach
Adds a new
pnpm test:quick-perfscript inpackages/performance-teststhat generates synthetic iModels at runtime and benchmarks the transformer with minimal instrumentation overhead.Key design decisions:
.envrequired. Creates aStandaloneDbwith 15 custom ECSchemas and 10kPhysicalObjectelements.BenchmarkTransformersubclass -- overridesprocessSchemas()andprocess()to capture phase-level wall-clock timing, plus counts elements/relationships/schemas via lightweight counter-only overrides (no per-callperformance.now()to avoid measurement overhead).initAllSchemasInIModel()since those metrics are useful but don't belong as logging in the library itself.Sample output
Notes
.cachedirectory generated byIModelHost.startup()is now gitignored.TransformerPerf.test.ts(1.78ms), with the gap explained by the 15 additional schemas adding metadata overhead.Closes #328