SchemaView performance improvements

[rschili]

imodel-native: iTwin/imodel-native#1479

Fixes: #9430

What

SchemaView was built and tuned against a very large iModel, balanced for the general mix of use cases.
Presentation has been testing it for adoption and the results are promising in almost every scenario, but
one critical scenario regresses on very large iModels - enough that they cannot adopt SchemaView as-is.

The scenario

• Very large iModel (~30 GB, ~100 schemas, hundreds of thousands of properties).
• Open the iModel and walk the model tree as fast as possible.

That path only needs BisCore, yet SchemaView paid to hydrate every schema in the file before returning. This PR makes SchemaView pay only for what the caller asks for, and removes a fixed cost that hurt every consumer.

Two changes, independent but shipped together.

Change 1: a cheaper schema-identity token (PRAGMA checksum(schema_token))

SchemaView uses a token to detect when an iModel's schemas have changed so it can drop its cached view.
It was obtaining that token from PRAGMA checksum(ecdb_schema), a SHA3 hash over the full contents of all schema tables (every class, property, and custom-attribute instance).

On the large iModel above, that checksum alone takes ~1.7 seconds, almost entirely CPU-bound on the hash - while loading BisCore plus its references via the binary blob is only ~30 ms. The token, meant to be a cheap "did anything change?" check, was dominating the whole operation, and every SchemaView consumer paid it regardless of how much schema they needed.

This PR adds PRAGMA checksum(schema_token), which hashes schema identity only - the name and version of each schema (one row per schema in ec_Schema, ordered by name). It is essentially free. The same cheap hash now also backs the schemaToken column returned by PRAGMA schema_view and the new PRAGMA schema_view_fragment, so a cached view and any later token check derive from the same value.

Measured on the ~30 GB / ~100-schema file: the token computation dropped from ~1744 ms to 1 ms, and the schema fragment pragma (see Change 2 below) from ~1759 ms to ~14 ms (cold ≈ warm, confirming the cost was CPU, not disk).

Known limitation (accepted): because the token hashes name + version only, it does not detect a schema whose contents change without a version bump. ECDb only allows in-place re-import for dynamic schemas, so that is the only case affected; We accept this for now and can strengthen the hash later (for example a cached per-schema content checksum) without changing the pragma's contract.

Frontend IModelConnection.invalidateSchemaViewIfChanged now queries PRAGMA checksum(schema_token) (reading .token) instead of PRAGMA checksum(ecdb_schema) (.sha3_256).

Change 2: getSchemaView({ schemas }) - load only a subset

getSchemaView() gains an optional argument. It is purely additive:

// Unchanged: loads every schema in the iModel using an optimized binary blob, exactly as before.
const full = await iModel.getSchemaView();

// New: ensure only BisCore and its references are loaded.
const view = await iModel.getSchemaView({ schemas: ["BisCore"] });
view.findClass("BisCore.Subject");        // present
view.findClass("Generic.PhysicalObject"); // undefined - not loaded

Behavior:

• The subset view is a accumulating instance. A later call with different schemas merges
  their closure into the same view, so schemas loaded earlier stay available.
• If the requested schemas (or all schemas) are already loaded, the call is a synchronous no-op that returns the existing view.
• Inside SchemaView: A schema that is not loaded looks identical to a schema the iModel does not contain: findClass and friends return undefined. Cross-schema walks (derivedClasses, etc.) are complete only over what is currently loaded.
• Schema names the iModel does not contain are ignored.

How it works

Manifest (cheap reference graph). On first subset request, the backend reads the schema reference graph from ECDbMeta (meta.ECSchemaDef + meta.SchemaHasSchemaReferences) - just names, versions, ids, and reference edges, no schema data. This is exposed as the new SchemaManifest type.

This has one incoming path via getSchemaView:

• If caller specified no filter, and nothing is loaded yet -> load the full blob.
• If filter specified OR (already some schemas loaded + no filter) -> calculate which schemas need to be loaded and load them as a fragment
• If no additional schemas need to be loaded, just return the object.

Serialized merges. Overlapping concurrent requests are serialized behind one in-flight promise and re-check the loaded-set inside the continuation, so two requests can never double-merge a schema.

Fixed Primitive Type enum width

The binary blob incorrectly used 8 bits for the primitive type, but we need 16 bits for it. I decided to fix this format in-place. The API is still beta, and the addon + backend are coupled together, so both should always speak the same language.

A point could be made that this warrants a v2 binary blob format - in the future such changes probably should, however, since no real consumer on frontend picked this up yet, and we will backport, I believe it's safe to fix this in place. It's a fix for the v1 format, not an evolution of it.

Performance numbers:

Ran this on a local release addon build against 2 very large iModels

First iModel (32 GB, 101 schemas, ~221k properties)

Scenario	Time
Subset (BisCore only): load + iterate	25.7 ms
Baseline: old full load + iterate	1389.4 ms

~54x faster.

Second iModel (9.4 GB, 33 schemas, ~117k properties)

Scenario	Time
Subset: load + iterate	18.3 ms
Baseline: old full load + iterate	607.6 ms

~33x faster.

[grigasp]

This PR adds PRAGMA schema_token, which hashes schema identity only - the name and version of each schema

Known limitation (accepted): because the token hashes name + version only, it does not detect a schema whose contents change without a version bump. ECDb only allows in-place re-import for dynamic schemas, so that is the only case affected; We accept this for now and can strengthen the hash later (for example a cached per-schema content checksum) without changing the pragma's contract.

How does this work with schema editing? Would editing a schema automatically increase the schema version?

[rschili]

@grigasp

How does this work with schema editing? Would editing a schema automatically increase the schema version?

Not automatically. When you import an edited schema, you have to increment the version yourself or ECDb will consider the schema unchanged. It's pretty safe to assume this works for all normal scenarios, with the only exception of dynamic schemas.

Our rules already state how callers should increment schema versions. Any change in non-dynamic schemas needs at least a minor digit increment. Frankly, we loosened the rules for dynamic schemas a few years ago, and I now believe that may have been a mistake. They incremented the minor version automatically very often and were worried it would eventually overflow.

If this is a problem, we can add a short term fix to force invalidate the schemaView whenever dynamic schemas are involved, but feels like beyond the scope of this PR.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

core/ecschema-metadata/src/SchemaView/SchemaViewBinaryReader.ts:260

• PropertyDefTable entries are a fixed-size 46-byte record (as the comment above notes), but validateCount uses minBytesPerEntry=8. On malformed/truncated blobs this can allow a much larger defCount to pass validation and allocate an unnecessarily huge new Array(defCount) before failing during reads. Use the actual fixed record size to bound the count and avoid large allocations on bad input.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated no new comments.

[grigasp]

Reviewed public api, docs and example code - looks good, I think the change should help with the problems we were seeing. I did not look at the implementation details.

[grigasp]

by "our metadata library" you mean the SchemaContext implementation, correct? Either way, I'm not sure we want to mention this drawback - IMO it's an implementation detail which doesn't need to be documented and could potentially be fixed in the future.

[rschili]

you're right, I removed that part, it doesn't really seem to add much value.

[grigasp]

The changelog comments should be user friendly messages with markdown syntax. Should start with uppercase letter, code in backticks, etc. And should be understandable by someone who's only familiar with the public api (at best).

[rschili]

Markdown? I adjusted the message but kept it short. Are you saying this should include detailed examples, too? That seems redundant to NextVersion.md and the documentation that is included.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated no new comments.

[hl662]

I think there's a lot of bloat in terms of in line comment that can be improved, deduped

[hl662]

This identity check can skip a required invalidation: a concurrent cached getSchemaView() replaces _viewPromise but still resolves to the same stale view, so a changed schema token is ignored. Maybe queue invalidation on the same serialized chain as loading, and add a regression test that pauses token fetching, performs a concurrent cache read, then returns a changed token and verifies the old view is discarded?

[rschili]

updated the code and added a test as you suggested

[hl662]

This duplicates the schemas property documentation, and the method already links to GetSchemaViewArgs. Can we remove it and keep this method comment focused on behavior that is not documented by an individual argument, such as the shared accumulating view and cache lifecycle?

[rschili]

the method doc now only states what the arguments don't

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

core/ecschema-metadata/src/SchemaView/SchemaViewBinaryReader.ts:310

• When merging fragments, the schema ecInstanceId -> schemaIdx map is updated unconditionally. If a schema is ever included twice (e.g., native writer/provider bug, or an accidental duplicate merge), this will silently overwrite the mapping and can corrupt subsequent cross-reference resolution. Consider failing fast if an ecInstanceId was already merged.

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

core/ecschema-metadata/src/SchemaView/SchemaViewBinaryReader.ts:263

• The blob layout is being changed in-place (e.g., primitiveType is now read as uint16), but the header version check still requires schemaViewFormatVersion === 1. This means a v1 blob produced by an older backend/native (where primitiveType was uint8) will still advertise version=1 and will be mis-parsed instead of failing cleanly. That undermines the frontend’s version-pinning strategy (PRAGMA schema_view(v1) / schema_view_fragment('v1;...')) for talking to older/newer backends and can lead to hard-to-diagnose runtime failures under version skew.

Consider introducing a new blob format version (e.g. bump schemaViewFormatVersion to 2) and keeping the old v1 reader/writer behavior for compatibility, or otherwise ensure the reader can detect and reject the legacy v1 layout before interpreting it as the new layout.