spike: benchmark dataset growth and fixture-loading strategies

[Ogstevyn]

closes #76

Summary

This spike answers how fixture loading and lookup should behave as the corpus grows from 12 entries to hundreds or thousands. It adds a reproducible benchmark harness under scripts/bench/ and writes the findings up in docs/benchmarks/dataset-scale.md.

No fixture files are touched. The recommendation is that nothing needs to change today.

Answer

The current two-file raw-JSON layout is comfortable to roughly 5,000 entries, needs a derived release artifact between 5,000 and 25,000, and only needs lazily-loaded split packs beyond that.

destinations.json + scores.json should stay the reviewable source of truth at every size - the labelling rubric, the changelog discipline and the reviewer checklist in CONTRIBUTING.md all depend on a fixture change being a readable diff. What should change as the corpus grows is what gets published, not what gets reviewed.

What was measured

Five corpus sizes (12 / 100 / 1,000 / 10,000 / 100,000) against five loading strategies:

#	Strategy	Artifact
1	Raw JSON + object lookup (today's baseline)	destinations.json + scores.json, verbatim
2	Raw JSON + Map index	identical artifact, consumer-side change only
3	Generated ES module	one fixtures.mjs of packed rows
4	Gzipped JSON archive	one fixtures.json.gz of packed rows
5	Split packs, lazily loaded	manifest.json + hash-addressed packs/N.json

Each cell is the median of 5 independent node --expose-gc processes, so no strategy inherits another's module cache, JIT state or heap. Reported per cell: cold load, warm load, sustained hit/miss lookup, retained heap, artifact size (raw, gzipped, and bytes-before-first-lookup), plus a cold-session figure - load the index and score 10 distinct destinations, which is what an extension session actually pays.

The 12-entry corpus is the real destinations.json/scores.json copied verbatim, so the baseline row is what consumers load today rather than a synthetic stand-in.

Headline numbers

Cold load (ms):

Strategy	12	1,000	10,000	100,000
Raw JSON + object lookup (baseline)	1.78	3.85	23.81	220.51
Gzipped JSON archive	2.45	3.35	11.40	129.76
Split packs, lazily loaded	1.46	1.04	1.25	1.62

Retained heap after load:

Strategy	12	1,000	10,000	100,000
Raw JSON + object lookup (baseline)	8.4 KB	414.7 KB	3.9 MB	41.3 MB
Gzipped JSON archive	18.5 KB	124.1 KB	1.2 MB	10.5 MB
Split packs, lazily loaded	4.2 KB	4.2 KB	2.9 KB	9.9 KB

The single most important number: at 100,000 entries a cold session (load, then score 10 destinations) costs 220 ms and 41 MB on the current strategy and 20 ms and 389 KB on split packs.

Full tables, including lookup latency and artifact sizes, are in the report.

Findings

1. Nothing is wrong today. At 12 entries the baseline loads in 1.8 ms and holds 8 KB.
2. Startup, not lookup, is the constraint. Lookup stays under 1 us everywhere - even the slowest cell is a rounding error against XDR decoding and a network call. Cold load spans 1.8 ms to 348 ms.
3. The baseline's cost is dominated by data the runtime never reads. The notes, address and type fields plus the JSON key names are roughly half the bytes and most of the retained heap.
4. Retained heap is the worst axis, at ~420 bytes/entry - a real problem for an MV3 service worker, which is evicted and respawned routinely and so pays cold load many times per browsing session rather than once per install.
5. Lazy loading only pays when the corpus is much larger than the working set. Split packs lose at 10,000 entries and win by 11x at 100,000; adopting them early would be a net loss.
6. Plain-object lookup is not prototype-safe - looking up the key constructor returns a function rather than undefined. Stellar strkeys and CODE:ISSUER asset IDs cannot collide with Object.prototype keys, so this is latent robustness rather than a live bug, but it is free to remove by indexing into a Map. The harness asserts this on every run.
7. Compression does most of what split packs do, at a fraction of the complexity - gzipped JSON is within 4% of split packs on total ship size.

Thresholds

Metric	Budget	Baseline breaches it at
Cold load (Node)	10 ms	~5,000 entries
Cold load (Node)	50 ms	~20,000 entries
Retained heap	5 MB	~12,000 entries
Bytes before first lookup	1 MB uncompressed	~2,500 entries
Release artifact	5 MB gzipped	~60,000 entries
Warm lookup	1 us	not breached at 100,000

The current approach becomes insufficient between roughly 2,500 and 5,000 entries - bytes-before-first-lookup goes first for a bundler-inlining consumer, then cold load.

Staged response: (1) no change now; (2) whenever convenient, have StubOracle build a Map and drop the parsed destinations array - small change, ~3.5x less retained heap, removes the prototype-key hazard; (3) at ~5,000 entries add a release build step emitting a packed gzipped artifact alongside the JSON; (4) at ~25,000 entries revisit split packs, re-running the harness first.

Reproducing

npm run bench
node scripts/bench/run-benchmarks.mjs --sizes 12,100,1000,10000,100000 --repeat 5

The first is the default sweep; the second is what produced the tables above. Results land in .bench/results.json and .bench/results.md. .bench/ is git-ignored and the corpora are regenerated deterministically, so nothing generated is committed. Synthetic account IDs are real Stellar strkeys built from a seeded PRNG; only the G... public-key form is implemented, so the harness cannot emit anything that resembles a secret seed.

Environment for the numbers above: Node v22.23.0, win32/x64, AMD Ryzen 7 PRO 7730U. The 1,000- and 10,000-entry sweeps were repeated on Node v20.20.2 (the version CI pins): load figures came out 10-30% slower, with the ordering, crossover points and per-entry constants unchanged. Absolute numbers are machine-specific; the ratios and scaling shapes are what the recommendation rests on.

Changes

• scripts/bench/ - the harness (corpus generator, strkey minter, strategies, per-process measurement, sweep driver)
• docs/benchmarks/dataset-scale.md - the report
• package.json - npm run bench, npm run bench:corpus
• .github/workflows/ci.yml - a smoke job running the two smallest corpora once, so the harness cannot bit-rot. It deliberately does not publish CI timings as results; shared runners are far too noisy for that.
• .gitignore - ignore the benchmark working directory
• README.md - a pointer to the report and to npm run bench

No fixture files changed, so no CHANGELOG.md entry is required per CONTRIBUTING.md.

Caveats

These are Node numbers, not browser numbers. In a browser each cold shard read becomes an async fetch or IndexedDB round trip, so a browser measurement should precede any split-pack adoption - the adapter's async IOracle interface survives that change, but the timings would shift. The 512-ID working set is also spread evenly across the corpus, whereas real lookups skew toward hot destinations, which favours lazy loading more than these numbers show. Both are recorded in the report.

CI fix included

CI on main had been red since #57: scripts/check-secrets.mjs failed to parse, so the secret-check job crashed and validate never ran (it declares needs: secret-check). At the maintainer's request this branch now fixes it, in its own commit kept separate from the spike.

Two problems, both from what looks like an encoding accident when the file was first written:

1. Syntax error - line 26 had its backticks replaced by backslashes and its template placeholders stripped, so the module crashed before scanning anything. Two Windows-1252 em-dash bytes elsewhere in the file are replaced with ASCII hyphens so this cannot recur.
2. The detector never fired. Stellar strkeys append the CRC16-XModem checksum little-endian (SEP-23), but isValidEd25519SecretSeed read it big-endian, so every genuine secret seed failed the checksum test and was discarded as a false positive. All 11 real account IDs in destinations.json validate little-endian and none validate big-endian. Planting a spec-correct seed in a tracked file confirms it: the guard now reports SECRET SEED FOUND: <file>:<line> - [REDACTED] and exits 1, where the previous logic reported a clean tree.

Fixing only the syntax error would have turned CI green while leaving a security guard that silently never fires - worse than a visibly broken one - so the file now self-tests on every run. It derives a well-formed seed at runtime from a fixed payload (no secret material is stored in the file) and exits non-zero if the detector stops recognising it. Run against the previous big-endian logic, that self-test reproduces the failure, so it would have caught this bug at merge time.

All three CI jobs pass locally: secret-check, validate (12 destinations, 12 scores) and bench-smoke.

[knytcomics-ui]

please pass failing test

[Ogstevyn]

@knytcomics-ui thanks for the review - the failing check is fixed, pushed as 8be3d4a in its own commit so it stays separate from the spike.

The failure was secret-check running scripts/check-secrets.mjs. It predates this branch: it has failed on every branch since #57 merged in July, and because validate declares needs: secret-check, the fixture validator has not actually run in CI since then either. While fixing it I found two distinct problems, both consistent with an encoding accident when the file was first written.

1. The syntax error that was failing CI. Line 26 had its backticks replaced with backslashes and its template placeholders stripped:

console.error(\SECRET SEED FOUND: \:\ ? [REDACTED]\);

so the module failed to parse and the job crashed before scanning a single file. Restored as a proper template literal reporting file, line and a redacted match. Two stray Windows-1252 em-dash bytes elsewhere in the file are now ASCII hyphens, so the same corruption cannot recur.

2. The detector would never have fired. This one is more serious than the crash. Stellar strkeys append the CRC16-XModem checksum little-endian (SEP-23), but isValidEd25519SecretSeed read it big-endian:

const expected = (bytes[bytes.length - 2] << 8) | bytes[bytes.length - 1];

The regex matched a leaked seed, the checksum test then rejected it as a false positive, and the script reported a clean tree. Two independent confirmations:

• All 11 real account IDs already in destinations.json validate as little-endian, and none validate as big-endian.
• Planting a spec-correct secret seed in a tracked file: the guard now prints SECRET SEED FOUND: <file>:<line> - [REDACTED] and exits 1, where the old logic exited 0 with "no secret seeds found". The planted file was removed afterwards and never committed.

3. Why I did not stop at the syntax fix. Fixing only the crash would have turned CI green while leaving a security guard that silently never fires, which is worse than a visibly broken one - nobody re-checks a green job. So the script now self-tests on every run: it derives a well-formed seed at runtime from a fixed payload (no secret material is stored in the file) and exits non-zero if the detector stops recognising it. Run against the old big-endian logic that self-test fails, so it would have caught this at merge time.

If you would rather keep this PR strictly to the spike, I am happy to move the check-secrets.mjs commit into its own PR against main and rebase this one on top - just say the word. I raised it as a separate-PR offer originally and only folded it in here because of your comment.

All three jobs pass locally:

secret-check  No Stellar secret seeds found - CI check passed.
validate      Fixture validation passed: 12 destinations, 12 scores.
bench-smoke   corpus 12 and 100, all five strategies, exit 0

The spike itself is unchanged: no fixture files are touched, and the recommendation is still that nothing needs to change today, with the current raw-JSON layout becoming insufficient between roughly 2,500 and 5,000 entries.

One thing I cannot do from my side: the workflow run is sitting at action_required because I am a first-time contributor to this repo, so the checks need your approval to run before they will show green here.