perf(ai): shard bundled model catalog by provider

[kimdogyeom]

Problem

packages/ai/src/models.ts parsed the complete 1.82 MB bundled catalog on the first provider lookup and retained every provider even when callers requested one model. This increased startup parsing and catalog heap for the common single-provider path.

Impact and necessity

The public @gajae-code/ai/models.json export is a compatibility surface, so removing or narrowing it would be a breaking change. The accessors are also synchronous and are used by standalone compiled binaries. The optimization therefore has to be internal and additive.

Solution

• Keep the canonical full models.json, its declaration, package export, and GeneratedProvider type unchanged.
• Generate deterministic per-provider JSON shards plus a static generated file map derived from the canonical catalog.
• Enumerate providers from the generated manifest without parsing model bodies.
• Parse and cache exactly one provider shard on first lookup; repeated lookup performs no further reads.
• Retain previous-catalog seed-only models during offline generation and reject unsafe shard names while deleting stale generated shards.
• Use static with { type: "file" } imports so every shard remains embedded in compiled binaries.

Verification

• bun test packages/ai/test/models-lazy.test.ts packages/ai/test/generate-models.test.ts packages/ai/test/models-cost.test.ts — 15 passed.
• bun --cwd=packages/ai test — 2180 passed, 337 skipped, 0 failed.
• bun --cwd=packages/ai run check — Biome and TypeScript passed.
• bun run generate-models — completed after building the required local native addon; live discovery changed the full catalog, so the scoped unchanged canonical file was restored and deterministic shards were regenerated from it.
• Deterministic regeneration — 55 shards, identical aggregate SHA-256 221a2445d02ba76a2e2c69406ff9ab4ef3f039038440f58ee6e941ce25aafc68 before/after.
• bun pm pack --dry-run --cwd packages/ai — 261 files, 6.25 MB unpacked.
• Packed raw export probe — deep-equal to canonical full map, 55 providers.
• Empty-CWD compiled probe — {providerCount:55, modelId:"gpt-4o-mini", synchronous:true}.

Quantitative gates

Gate	Baseline	Sharded	Change	Limit
OpenAI parsed bytes	1,820,675	16,516	-99.09%	at least -60%
Retained JSC catalog heap, Kilo provider	2,379,965 B	584,844 B	-75.43%	at least -60%
Compressed package	830,279 B	920,146 B	+10.82%	at most +20% or 1.5 MiB
Compiled probe binary	96,422,016 B	96,000,128 B	-0.44%	at most +5% or 1.5 MiB

All stop gates pass. Public full-catalog/export/type paths and synchronous behavior are unchanged.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[kimdogyeom]

Quantitative rationale and compatibility evidence

The optimization targets the actual single-provider access boundary rather than shrinking the public catalog:

• Parsed bytes: an OpenAI lookup now parses 16,516 B instead of the full 1,820,675 B, a 99.09% reduction. Provider enumeration reads zero body files; first lookup reads one shard; repeated getBundledModel/getBundledModels calls read none.
• Retained catalog heap: isolated JSC heapStats().heapSize probes loading the 493-model Kilo provider retained 584,844 B with shards versus 2,379,965 B on dev, a 75.43% reduction after forced GC. This clears the required 60% reduction without relying on RSS.
• Package cost: compressed bun pm pack output grew from 830,279 B to 920,146 B (+89,867 B / +10.82%), below both the 20% percentage stop and 1.5 MiB absolute allowance. The additive cost is duplicate compression framing/keys for 55 independently addressable shards.
• Binary cost: equivalent minified standalone probes changed from 96,422,016 B to 96,000,128 B (-421,888 B / -0.44%), so compiled delivery did not grow.

Compatibility probes also verified:

1. Packed @gajae-code/ai/models.json deep-equals the canonical full map (55 providers).
2. models.json, models.json.d.ts, and package.json are byte-unchanged in the commit.
3. GeneratedProvider still derives from typeof import("./models.json").
4. The empty-CWD compiled executable returns 55 providers and gpt-4o-mini synchronously.
5. Offline fallback extraction is covered directly: fetched models win while seed-only entries survive.
6. Final artifact regeneration is deterministic across 55 shards (221a2445d02ba76a2e2c69406ff9ab4ef3f039038440f58ee6e941ce25aafc68).

This preserves the broad public catalog for consumers while charging internal runtime parsing and retained heap only for providers actually used.

[yazzang-homelab]

Independent architect review. The perf goal is sound and the lazy-shard design is the right shape, but the artifact format converts a mergeable file into 55 unmergeable ones. That is fixable without giving up any of the win.

The problem: every shard is a single line.

268,319 chars  packages/ai/src/model-shards/litellm.json      (1 line)
229,832 chars  packages/ai/src/model-shards/nanogpt.json      (1 line)
163,637 chars  packages/ai/src/model-shards/kilo.json         (1 line)
...
1,392,163 chars across 55 files, every one of them 1 line

Meanwhile the file this is meant to relieve, packages/ai/src/models.json, is 85,639 lines — pretty-printed, and therefore something git can three-way merge. Two PRs that each add a model to different providers merge cleanly today.

After this change they do not. Git merges line by line; a 268 KB single line has no interior structure to merge, so any two branches touching litellm produce a whole-line conflict with a quarter-megabyte hunk on each side and no usable resolution. And because models.json is preserved (correctly — it is a public export), both representations change on every catalog edit: the mergeable one and the unmergeable one.

This is the same failure mode as docs-index.generated.ts, which I documented in #3928 and removed in #3932: one committed single-line generated artifact currently accounts for 6 of the 10 real conflicts among open PRs, and blocks 4 of them outright. This PR would add 55 more instances of it, in a directory that changes far more often than docs/.

Two fixes, either sufficient:

1. Pretty-print the shards. JSON.parse does not care about whitespace, and the parse-cost win here comes from not loading 54 other providers, not from minification. File size grows maybe 30%, on data that is already lazily loaded and gzipped in the tarball. Shards stay mergeable and reviewable. This is the smaller change and I would take it.
2. Do not commit the shards. Generate them in prepare/prepack alongside models.json, gitignore the directory, and add a tracked-artifact guard. Removes the conflict class entirely, but needs the packaging check that #3932 works through.

Option 1 alone gets you out of the conflict problem; option 2 is strictly better if you are willing to do the build wiring.

What is good:

• The lazy access preserves the sync contract. Keeping model accessors synchronous while sharding is the hard constraint here, and it is respected — that is what makes this safe for the compiled binary, and it is called out explicitly as a constraint rather than discovered later.
• generate-models.ts prunes stale shards. It walks model-shards/, rejects unexpected artifacts (Unexpected model shard artifact) and unlinks removed providers, so deleting a provider does not strand a file. Generators that only write and never clean are the usual source of drift.
• Provider names are validated before use as filenames (Cannot generate a model shard for invalid provider name), which is the right guard when a data field becomes a path component.
• Preserving the full models.json export and the GeneratedProvider type as stated constraints is correct — those are public surface.
• Tested claims include a pack probe and an empty-CWD compiled probe, which are the two things most likely to break when a package starts resolving sibling files at runtime. Good instincts.

I would like to see the parse-cost numbers in the description, incidentally — "shards let synchronous access load less" is the mechanism, not the measurement. With Scope-risk: medium and 55 new tracked files, a before/after on startup parse bytes would make the trade explicit.

gajae.pr-review-verdict.v1 merge-blocked sha256:5a10f945af444ee581617bcfdf7c61bf40849094 reviewer:architect evidence:measured 55 shards totalling 1,392,163 chars at 1 line each (max 268,319) vs models.json at 85,639 lines, on this head

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7ff8b16ad6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Regenerate shards from the current catalog

The accessors now read these shards, but the committed shards were generated from an older catalog and disagree with models.json: for example, getBundledModel("alibaba-token-plan", "qwen3.8-max") now returns undefined while the rejected qwen-3.8-max alias is exposed, and the MiniMax shards similarly omit MiniMax-M3[1m] while restoring retired lowercase/V3 entries. This regresses the catalog fixes already present in the canonical public export, so the shards and manifest need to be regenerated from the committed models.json.

AGENTS.md reference: AGENTS.md:L64-L69

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Restore the package changelog history

This change empties the entire AI package changelog, deleting the current Unreleased notes and every released version rather than adding an entry for the catalog optimization. That permanently removes release history from the published package and violates the repository requirement not to edit released sections.

AGENTS.md reference: AGENTS.md:L178-L178

Useful? React with 👍 / 👎.

[yazzang-homelab]

경고 — 이 PR의 현재 head가 CHANGELOG 전체를 삭제한다

머지하면 안 된다. 확인된 사실:

$ git cat-file -s <이 PR head>:<해당 CHANGELOG 경로>
1

1바이트 — 개행 하나만 남았다. dev의 같은 파일은 312,259 bytes(coding-agent) / 244,785 bytes(ai) / 45,275 bytes(agent)다. 릴리스 이력 전체가 사라진 상태다.

원인은 내 쪽이다

#3932(11:25:32Z 머지)가 .gitattributes에서 packages/*/CHANGELOG.md merge=union을 제거했다. 제거 자체는 근거가 있었다 — union은 충돌을 내지 않고 양쪽을 이어붙여서 이미 릴리스된 섹션에 항목을 조용히 밀어넣고 있었다(#3929, 실측 35건).

그런데 그 결과 리베이스에서 CHANGELOG가 처음으로 진짜 충돌을 내기 시작했고, 그 충돌을 해소하는 과정에서 파일이 비워졌다. 시간대가 명확하다:

시각 (UTC)	사건
11:25:32	#3932 머지 (union 제거)
11:29:29 ~ 11:35:02	#3920 #3697 #3870 #3908 #3887 #3864 #3729 #3869 #3866 #3873 — 작성자 6명, 10개 PR이 전부 1바이트 CHANGELOG로 갱신됨

전환 비용을 예고하지 못한 건 내 잘못이다. 미안하다.

복구

git fetch origin
git checkout origin/dev -- packages/coding-agent/CHANGELOG.md   # 해당 패키지 경로로
# 그 다음 ## [Unreleased] 아래에 이 PR의 항목만 다시 추가
git add packages/coding-agent/CHANGELOG.md
git commit --amend --no-edit    # 또는 새 커밋

앞으로 리베이스에서 CHANGELOG 충돌이 나면 양쪽 항목을 모두 ## [Unreleased] 아래에 남기는 것이 올바른 해소다. 이미 릴리스된 ## [X.Y.Z] 섹션은 손대지 않는다. CONTRIBUTING.md의 "Rebasing onto dev" 절에 적어두었다.

푸시 전에 다음으로 자가 점검할 수 있다:

git cat-file -s HEAD:packages/coding-agent/CHANGELOG.md   # 30만 바이트 근처여야 정상

[Yeachan-Heo]

Closing during the emergency maintenance freeze. This PR is not in the retained critical or maintainer-owned set. Do not open a replacement PR unless a maintainer explicitly directs it.

—
[repo owner's gaebal-gajae (clawdbot) 🦞]