Packaged Brain Console omits model-catalog.json, falsely reporting OpenAI/Gemini keys as absent

[sparkling]

Summary

The Brain Console's provider checklist gives false negatives in the published/persistent runtime: OpenAI and Google/Gemini are shown as unavailable even when their environment variables are present and Brain's own subscription detector sees them.

This is a release-boundary regression in the implementation delivered for #24 / PR #26, not a credential-configuration problem.

Reproduction

With the published 4.0.2 runtime and synthetic or real OPENAI_API_KEY plus GOOGLE_API_KEY:

1. Start the Brain Console.
2. Request GET /api/state.
3. Compare sections.savings.routerEngine.keys with sections.savings.routerEngine.subscriptions.

Observed on a live packaged runtime:

{
  "keys": { "openrouter": true },
  "subscriptions": {
    "openai": { "apiKey": true },
    "google": { "apiKey": true }
  }
}

No credential values are exposed. The contradiction is enough to demonstrate the defect.

Root cause

Current main and the published package have a broken asset boundary:

• scripts/model-catalog.mjs loads ../data/model-catalog.json.
• package.json#files includes scripts/ but does not publish data/model-catalog.json.
• beginConsoleRuntimeTransaction() copies the Console and scripts into the persistent runtime, but not the catalog data.
• gatherRouterEngine() performs catalog loading and provider-key detection in one try.
• When loadCatalog() throws, the catch supplies only the fallback house provider; providerKeys remains empty.
• The API then returns only the separately detected OpenRouter entry, although detectSubscriptions() correctly sees OpenAI and Google/Gemini.

The same missing asset was reproduced in clean installed/npx 4.0.2 package roots and in the persistent .console-runtime.

Why the previous fix is incomplete

#24 and PR #26 added the correct detect_env-driven provider checks. Those checks work from a developer checkout containing data/model-catalog.json, but the released npm artifact and staged Console runtime omit the required catalog. The feature therefore fails at the package/runtime boundary.

Expected behavior

A clean published install and the persistent Console runtime must both contain every required provider-catalog asset. The Console must report the same provider availability that Brain's native detectors observe.

If a required catalog cannot be loaded, the runtime must expose an explicit degraded/error state or fail staging. It must not silently convert an internal asset failure into apparently valid “key absent” results.

Proposed upstream implementation

1. Add the exact required data/model-catalog.json asset to the npm files allow-list.
2. Add the catalog to beginConsoleRuntimeTransaction()'s required asset set.
3. Before activating a staged persistent runtime, load and validate the catalog from the staged location.
4. Separate catalog-load health from credential detection:
   • preserve the catalog-driven provider matrix when available;
   • use the existing detectSubscriptions() result as the known-provider fallback if catalog loading fails;
   • expose that the catalog-dependent view is degraded rather than presenting false negatives.
5. Keep all credential handling boolean-only; never return key values.

Required regression coverage

• Pack the release artifact and assert data/model-catalog.json is present.
• Build the persistent Console runtime from that packed artifact and assert the catalog is copied and loadable there.
• Launch the packed/staged runtime with dummy OPENAI_API_KEY and GOOGLE_API_KEY; assert /api/state reports both as available.
• Cover GEMINI_API_KEY as the Google-provider alias.
• Include unset-key negative controls.
• Remove/withhold the catalog and assert staging fails explicitly, or the API reports a degraded catalog state while retaining correct native subscription detections.
• Run these checks in release CI so a package cannot be published when source-checkout behavior and packed-runtime behavior diverge.

Verified against

• ruvnet-brain 4.0.2
• current upstream main at 924d29a8a5d7b5c09effdd21d75250ea15140c60
• macOS persistent Console runtime

Related: #24, PR #26.

[stuinfla]

🤖 Automated acknowledgment — received and opened. The maintainer has been paged and this is being worked. The next update here will be a fix, findings, or the maintainer in person.

[stuinfla]

Confirmed in the 4.0.4 package boundary: data/model-catalog.json is still absent from the npm allow-list and persistent Console runtime. Closure requires packed-artifact, staged-runtime, dummy-key/alias, negative-control, and explicit degraded-state tests.

[stuinfla]

This one was two bugs wearing one title, and your framing — "falsely reporting OpenAI/Google state" — was the important half.

The packaging half was fixed shortly before I got here: data/model-catalog.json is in package.json#files and the runtime transaction stages and validates it.

The honesty half was still live, and it's the one that matters. The server was already careful — gatherRouterEngine() published providerCatalog: {status: 'degraded'} when the catalog couldn't load. But console/app.js:2342 built the "Other keys found:" checklist straight off keys as a hard binary, with the tooltip "No API key for Gemini found on this machine". providerCatalog appeared zero times in the frontend. So when the asset was unreachable, an internal packaging failure was rendered to you as a verified finding about your credentials — which is the worst kind of wrong, because it's unfalsifiable from your side.

There is now a third state. The API publishes keysVerified, and the Console renders ? / "Not checked — Brain could not load its provider catalog" instead of a confident ✗. providerKeyChip() returns that branch before it reads keys, so the false claim is structurally unreachable without a loaded catalog. It also matches vocabulary the Console already uses elsewhere ("Learning hooks / not checked"), so it isn't a new dialect.

The guard is the part I'd point at: it derives its required-asset set from the installer's own CONSOLE_RUNTIME_SURFACE declaration and checks it against real npm pack --dry-run --json output — no hardcoded list to drift. Two independently-maintained lists with nothing asserting they agree is precisely what shipped this bug. Proven by reintroducing your exact regression and watching the new guard fail.

[stuinfla]

Fixed and merged to main in 257cbb0 (release 4.0.9).

The root cause and the proof are in the comment above — each fix was verified red against the broken code before being accepted, not merely green after. Full CI on the merge commit: unit, integration, release-qe, warm-brain, windows-unit and ux-qe on ubuntu/macOS/Windows, all green.

Thank you for the report — the detail you gave is what made these fixable at the class level rather than patched one file at a time. If any of this behaves differently on your machine, please reopen.