fix(routing): bench a denied credential across all models, not one - #12
Merged
Merged
Conversation
Every Google key reads "healthy" in the dashboard while one of them fails
every real request. Probing all 16 live keys on the VPS found the cause and
identified the bad one.
WHY THE DASHBOARD LIES. GoogleProvider.validateKey calls GET /models, a
listing endpoint. A project answering "403: Your project has been denied
access" on generateContent still returns 200 from GET /models — and, tested
alongside it, 200 from countTokens too. No free endpoint distinguishes the
denial; only generateContent does, and health.ts sweeps every 5 minutes, so
probing it there would burn ~4,600 free-tier requests a day purely on health
checks. validateKey is left as a liveness check and now says so.
WHY IT REPEATED 50 TIMES. classifyProviderError matched the message on '403'
and returned model_unavailable — a per-model verdict for an account-level
fact. Cooldowns are keyed by (platform, model, key), so the router benched
the credential for the one model that failed and then re-selected it for the
next model, and the next. One denied project therefore produced 50 logged
403s spread across the catalog.
Account-level denials ("project has been denied access", "organization has
been restricted", "account suspended") are now classified as auth failures
carrying cooldownScope: 'key', and setKeyCooldown benches the credential for
every model on its platform for 24h. skipModel stays false: the model is
fine, every other key serves it. The pattern is matched before the generic
403 branch, which would otherwise swallow it.
Verified by reverting the classifier pattern — the denial falls back to
model_unavailable and the new tests fail.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every Google key reads healthy in the dashboard while one of them fails every real request. I probed all 16 live keys on the VPS, found the cause, and identified the bad one.
The bad key:
Alisaad2026.workAlisaad2026.work15 of 16 keys are fine. (Probe ran on the server, decrypted locally, and printed only labels and status codes — no key material left the box, and the script was deleted afterwards.)
Why the dashboard lies
GoogleProvider.validateKeycallsGET /models— a listing endpoint. A denied project still gets 200 from it, and — tested alongside — 200 fromcountTokenstoo. No free endpoint distinguishes the denial. OnlygenerateContentdoes, andhealth.tssweeps every 5 minutes, so probing it there would burn ~4,600 free-tier requests a day purely on health checks.So
validateKeystays a liveness check and now documents exactly that, with the measurements behind the decision. The denial is caught where it actually surfaces instead.Why it repeated 50 times
classifyProviderErrormatched the message on'403'and returnedmodel_unavailable— a per-model verdict for an account-level fact. Cooldowns are keyed by(platform, model, key), so the router benched the credential for the one model that failed, then re-selected it for the next model, and the next. One denied project produced 50 logged 403s spread across the catalog.Account-level denials —
project has been denied access,organization has been restricted,account suspended— are now classified as auth failures carryingcooldownScope: 'key', and the newsetKeyCooldownbenches that credential across every model on its platform for 24h.skipModelstaysfalseon purpose: the model is fine, every other key serves it. The pattern is matched before the generic 403 branch, which would otherwise swallow it.Verification
tsccleanmodel_unavailableand the new tests failNote
This fixes the routing waste, not the key itself —
Alisaad2026.workstill needs to be replaced or removed in the dashboard. With this change the router will bench it for 24h after the first 403 instead of retrying it once per model.🤖 Generated with Claude Code