Skip to content

fix(routing): bench a denied credential across all models, not one - #12

Merged
aliihsaad merged 2 commits into
mainfrom
fix/account-denial-key-cooldown
Aug 23, 2026
Merged

aliihsaad merged 2 commits into
mainfrom
fix/account-denial-key-cooldown

Conversation

@aliihsaad

Copy link
Copy Markdown
Owner

Every Google key reads healthy in the dashboard while one of them fails every real request. I probed all 16 live keys on the VPS, found the cause, and identified the bad one.

The bad key: Alisaad2026.work

GET /models countTokens generateContent
Alisaad2026.work 200 200 403 — "Your project has been denied access"
the other 15 200 200 200

15 of 16 keys are fine. (Probe ran on the server, decrypted locally, and printed only labels and status codes — no key material left the box, and the script was deleted afterwards.)

Why the dashboard lies

GoogleProvider.validateKey calls GET /models — a listing endpoint. A denied project still gets 200 from it, and — tested alongside — 200 from countTokens too. No free endpoint distinguishes the denial. Only generateContent does, and health.ts sweeps every 5 minutes, so probing it there would burn ~4,600 free-tier requests a day purely on health checks.

So validateKey stays a liveness check and now documents exactly that, with the measurements behind the decision. The denial is caught where it actually surfaces instead.

Why it repeated 50 times

classifyProviderError matched the message on '403' and returned model_unavailable — a per-model verdict for an account-level fact. Cooldowns are keyed by (platform, model, key), so the router benched the credential for the one model that failed, then re-selected it for the next model, and the next. One denied project produced 50 logged 403s spread across the catalog.

Account-level denials — project has been denied access, organization has been restricted, account suspended — are now classified as auth failures carrying cooldownScope: 'key', and the new setKeyCooldown benches that credential across every model on its platform for 24h.

skipModel stays false on purpose: the model is fine, every other key serves it. The pattern is matched before the generic 403 branch, which would otherwise swallow it.

Verification

  • 309/309 server tests pass, tsc clean
  • Verified by reverting the classifier pattern — the denial falls back to model_unavailable and the new tests fail
  • Tests cover the classification, key-scoped cooldown isolation (other keys and other platforms untouched), expiry, and that an ordinary 403 stays per-model

Note

This fixes the routing waste, not the key itself — Alisaad2026.work still needs to be replaced or removed in the dashboard. With this change the router will bench it for 24h after the first 403 instead of retrying it once per model.

🤖 Generated with Claude Code

Every Google key reads "healthy" in the dashboard while one of them fails
every real request. Probing all 16 live keys on the VPS found the cause and
identified the bad one.

WHY THE DASHBOARD LIES. GoogleProvider.validateKey calls GET /models, a
listing endpoint. A project answering "403: Your project has been denied
access" on generateContent still returns 200 from GET /models — and, tested
alongside it, 200 from countTokens too. No free endpoint distinguishes the
denial; only generateContent does, and health.ts sweeps every 5 minutes, so
probing it there would burn ~4,600 free-tier requests a day purely on health
checks. validateKey is left as a liveness check and now says so.

WHY IT REPEATED 50 TIMES. classifyProviderError matched the message on '403'
and returned model_unavailable — a per-model verdict for an account-level
fact. Cooldowns are keyed by (platform, model, key), so the router benched
the credential for the one model that failed and then re-selected it for the
next model, and the next. One denied project therefore produced 50 logged
403s spread across the catalog.

Account-level denials ("project has been denied access", "organization has
been restricted", "account suspended") are now classified as auth failures
carrying cooldownScope: 'key', and setKeyCooldown benches the credential for
every model on its platform for 24h. skipModel stays false: the model is
fine, every other key serves it. The pattern is matched before the generic
403 branch, which would otherwise swallow it.

Verified by reverting the classifier pattern — the denial falls back to
model_unavailable and the new tests fail.
@coderabbitai

coderabbitai Bot commented Aug 23, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: adea3145-cd71-498c-8990-332143948789


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@aliihsaad
aliihsaad merged commit b7ac5eb into main Aug 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant