Skip to content

fix(routing): honor an explicitly pinned model over the fallback toggle - #13

Merged
aliihsaad merged 2 commits into
mainfrom
fix/honor-pinned-model
Aug 23, 2026
Merged

fix(routing): honor an explicitly pinned model over the fallback toggle#13
aliihsaad merged 2 commits into
mainfrom
fix/honor-pinned-model

Conversation

@aliihsaad

Copy link
Copy Markdown
Owner

Naming a model in the request body could silently serve a different one. Confirmed against the live router and reported independently by the maintainer.

The bug

fallback_config.enabled governs participation in automatic fallback, but routeRequestInternal applied it to every candidate — including one the client pinned by name. resolveRoutableModel doesn't consult that table, so the pin validated fine and was then skipped at dispatch, falling through to the next model and returning 200.

Reproduced on production: pinning nvidia/nemotron-3-super-120b-a12b returned google/gemini-2.5-flash, with no NVIDIA attempt logged at all.

Scope on the live VPS

39 chat-capable models are enabled in the catalog but switched off in the chain, so they were unreachable even when named explicitly:

Provider Examples
groq llama-3.1-8b-instant, groq/compound, groq/compound-mini, gpt-oss-120b, gpt-oss-20b, llama-4-scout, qwen3-32b
ollama mistral-large-3:675b, cogito-2.1:671b, deepseek-v3.2, kimi-k2-thinking, glm-4.7, devstral-2:123b
cerebras qwen-3-235b-a22b-instruct-2507, llama3.1-8b
cohere command-a-03-2025, command-r-plus-08-2024
nvidia nemotron-3-super-120b-a12b, gemma-4-31b-it
sambanova DeepSeek-V3.1-cb, Llama-4-Maverick, gemma-3-12b-it

A further 8 rows are in the same state but are correctly off — embeddings, image and video models are reached through routeCapabilityRequest, which reads model_capabilities, not the chat chain. Those are untouched.

The fix

routeRequest takes a separate pinnedModelDbId that exempts just that model from the enabled check. Deliberately narrow:

  • Sticky-session preference still respects the toggle — it's an implicit choice, not something the caller asked for
  • models.enabled is untouched, so a retired row stays unreachable however it's requested
  • Automatic routing still skips disabled entries, so the Fallback Chain UI keeps working as designed

Also: a misleading diagnostic

With no upstream error, X-Fallback-Reason claimed "rate-limited or no healthy key" — sending people to hunt through keys and quotas when the real cause was a runtime-health block or a context window too small for the request. It now names every pre-dispatch skip reason.

Verification

  • 313/313 server tests pass, tsc clean
  • Verified by reverting the router guard: the pinned model resolves to the other one and the new test fails
  • Tests cover the pin bypass, that sticky preference does not bypass, that a catalog-disabled model stays unreachable when pinned, and that auto-routing still skips disabled entries

🤖 Generated with Claude Code

Naming a model in the request body could silently serve a different one.

fallback_config.enabled governs participation in AUTOMATIC fallback, but
routeRequestInternal applied it to every candidate, including one the client
pinned by name. resolveRoutableModel does not consult that table, so the pin
validated fine and was then skipped at dispatch — the request fell through to
the next model in the chain and returned 200.

Production has 39 chat-capable models in exactly this state: enabled in the
catalog, switched off in the chain, therefore unreachable even when named.
Among them groq/llama-3.1-8b-instant, both Groq compound models,
cerebras/qwen-3-235b, both Cohere command models, six Ollama frontier models
and nvidia/nemotron-3-super-120b-a12b. Reproduced against the live router:
pinning nemotron-3-super returned google/gemini-2.5-flash with no NVIDIA
attempt logged at all.

routeRequest now takes a separate pinnedModelDbId that exempts that one model
from the enabled check. Kept deliberately narrow:

  - sticky-session preference still respects the toggle, since it is an
    implicit choice rather than something the caller asked for;
  - models.enabled is untouched, so a retired row stays unreachable however
    it is requested;
  - automatic routing still skips disabled entries.

Also widened X-Fallback-Reason. With no upstream error it claimed
"rate-limited or no healthy key", which sent people looking at keys and
quotas when the cause was a runtime-health block or a context window too
small for the request. It now names every pre-dispatch skip reason.

Verified by reverting the router guard: the pinned model resolves to the
other one and the new test fails.
@coderabbitai

coderabbitai Bot commented Aug 23, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 77d55678-80e3-46ba-8d02-b9b989d7a5d8


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@aliihsaad
aliihsaad merged commit c08517d into main Aug 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant