fix(routing): honor an explicitly pinned model over the fallback toggle - #13
Merged
Conversation
Naming a model in the request body could silently serve a different one.
fallback_config.enabled governs participation in AUTOMATIC fallback, but
routeRequestInternal applied it to every candidate, including one the client
pinned by name. resolveRoutableModel does not consult that table, so the pin
validated fine and was then skipped at dispatch — the request fell through to
the next model in the chain and returned 200.
Production has 39 chat-capable models in exactly this state: enabled in the
catalog, switched off in the chain, therefore unreachable even when named.
Among them groq/llama-3.1-8b-instant, both Groq compound models,
cerebras/qwen-3-235b, both Cohere command models, six Ollama frontier models
and nvidia/nemotron-3-super-120b-a12b. Reproduced against the live router:
pinning nemotron-3-super returned google/gemini-2.5-flash with no NVIDIA
attempt logged at all.
routeRequest now takes a separate pinnedModelDbId that exempts that one model
from the enabled check. Kept deliberately narrow:
- sticky-session preference still respects the toggle, since it is an
implicit choice rather than something the caller asked for;
- models.enabled is untouched, so a retired row stays unreachable however
it is requested;
- automatic routing still skips disabled entries.
Also widened X-Fallback-Reason. With no upstream error it claimed
"rate-limited or no healthy key", which sent people looking at keys and
quotas when the cause was a runtime-health block or a context window too
small for the request. It now names every pre-dispatch skip reason.
Verified by reverting the router guard: the pinned model resolves to the
other one and the new test fails.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Naming a model in the request body could silently serve a different one. Confirmed against the live router and reported independently by the maintainer.
The bug
fallback_config.enabledgoverns participation in automatic fallback, butrouteRequestInternalapplied it to every candidate — including one the client pinned by name.resolveRoutableModeldoesn't consult that table, so the pin validated fine and was then skipped at dispatch, falling through to the next model and returning 200.Reproduced on production: pinning
nvidia/nemotron-3-super-120b-a12breturnedgoogle/gemini-2.5-flash, with no NVIDIA attempt logged at all.Scope on the live VPS
39 chat-capable models are enabled in the catalog but switched off in the chain, so they were unreachable even when named explicitly:
llama-3.1-8b-instant,groq/compound,groq/compound-mini,gpt-oss-120b,gpt-oss-20b,llama-4-scout,qwen3-32bmistral-large-3:675b,cogito-2.1:671b,deepseek-v3.2,kimi-k2-thinking,glm-4.7,devstral-2:123bqwen-3-235b-a22b-instruct-2507,llama3.1-8bcommand-a-03-2025,command-r-plus-08-2024nemotron-3-super-120b-a12b,gemma-4-31b-itDeepSeek-V3.1-cb,Llama-4-Maverick,gemma-3-12b-itA further 8 rows are in the same state but are correctly off — embeddings, image and video models are reached through
routeCapabilityRequest, which readsmodel_capabilities, not the chat chain. Those are untouched.The fix
routeRequesttakes a separatepinnedModelDbIdthat exempts just that model from the enabled check. Deliberately narrow:models.enabledis untouched, so a retired row stays unreachable however it's requestedAlso: a misleading diagnostic
With no upstream error,
X-Fallback-Reasonclaimed"rate-limited or no healthy key"— sending people to hunt through keys and quotas when the real cause was a runtime-health block or a context window too small for the request. It now names every pre-dispatch skip reason.Verification
tscclean🤖 Generated with Claude Code