feat(scout): retire models the probe finds gone on repeated checks - #14
Merged
Conversation
The scout has probed every model every 30 minutes for months and never
retired anything. The only function that disables dead rows, applyPricingDrift,
is gated to one platform:
if (platform === 'bazaarlink') { applyPricingDrift(...) }
So NVIDIA, OpenRouter, Hugging Face, LLM7 and Google were probed, classified,
recorded — and never acted on. That is why 4 end-of-lifed NVIDIA models, 159
dead Hugging Face rows and 15 discontinued OpenRouter :free routes stayed
enabled until they were removed by hand.
Two gaps fixed.
1. HTTP 410 was never recognised. checkModelAvailability matched only '404'
and 'not found', so NVIDIA's end-of-life responses fell through to 'error'
and were invisible to any retirement logic. isGoneMessage() now covers the
phrasings confirmed live on 2026-08-23 across NVIDIA, OpenRouter, LLM7,
Cerebras and Google.
2. Nothing acted on a gone verdict. model_availability gains gone_streak, and
a model is disabled after GONE_STREAK_TO_RETIRE (3) consecutive gone
probes — about 90 minutes across three independent checks. Any other result
resets the counter, so an upstream blip cannot retire anything.
Safety properties, each covered by a test:
- 401/403 never counts as gone. A lapsed key answers 401 for every model on
a platform, and counting that would retire a whole provider three cycles
after the key expired.
- 402, 429, timeouts and 5xx all reset the streak: quota, rate limit and
outage are not deprecation.
- Presence reconciliation against a provider's bulk /models list is
deliberately NOT used. OpenRouter's list returns chat models only, so
diffing against it reports embedding and image rows as missing — the exact
false positive that nearly retired five live models earlier.
Verified by reverting the auth guard in isGoneMessage: a 401 is then treated
as gone and the test fails.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the gap that made this whole week of manual catalog cleanup necessary.
The scout was probing and then ignoring the result
It has run every 30 minutes for months. But the only function that disables dead rows,
applyPricingDrift, is gated to one platform:NVIDIA, OpenRouter, Hugging Face, LLM7 and Google were probed, classified, recorded — and never acted on. That is why 4 end-of-lifed NVIDIA models, 159 dead Hugging Face rows and 15 discontinued OpenRouter
:freeroutes stayed enabled until they were removed by hand in #9, #10 and #12.Two gaps
1. HTTP 410 was never recognised.
checkModelAvailabilitymatched only'404'and'not found', so NVIDIA's end-of-life responses fell through to'error'— invisible to any retirement logic.isGoneMessage()now covers the phrasings confirmed live on 2026-08-23 across NVIDIA, OpenRouter, LLM7, Cerebras and Google.2. Nothing acted on a gone verdict.
model_availabilitygainsgone_streak; a model is disabled after 3 consecutive gone probes (~90 minutes across three independent checks). Any other result resets the counter.Safety properties, each with a test
/modelslist is deliberately not used. OpenRouter's list returns chat models only, so diffing against it reports embedding and image rows as missing — the exact false positive that nearly retiredtext-embedding-3-small, bothriverflowrows andflux.2-klein-4bin fix(catalog): retire dead providers and ended free tiers (V22) #10.Verification
tsccleanisGoneMessage: a 401 is then treated as gone and the test failsNote on the original request
There is no Hermes cron job for llm-hub model maintenance — it does not exist in either profile, and the weekly self-maintenance script only pings
/health. Hermes cron itself is healthy (the staleticker_heartbeatunder/root/.hermes/cron/is an abandoned default-profile store; the activejarvisvpsprofile ticks normally). A cron job would have duplicated a scheduler that already runs — the gap was in what the scheduler did with its results.🤖 Generated with Claude Code