Skip to content

feat(scout): retire models the probe finds gone on repeated checks - #14

Merged
aliihsaad merged 2 commits into
mainfrom
feat/scout-auto-retire
Aug 24, 2026
Merged

feat(scout): retire models the probe finds gone on repeated checks#14
aliihsaad merged 2 commits into
mainfrom
feat/scout-auto-retire

Conversation

@aliihsaad

Copy link
Copy Markdown
Owner

Closes the gap that made this whole week of manual catalog cleanup necessary.

The scout was probing and then ignoring the result

It has run every 30 minutes for months. But the only function that disables dead rows, applyPricingDrift, is gated to one platform:

if (platform === 'bazaarlink') { applyPricingDrift(...) }

NVIDIA, OpenRouter, Hugging Face, LLM7 and Google were probed, classified, recorded — and never acted on. That is why 4 end-of-lifed NVIDIA models, 159 dead Hugging Face rows and 15 discontinued OpenRouter :free routes stayed enabled until they were removed by hand in #9, #10 and #12.

Two gaps

1. HTTP 410 was never recognised. checkModelAvailability matched only '404' and 'not found', so NVIDIA's end-of-life responses fell through to 'error' — invisible to any retirement logic. isGoneMessage() now covers the phrasings confirmed live on 2026-08-23 across NVIDIA, OpenRouter, LLM7, Cerebras and Google.

2. Nothing acted on a gone verdict. model_availability gains gone_streak; a model is disabled after 3 consecutive gone probes (~90 minutes across three independent checks). Any other result resets the counter.

Safety properties, each with a test

  • 401/403 never counts as gone. A lapsed key answers 401 for every model on a platform — counting it would retire an entire provider three cycles after a key expired. Given the Groq org restriction and the denied Google project currently live on the VPS, this one matters.
  • 402, 429, timeouts and 5xx reset the streak. Quota, rate limit and outage are not deprecation.
  • Presence reconciliation against a bulk /models list is deliberately not used. OpenRouter's list returns chat models only, so diffing against it reports embedding and image rows as missing — the exact false positive that nearly retired text-embedding-3-small, both riverflow rows and flux.2-klein-4b in fix(catalog): retire dead providers and ended free tiers (V22) #10.
  • Retirement is announced once, not on every subsequent probe.

Verification

  • 333/333 server tests pass, tsc clean
  • 20 new tests, including every live error string captured this week as a positive or negative case
  • Verified by reverting the auth guard in isGoneMessage: a 401 is then treated as gone and the test fails

Note on the original request

There is no Hermes cron job for llm-hub model maintenance — it does not exist in either profile, and the weekly self-maintenance script only pings /health. Hermes cron itself is healthy (the stale ticker_heartbeat under /root/.hermes/cron/ is an abandoned default-profile store; the active jarvisvps profile ticks normally). A cron job would have duplicated a scheduler that already runs — the gap was in what the scheduler did with its results.

🤖 Generated with Claude Code

The scout has probed every model every 30 minutes for months and never
retired anything. The only function that disables dead rows, applyPricingDrift,
is gated to one platform:

    if (platform === 'bazaarlink') { applyPricingDrift(...) }

So NVIDIA, OpenRouter, Hugging Face, LLM7 and Google were probed, classified,
recorded — and never acted on. That is why 4 end-of-lifed NVIDIA models, 159
dead Hugging Face rows and 15 discontinued OpenRouter :free routes stayed
enabled until they were removed by hand.

Two gaps fixed.

1. HTTP 410 was never recognised. checkModelAvailability matched only '404'
   and 'not found', so NVIDIA's end-of-life responses fell through to 'error'
   and were invisible to any retirement logic. isGoneMessage() now covers the
   phrasings confirmed live on 2026-08-23 across NVIDIA, OpenRouter, LLM7,
   Cerebras and Google.

2. Nothing acted on a gone verdict. model_availability gains gone_streak, and
   a model is disabled after GONE_STREAK_TO_RETIRE (3) consecutive gone
   probes — about 90 minutes across three independent checks. Any other result
   resets the counter, so an upstream blip cannot retire anything.

Safety properties, each covered by a test:

  - 401/403 never counts as gone. A lapsed key answers 401 for every model on
    a platform, and counting that would retire a whole provider three cycles
    after the key expired.
  - 402, 429, timeouts and 5xx all reset the streak: quota, rate limit and
    outage are not deprecation.
  - Presence reconciliation against a provider's bulk /models list is
    deliberately NOT used. OpenRouter's list returns chat models only, so
    diffing against it reports embedding and image rows as missing — the exact
    false positive that nearly retired five live models earlier.

Verified by reverting the auth guard in isGoneMessage: a 401 is then treated
as gone and the test fails.
@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 548f180a-95ca-4a7a-95e6-e794ec22c95d


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@aliihsaad
aliihsaad merged commit 672d9b8 into main Aug 24, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant