Skip to content

Add throttle fit: the fewest GPUs your model needs (offline estimate) - #95

Merged
KushagraKanaujia merged 2 commits into
mainfrom
feat/fit
Nov 17, 2026
Merged

KushagraKanaujia merged 2 commits into
mainfrom
feat/fit

Conversation

@KushagraKanaujia

Copy link
Copy Markdown
Owner

What

throttle fit: the fewest GPUs a model needs at your context length and number of chats at once, per GPU type. Offline, no GPU, no keys, labelled ESTIMATE, never predicts speed. With --price it shows monthly cost; with --current h100=4 it says whether fewer cards look like enough and what that would save if a measurement agrees. It always ends with the throttle autotune (or throttle check for llama.cpp) command that measures the claim on the user's own machine.

throttle fit --model llama-3.3-70b --quant fp8 --context 8k --users 16 --gpu h100 --price h100=2.49 --current h100=4

Why (customer evidence)

  • A Reddit commenter shared a home rig benchmark repo: their first question is always "does this model fit at my context on my cards", and in their tests the fewest cards that fit ran fastest.
  • AlexsJones/llmfit (about 39k stars) answers "what runs on my hardware"; its users ask for concurrent user estimates and report speed estimates about 3x too high. So: estimate memory, never speed, then measure.
  • Our own WHAT_TO_BUILD pick Add lightweight user feedback tracking #3 ("Fewer GPUs" check).

Drift test

  1. Layer: Measure (free funnel), handing off to Decide (autotune).
  2. Metric: paid pilots. A spare card finding on --current is a concrete pilot opener; slow, not this week.
  3. Dataset: indirectly, it routes users into autotune/check, which produce measured rows.
  4. vs vLLM tooling: vLLM tells you if a load fails, not the fewest cards across GPU types at your prices.
  5. Smallest version: one module, no network, no new dependencies.

Review

An independent reviewer checked the math and the 11 built-in model configs against the published config.json files, and found 7 bugs (autotune command missing --gpu-hourly-rate, llama.cpp rate N times too low, illegal tp in --space, --current claim not measured, wrong wording when fewer cards than needed, nan/inf crashes, fp8/AWQ embeddings undercounted). All are fixed, with regression tests.

Tests

pytest: 1406 passed, 8 skipped. tests/test_coldstart.py failed once in a full run and passed on rerun, with and without this change (flaky, unrelated).

🤖 Generated with Claude Code

Kushagra Kanaujia and others added 2 commits November 16, 2026 22:42
Memory math per GPU type (weights plus full-context KV for every chat at
once, tensor parallel limits, KV copies past the KV head count), monthly
cost at your prices, a --current check for spare cards, and the autotune
command that measures the shortlist. Labelled ESTIMATE; never predicts speed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Always give autotune a rate (placeholder when unpriced); total rate for
llama.cpp check; only legal tp counts in --space and never a single value;
measure the user's own card when --current is given, and never pick a GPU
that only came in through --current; needs_more status; reject inf/nan and
negative params; keep embeddings 16 bit for fp8/AWQ/GPTQ; treat sold GB as
GiB; pass context, fp8 and concurrency into the measure command; Qwen2.5
14B/32B parameter counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@KushagraKanaujia
KushagraKanaujia merged commit 99263ec into main Nov 17, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant