Repository navigation
Add throttle fit: the fewest GPUs your model needs (offline estimate) - #95
Merged
Merged
Conversation
Memory math per GPU type (weights plus full-context KV for every chat at once, tensor parallel limits, KV copies past the KV head count), monthly cost at your prices, a --current check for spare cards, and the autotune command that measures the shortlist. Labelled ESTIMATE; never predicts speed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Always give autotune a rate (placeholder when unpriced); total rate for llama.cpp check; only legal tp counts in --space and never a single value; measure the user's own card when --current is given, and never pick a GPU that only came in through --current; needs_more status; reject inf/nan and negative params; keep embeddings 16 bit for fp8/AWQ/GPTQ; treat sold GB as GiB; pass context, fp8 and concurrency into the measure command; Qwen2.5 14B/32B parameter counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
throttle fit: the fewest GPUs a model needs at your context length and number of chats at once, per GPU type. Offline, no GPU, no keys, labelled ESTIMATE, never predicts speed. With--priceit shows monthly cost; with--current h100=4it says whether fewer cards look like enough and what that would save if a measurement agrees. It always ends with thethrottle autotune(orthrottle checkfor llama.cpp) command that measures the claim on the user's own machine.Why (customer evidence)
AlexsJones/llmfit(about 39k stars) answers "what runs on my hardware"; its users ask for concurrent user estimates and report speed estimates about 3x too high. So: estimate memory, never speed, then measure.Drift test
autotune).--currentis a concrete pilot opener; slow, not this week.autotune/check, which produce measured rows.Review
An independent reviewer checked the math and the 11 built-in model configs against the published config.json files, and found 7 bugs (autotune command missing
--gpu-hourly-rate, llama.cpp rate N times too low, illegal tp in--space,--currentclaim not measured, wrong wording when fewer cards than needed, nan/inf crashes, fp8/AWQ embeddings undercounted). All are fixed, with regression tests.Tests
pytest: 1406 passed, 8 skipped.tests/test_coldstart.pyfailed once in a full run and passed on rerun, with and without this change (flaky, unrelated).🤖 Generated with Claude Code