Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,19 @@ All notable changes to Throttle will be documented in this file.

## [Unreleased]

### Added
- `throttle fit`: the fewest GPUs your model needs at your context length
and number of chats at once, per GPU type, before you rent anything.
Offline, no GPU, no keys. Memory math only (weights plus the KV cache for
every chat at the full context, with vLLM/SGLang's tensor parallel limits
and the KV copies when cards outnumber KV heads), labelled ESTIMATE, and it
never predicts speed. `--price NAME=$/HOUR` adds the monthly cost of each
option; `--current h100=4` says if fewer cards look like enough and what
that would save if a measurement agrees. It ends with the `throttle
autotune` (or, for llama.cpp, `throttle check`) command that measures the
shortlist. 11 built-in models, 12 built-in GPUs, `--config config.json`
for any other model, `--gpu name=GB` for any other card, `--json`.

## [0.8.0] - 2026-11-12

### Added
Expand Down
17 changes: 16 additions & 1 deletion src/throttle/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,8 @@
never moved silently.

Serving tools (advanced), for the server your own model runs on:
check ($/M tokens with 95% CIs), autotune, coldstart, compare, gate,
fit (fewest GPUs for your model, offline estimate), check ($/M tokens with
95% CIs), autotune, coldstart, compare, gate,
savings, watch, ui, proxy, sessions. Evidence runs: plan, smoke, benchmark,
golden, golden-report.

Expand Down Expand Up @@ -657,6 +658,17 @@ def build_parser() -> argparse.ArgumentParser:
)
add_coldstart_arguments(coldstart)

from .fit import FIT_DESCRIPTION, FIT_EPILOG, add_fit_arguments
fit = subparsers.add_parser(
"fit",
help="the fewest GPUs your model needs at your context and users (offline "
"estimate, no GPU); then the command to measure it",
description=FIT_DESCRIPTION,
epilog=FIT_EPILOG,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
add_fit_arguments(fit)

from .gate import GATE_DESCRIPTION, add_gate_arguments
gate = subparsers.add_parser(
"gate",
Expand Down Expand Up @@ -2488,6 +2500,9 @@ def main(argv: Sequence[str] | None = None) -> int:
if args.command == "autotune":
from .autotune import handle_autotune
return handle_autotune(args)
if args.command == "fit":
from .fit import handle_fit
return handle_fit(args)
if args.command == "coldstart":
from .coldstart import handle_coldstart
return handle_coldstart(args)
Expand Down
49 changes: 49 additions & 0 deletions src/throttle/console/docs/guides/fewest-gpus.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Fewest GPUs for your model

*Before you rent cards: how many does your model need at your traffic?*

`throttle fit` is memory math. It runs on your laptop in a second, with no GPU and no keys, and every number it prints is an **estimate**.

```bash
throttle fit --model llama-3.3-70b --quant fp8 --context 32k --users 16 \
--gpu h100 --gpu l40s --price h100=2.49 --price l40s=0.99 --current h100=4
```

For each GPU it shows:

| Column | Meaning |
|---|---|
| fewest cards | the smallest card count that holds the weights plus a full-length KV cache for every chat at once |
| chats they hold | how many full-length chats that many cards can hold |
| per month | that many cards at your `--price`, 730 hours a month |

With `--current`, it tells you if fewer cards look like enough, and what that would save **if a measurement agrees**.

### What it does not know

- **Speed.** More cards can be slower, because they have to talk to each other. Measure before you change anything.
- **Your real chat lengths.** It assumes every chat at the full `--context` at the same time. Real traffic is usually lighter, so this errs on the safe side.
- **Weight sizes are approximate.** fp8, AWQ and GPTQ keep the embeddings in 16 bit, which is counted; other layers left unquantized are not.
- **Exotic attention.** Sliding-window and latent attention models need less KV memory than this math assumes.

### Then measure it

`throttle fit` ends with the command to run on your server, for example:

```bash
throttle autotune --url http://localhost:8000 --model meta-llama/Llama-3.3-70B-Instruct \
--gpu-hourly-rate 2.49 --concurrency 16 \
--launch "vllm serve {model} --tensor-parallel-size {tp} --max-model-len 32768 --quantization fp8" \
--space tp=2,4
```

It measures cost per million tokens on your server for each card count, with a range that shows how sure the measurement is. Run it on the machine type you actually rent; replace `PRICE_PER_GPU_HOUR` if you did not give `--price`.

### Any model, any card

```bash
throttle fit --config ./config.json --params 32.8 --quant awq --context 8k --users 8
throttle fit --model qwen2.5-32b --quant q4_k_m --engine llamacpp --gpu rtx3090 --gpu 1060=6
```

`--list-models` and `--list-gpus` show what is built in. `--json` prints the result for scripts.
1 change: 1 addition & 0 deletions src/throttle/console/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@
("guides/sglang", "SGLang"),
("guides/ollama", "Ollama"),
("guides/other-engines", "Other servers"),
("guides/fewest-gpus", "Fewest GPUs for your model"),
("guides/compare-gpus", "Compare two GPUs"),
("guides/test-quantization", "Test a quantization"),
("guides/ci-github-action", "Gate costs in CI"),
Expand Down
Loading
Loading