Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,46 @@ All notable changes to Throttle will be documented in this file.

## [Unreleased]

## [0.8.0] - 2026-11-12

### Added
- `throttle scan [PATH]`: the first run, offline, with no API key, no
network and no traffic, and it writes nothing. It finds the direct OpenAI
and Anthropic SDK calls it can read (Python, JavaScript, TypeScript),
prices each at list price per 1,000 calls (or per month with
`--calls-per-day N`), and names up to two cheaper models to test: one tier
down from the same provider, then the other provider's model at a similar
price. Every dollar figure is labelled ESTIMATE (list price x an ASSUMED
1,000 tokens in and 300 out per call; `--tokens-in`/`--tokens-out` change
it, and a literal `max_tokens` in the code caps the output). Every cheaper
model is labelled UNTESTED. A model set at runtime is priced from a literal
default next to it (`os.getenv("MODEL", "gpt-4o")`, `?? "gpt-4o"`, a
parameter default), labelled "(default)", or from `--model NAME`, labelled
"(--model)"; otherwise it is listed, not priced. Tests, specs, examples,
evals and benchmarks are left out of the totals (`--all` puts them back).
The table shows the priced calls, most expensive first, up to 15. It ends
with the `capture`, `shadow`, `eval` and `report` commands that test the
first pick on real traffic (a Claude pick uses Anthropic's
OpenAI-compatible endpoint). `--json` prints schema `throttle.own.scan/1`.
Listed first in `throttle --help` and the welcome screen. Not in 0.7.0:
the README install block gains `throttle scan .` with the 0.8.0 release.

### Changed
- The code scanner (`throttle jobs scan`, `pr-check`) no longer walks into
`node_modules`, virtualenvs and the other skipped folders before skipping
their files, and skips a file without any AI call word before parsing it,
so it is much faster on large repos.
- The code scanner now also finds `client.messages.create(**params)` in a
file that imports `anthropic` (and `anthropic.messages.create(params)` in
JS/TS), calls through `with_raw_response` / `with_streaming_response`, and
reads the model and prompt from a `**params` dict literal defined in the
same function or module. A job found this way is new to `jobs scan` and
`pr-check`, and a job whose model or prompt is now read from such a dict
gets a new approval key, so it needs a new eval before it can move.
- `streamText(x)` and the other AI SDK names are no longer taken for an AI
call when the argument is not an options object (for example a local
`streamText(part.text)` helper).

## [0.7.0] - 2026-11-10

Stop renting your AI. First release of the loop on PyPI: `pip install "throttle-pro==0.7.0"`.
Expand Down
23 changes: 21 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,29 @@ MI300X with vLLM, [raw data](validation/hotaisle-mi300x-20261001/)):
## Try it in a minute

```sh
pip install "throttle-pro==0.7.0"
pip install "throttle-pro==0.8.0"
throttle scan .
throttle demo
```

Pin the version so what you install never changes under you.

`throttle scan .` reads the code in the current
folder and lists the OpenAI and Anthropic SDK calls it can read: where each
one is, which model it uses, what it costs per 1,000 calls at list price, and
up to two cheaper models to test. It needs no API key, no network and no
traffic, and it writes nothing. It takes seconds on a normal repo. Every
dollar figure is an ESTIMATE: list price times an assumed 1,000 tokens in and
300 out per call (change it with `--tokens-in` and `--tokens-out`; add
`--calls-per-day N` for a monthly figure). The cheaper models are UNTESTED.
Scan says they cost less, not that they do the job as well. Tests, examples
and evals are left out of the totals (`--all` puts them back). A model set at
runtime is priced from a default written in the code, or from `--model NAME`.
Calls through your own wrapper or an agent framework are not found; `throttle
capture` sees them on real traffic. Scan ends with the commands that prove it
on your own traffic ([The loop](#the-loop)). `--json` gives the same data for
tools (schema `throttle.own.scan/1`).

`throttle demo` runs the whole loop offline on made-up traffic: capture,
workloads, shadow, eval and report. It needs no API key, no GPU and no
network. It writes `throttle-demo/report.html`. Every number in it is
Expand Down Expand Up @@ -129,7 +146,7 @@ with the date each price was checked and the page it came from. `--prices
FILE` adds or replaces prices; those are labelled ASSUMED. The monthly figure
scales the capture window to 30 days and is refused for a window shorter than
a day. `--embed` also splits each kind into clusters by prompt similarity
(needs `pip install "throttle-pro[embeddings]==0.7.0"`).
(needs `pip install "throttle-pro[embeddings]==0.8.0"`).

### 3. Shadow

Expand Down Expand Up @@ -413,6 +430,8 @@ same function or file.
| LIST PRICE | A published API price, with the date it was checked and its source page. |
| ASSUMED | A number you gave us: your GPU hourly rate, or a price from `--prices`. |
| DEMO | Made up, from `throttle demo`. Not a measurement of anything. |
| ESTIMATE | From `throttle scan`: list price times a token count per call that is ASSUMED or that you gave. Not measured. |
| UNTESTED | A cheaper model `throttle scan` names. It costs less at list price. Nobody has checked yet that it does the job as well. |

`route savings --owned-cost-usd` is labelled OPERATOR-REPORTED.

Expand Down
4 changes: 2 additions & 2 deletions docs/ci-gate.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: python -m pip install "throttle-pro==0.7.0"
- run: python -m pip install "throttle-pro==0.8.0"
- name: Deploy main's config
run: ./deploy/inference/deploy.sh # your deploy; wait until /v1/models answers
- name: Measure the baseline 3 times (calibrates the noise bound)
Expand All @@ -102,7 +102,7 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- run: python -m pip install "throttle-pro==0.7.0"
- run: python -m pip install "throttle-pro==0.8.0"
- uses: actions/cache/restore@v4
with:
path: ${{ env.THROTTLE_CHECK_HISTORY_DIR }}
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

[project]
name = "throttle-pro"
version = "0.7.0"
version = "0.8.0"
description = "Own your intelligence: prove which share of your OpenAI/Anthropic traffic an open-weight model you run can handle without losing quality, move it with fallback, and measure the savings"
keywords = ["llm", "openai", "anthropic", "open-weight", "self-hosted", "inference-cost", "llm-evaluation", "llm-judge", "vllm", "sglang", "ollama", "model-routing"]
readme = "README.md"
Expand Down
2 changes: 1 addition & 1 deletion src/throttle/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
"""Throttle: safety-first measurement for OpenAI-compatible endpoints."""

__version__ = "0.7.0"
__version__ = "0.8.0"
20 changes: 13 additions & 7 deletions src/throttle/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,13 +81,18 @@
# The order `throttle --help` lists commands in: the money loop first, then
# the code-review commands, then the serving tools (advanced).
HELP_ORDER = (
"demo", "capture", "workloads", "shadow", "eval", "report", "route", "refresh",
"scan", "demo", "capture", "workloads", "shadow", "eval", "report", "route", "refresh",
"jobs", "pr-check", "pr-comment",
)

GETTING_STARTED = """\
Start here: the whole loop on made-up data (offline, no keys, every number
labelled DEMO):
Start here: find the AI calls in your code, what they cost at list price and
which cheaper models to test (offline, no keys, about 30 seconds; every
dollar is an ESTIMATE and every cheaper model is UNTESTED):
throttle scan .

Then the whole loop on made-up data (offline, no keys, every number labelled
DEMO):
throttle demo

The money loop, on your own traffic:
Expand Down Expand Up @@ -121,7 +126,7 @@
throttle refresh

In your code:
throttle jobs scan finds every OpenAI/Anthropic call and tracks it as a job.
throttle jobs scan finds the OpenAI/Anthropic SDK calls, one job each.
throttle pr-check the pull request comment (or the Throttle for GitHub
action): what an added or changed AI job costs, the
proof, and what happens next. A changed prompt is
Expand Down Expand Up @@ -2348,13 +2353,14 @@ def _render_watch_snap(snap) -> None:

WELCOME_TAGLINE = "Throttle: stop renting your AI. Cut the bill."
WELCOME_STEPS = (
("Find the AI calls in your code, offline, no keys:", None, (
"throttle scan .",
)),
("See the whole loop on made-up data:", "offline, no keys, every number labelled DEMO.", (
"throttle demo",
)),
("Capture your OpenAI / Anthropic traffic:", None, (
("Capture your traffic, see what each job costs:", None, (
"throttle capture",
)),
("See each kind of AI job and what it costs:", None, (
"throttle workloads",
)),
("Test a cheaper model on it, then eval it:", None, (
Expand Down
10 changes: 5 additions & 5 deletions src/throttle/console/docs/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@

Throttle needs **Python 3.11 or newer**. The command is `throttle`.

Install v0.7.0 from PyPI. Pin the version so what you install never changes under you.
Install v0.8.0 from PyPI. Pin the version so what you install never changes under you.

```bash pip
python3 -m pip install "throttle-pro==0.7.0"
python3 -m pip install "throttle-pro==0.8.0"
throttle demo
```

```bash pipx
pipx install "throttle-pro==0.7.0"
pipx install "throttle-pro==0.8.0"
throttle --version
```

Expand All @@ -21,15 +21,15 @@ That gives you the whole loop (`capture`, `workloads`, `shadow`, `eval`, `report
To install straight from the source you can read, use the release tag:

```bash
pip install "git+https://github.com/KushagraKanaujia/throttle@v0.7.0"
pip install "git+https://github.com/KushagraKanaujia/throttle@v0.8.0"
```


## Optional extras

| Extra | Install | What it adds |
| --- | --- | --- |
| `embeddings` | `pip install "throttle-pro[embeddings]==0.7.0"` | `throttle workloads --embed` (similarity clusters) and the opt-in semantic tier of the caching proxy. |
| `embeddings` | `pip install "throttle-pro[embeddings]==0.8.0"` | `throttle workloads --embed` (similarity clusters) and the opt-in semantic tier of the caching proxy. |

## Check your setup

Expand Down
16 changes: 12 additions & 4 deletions src/throttle/console/docs/getting-started/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,28 @@
## 1. Install

```bash
pip install "throttle-pro==0.7.0"
pip install "throttle-pro==0.8.0"
```

This pins v0.7.0. To install from the source tag instead, see Installation.
This pins v0.8.0. To install from the source tag instead, see Installation.

## 2. Run the demo
## 2. Scan your code

```bash
throttle scan .
```

It finds the OpenAI and Anthropic SDK calls in the folder, prices each one at list price, and names up to two cheaper models to test. Offline, no keys, in seconds. Every dollar is an ESTIMATE and every cheaper model is UNTESTED. See [Scan your code](scan.md).

## 3. Run the demo

```bash
throttle demo
```

It runs capture, workloads, shadow, eval and report on made-up traffic, offline, with no keys and no GPU. Open `throttle-demo/report.html`. Every number is labelled DEMO. See [Demo](../loop/demo.md).

## 3. Run it on your traffic
## 4. Run it on your traffic

```bash
# capture: a local pass-through proxy
Expand Down
70 changes: 70 additions & 0 deletions src/throttle/console/docs/getting-started/scan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# Scan your code

*Find the AI calls in your code, what each costs at list price, and which cheaper models to test. Offline, in seconds.*

```bash
throttle scan .
```

Needs Throttle 0.8.0 or later. It needs no API key, no network, no GPU and no traffic, and it writes nothing. It reads the Python, JavaScript and TypeScript files in the folder (skipping `node_modules`, virtualenvs, `dist`, `build` and similar) and finds the direct OpenAI and Anthropic SDK calls it can read, the same way `throttle jobs scan` does.

For each call it shows:

- **Where it is**: the job name (a `# throttle-job: name` tag, or the function) and `file:line`.
- **The model**: the literal name in the code. When the code sets it at runtime, the default written next to it (`os.getenv("MODEL", "gpt-4o")`, `process.env.MODEL ?? "gpt-4o"`, `def f(model="gpt-4o")`), marked "(default)", or the model you name with `--model NAME`, marked "(--model)".
- **What it costs**: per 1,000 calls at list price, from the bundled price table. With `--calls-per-day N` it shows dollars per month instead (N calls a day for each job, x 30 days).
- **Up to two cheaper models to test**, each marked UNTESTED.

The table lists the priced calls, most expensive first, up to 15 (`--all` lists every one). It ends with one summary line, for example "6 AI calls found, 4 priced. Moving the priced ones to the first candidate could cut about 61% of their list-price cost, if every call site runs equally often (ESTIMATE)", and the commands that test the first candidate on your own traffic.

Calls in tests, specs, examples, evals and benchmarks (`tests/`, `__tests__/`, `*.test.ts`, `test_*.py`, `examples/`, `evals/` and similar) are left out of the totals and counted on one line. `--all` puts them back. When every call is in one of them, they are all counted.

## What the numbers are

Every dollar figure is an **ESTIMATE**: list price times a token count per call.

| Input | Default | Change it with |
| --- | --- | --- |
| Input tokens per call | 1,000 (ASSUMED) | `--tokens-in N` |
| Output tokens per call | 300 (ASSUMED) | `--tokens-out N` |
| Calls per day, per job | none: shows $ per 1,000 calls | `--calls-per-day N` |
| Prices | bundled table, LIST PRICE with its date | `--prices FILE` (ASSUMED) |

When the code passes a literal output cap (`max_tokens`, `max_completion_tokens`, `max_output_tokens`, `maxTokens` or `maxOutputTokens`) that is smaller than the output tokens above, scan uses the cap and marks the model with `*`. Cached input is not counted.

## How the cheaper models are picked

All comparisons are list cost per call at the token counts above. A model must cost at least 10% less to be listed. Dated snapshots and models built only for code are left out.

1. **First pick, same provider.** One tier down in the same family: `gpt-4o` to `gpt-4o-mini`, `claude-sonnet-4-5` to `claude-haiku-4-5`. When the family has no cheaper tier, the same provider's next price step down.
2. **Second pick, other provider.** The model whose cost per call is closest to the first pick's, so it sits in the same price class.

> **Note:** A cheaper model is only cheaper. Scan does not test quality, so every pick is UNTESTED. A model with a lower list price can still cost more in real use if it writes longer answers or thinks longer. Prove it with the loop below, which uses measured tokens.

## What it cannot price

- **A model set at runtime with no default in the code** (`model=os.environ["CHAT_MODEL"]`, `model: config.model`, `create(**params)`). It is listed with the code that sets it. Rerun with `--model NAME` to price those calls as the model you run.
- **A model that is not in the price table**, such as a fine-tune, a deployment name, another provider's model or an older model the price page no longer lists. It is listed by name. Add a price with `--prices FILE`.
- **Calls through your own wrapper, another SDK or an agent framework.** They are not found. `throttle capture` sees them on real traffic.

When nothing is found, scan says what it looks for and exits 0.

## Prove it on your own traffic

```bash
throttle capture
export OPENAI_BASE_URL=http://127.0.0.1:8790/v1
export ANTHROPIC_BASE_URL=http://127.0.0.1:8790

# after real traffic has gone through
throttle shadow --candidate-url https://api.openai.com/v1 \
--api-key-env OPENAI_API_KEY --candidate-model gpt-4o-mini --label gpt-4o-mini
throttle eval --candidate gpt-4o-mini --judge-model gpt-4.1 --judge-key-env OPENAI_API_KEY
throttle report --candidate gpt-4o-mini --candidate-model gpt-4o-mini
```

Scan prints these commands with its own first pick filled in. When the pick is a Claude model, the shadow step uses `https://api.anthropic.com/v1` (Anthropic's OpenAI-compatible endpoint, offered for testing) with `ANTHROPIC_API_KEY`, and when the code has no OpenAI calls the judge is a Claude model (`--judge-provider anthropic`). Each step has its own page under [The loop](../loop/overview.md).

## For tools

`throttle scan . --json` prints the same data as JSON with `"schema": "throttle.own.scan/1"`: the assumptions, a summary, one entry per call, tests and examples included with `"counted": false` (with `status` `priced`, `model_set_at_runtime`, `model_not_in_price_table` or `not_priced_at_this_size`) and each candidate with `"status": "UNTESTED"`.
1 change: 1 addition & 0 deletions src/throttle/console/docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ capture ──► workloads ──► shadow ──► eval ──► report ─

## Start here

- **[Scan your code](getting-started/scan.md)**: `throttle scan .` finds the AI calls in your code, what each costs at list price and which cheaper models to test. Offline, no keys.
- **[Quickstart](getting-started/quickstart.md)**: Run the whole loop on made-up data in a minute, then on your own traffic.
- **[The loop](loop/overview.md)**: One page per stage, with every command and flag.
- **[Honest numbers](loop/honest-numbers.md)**: The labels, and the rules that keep a savings number defensible.
Expand Down
1 change: 1 addition & 0 deletions src/throttle/console/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
{"tab": "docs", "group": "Getting Started", "pages": [
("index", "Introduction"),
("getting-started/quickstart", "Quickstart"),
("getting-started/scan", "Scan your code"),
("getting-started/installation", "Installation"),
("getting-started/first-check", "Your first cost check"),
("getting-started/free-gpu", "Try on a free GPU"),
Expand Down
1 change: 1 addition & 0 deletions src/throttle/own/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ def handle(args) -> int # runs it; dispatched on args.command

# command name -> module path. Order is the order shown in --help.
STAGES: dict[str, str] = {
"scan": "throttle.own.scan_cost",
"capture": "throttle.own.capture",
"workloads": "throttle.own.workloads",
"shadow": "throttle.own.shadow",
Expand Down
Loading
Loading