diff --git a/CHANGELOG.md b/CHANGELOG.md index 38c47bf..50a5305 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,46 @@ All notable changes to Throttle will be documented in this file. ## [Unreleased] +## [0.8.0] - 2026-11-12 + +### Added +- `throttle scan [PATH]`: the first run, offline, with no API key, no + network and no traffic, and it writes nothing. It finds the direct OpenAI + and Anthropic SDK calls it can read (Python, JavaScript, TypeScript), + prices each at list price per 1,000 calls (or per month with + `--calls-per-day N`), and names up to two cheaper models to test: one tier + down from the same provider, then the other provider's model at a similar + price. Every dollar figure is labelled ESTIMATE (list price x an ASSUMED + 1,000 tokens in and 300 out per call; `--tokens-in`/`--tokens-out` change + it, and a literal `max_tokens` in the code caps the output). Every cheaper + model is labelled UNTESTED. A model set at runtime is priced from a literal + default next to it (`os.getenv("MODEL", "gpt-4o")`, `?? "gpt-4o"`, a + parameter default), labelled "(default)", or from `--model NAME`, labelled + "(--model)"; otherwise it is listed, not priced. Tests, specs, examples, + evals and benchmarks are left out of the totals (`--all` puts them back). + The table shows the priced calls, most expensive first, up to 15. It ends + with the `capture`, `shadow`, `eval` and `report` commands that test the + first pick on real traffic (a Claude pick uses Anthropic's + OpenAI-compatible endpoint). `--json` prints schema `throttle.own.scan/1`. + Listed first in `throttle --help` and the welcome screen. Not in 0.7.0: + the README install block gains `throttle scan .` with the 0.8.0 release. + +### Changed +- The code scanner (`throttle jobs scan`, `pr-check`) no longer walks into + `node_modules`, virtualenvs and the other skipped folders before skipping + their files, and skips a file without any AI call word before parsing it, + so it is much faster on large repos. +- The code scanner now also finds `client.messages.create(**params)` in a + file that imports `anthropic` (and `anthropic.messages.create(params)` in + JS/TS), calls through `with_raw_response` / `with_streaming_response`, and + reads the model and prompt from a `**params` dict literal defined in the + same function or module. A job found this way is new to `jobs scan` and + `pr-check`, and a job whose model or prompt is now read from such a dict + gets a new approval key, so it needs a new eval before it can move. +- `streamText(x)` and the other AI SDK names are no longer taken for an AI + call when the argument is not an options object (for example a local + `streamText(part.text)` helper). + ## [0.7.0] - 2026-11-10 Stop renting your AI. First release of the loop on PyPI: `pip install "throttle-pro==0.7.0"`. diff --git a/README.md b/README.md index 80e5192..6d1cedf 100644 --- a/README.md +++ b/README.md @@ -32,12 +32,29 @@ MI300X with vLLM, [raw data](validation/hotaisle-mi300x-20261001/)): ## Try it in a minute ```sh -pip install "throttle-pro==0.7.0" +pip install "throttle-pro==0.8.0" +throttle scan . throttle demo ``` Pin the version so what you install never changes under you. +`throttle scan .` reads the code in the current +folder and lists the OpenAI and Anthropic SDK calls it can read: where each +one is, which model it uses, what it costs per 1,000 calls at list price, and +up to two cheaper models to test. It needs no API key, no network and no +traffic, and it writes nothing. It takes seconds on a normal repo. Every +dollar figure is an ESTIMATE: list price times an assumed 1,000 tokens in and +300 out per call (change it with `--tokens-in` and `--tokens-out`; add +`--calls-per-day N` for a monthly figure). The cheaper models are UNTESTED. +Scan says they cost less, not that they do the job as well. Tests, examples +and evals are left out of the totals (`--all` puts them back). A model set at +runtime is priced from a default written in the code, or from `--model NAME`. +Calls through your own wrapper or an agent framework are not found; `throttle +capture` sees them on real traffic. Scan ends with the commands that prove it +on your own traffic ([The loop](#the-loop)). `--json` gives the same data for +tools (schema `throttle.own.scan/1`). + `throttle demo` runs the whole loop offline on made-up traffic: capture, workloads, shadow, eval and report. It needs no API key, no GPU and no network. It writes `throttle-demo/report.html`. Every number in it is @@ -129,7 +146,7 @@ with the date each price was checked and the page it came from. `--prices FILE` adds or replaces prices; those are labelled ASSUMED. The monthly figure scales the capture window to 30 days and is refused for a window shorter than a day. `--embed` also splits each kind into clusters by prompt similarity -(needs `pip install "throttle-pro[embeddings]==0.7.0"`). +(needs `pip install "throttle-pro[embeddings]==0.8.0"`). ### 3. Shadow @@ -413,6 +430,8 @@ same function or file. | LIST PRICE | A published API price, with the date it was checked and its source page. | | ASSUMED | A number you gave us: your GPU hourly rate, or a price from `--prices`. | | DEMO | Made up, from `throttle demo`. Not a measurement of anything. | +| ESTIMATE | From `throttle scan`: list price times a token count per call that is ASSUMED or that you gave. Not measured. | +| UNTESTED | A cheaper model `throttle scan` names. It costs less at list price. Nobody has checked yet that it does the job as well. | `route savings --owned-cost-usd` is labelled OPERATOR-REPORTED. diff --git a/docs/ci-gate.md b/docs/ci-gate.md index 75ea815..d0fb78c 100644 --- a/docs/ci-gate.md +++ b/docs/ci-gate.md @@ -77,7 +77,7 @@ jobs: - uses: actions/setup-python@v5 with: python-version: "3.12" - - run: python -m pip install "throttle-pro==0.7.0" + - run: python -m pip install "throttle-pro==0.8.0" - name: Deploy main's config run: ./deploy/inference/deploy.sh # your deploy; wait until /v1/models answers - name: Measure the baseline 3 times (calibrates the noise bound) @@ -102,7 +102,7 @@ jobs: - uses: actions/setup-python@v5 with: python-version: "3.12" - - run: python -m pip install "throttle-pro==0.7.0" + - run: python -m pip install "throttle-pro==0.8.0" - uses: actions/cache/restore@v4 with: path: ${{ env.THROTTLE_CHECK_HISTORY_DIR }} diff --git a/pyproject.toml b/pyproject.toml index 9af0437..57434b0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "throttle-pro" -version = "0.7.0" +version = "0.8.0" description = "Own your intelligence: prove which share of your OpenAI/Anthropic traffic an open-weight model you run can handle without losing quality, move it with fallback, and measure the savings" keywords = ["llm", "openai", "anthropic", "open-weight", "self-hosted", "inference-cost", "llm-evaluation", "llm-judge", "vllm", "sglang", "ollama", "model-routing"] readme = "README.md" diff --git a/src/throttle/__init__.py b/src/throttle/__init__.py index 86763ac..1b07e8d 100644 --- a/src/throttle/__init__.py +++ b/src/throttle/__init__.py @@ -1,3 +1,3 @@ """Throttle: safety-first measurement for OpenAI-compatible endpoints.""" -__version__ = "0.7.0" +__version__ = "0.8.0" diff --git a/src/throttle/cli.py b/src/throttle/cli.py index 0c82f8c..537a798 100644 --- a/src/throttle/cli.py +++ b/src/throttle/cli.py @@ -81,13 +81,18 @@ # The order `throttle --help` lists commands in: the money loop first, then # the code-review commands, then the serving tools (advanced). HELP_ORDER = ( - "demo", "capture", "workloads", "shadow", "eval", "report", "route", "refresh", + "scan", "demo", "capture", "workloads", "shadow", "eval", "report", "route", "refresh", "jobs", "pr-check", "pr-comment", ) GETTING_STARTED = """\ -Start here: the whole loop on made-up data (offline, no keys, every number -labelled DEMO): +Start here: find the AI calls in your code, what they cost at list price and +which cheaper models to test (offline, no keys, about 30 seconds; every +dollar is an ESTIMATE and every cheaper model is UNTESTED): + throttle scan . + +Then the whole loop on made-up data (offline, no keys, every number labelled +DEMO): throttle demo The money loop, on your own traffic: @@ -121,7 +126,7 @@ throttle refresh In your code: - throttle jobs scan finds every OpenAI/Anthropic call and tracks it as a job. + throttle jobs scan finds the OpenAI/Anthropic SDK calls, one job each. throttle pr-check the pull request comment (or the Throttle for GitHub action): what an added or changed AI job costs, the proof, and what happens next. A changed prompt is @@ -2348,13 +2353,14 @@ def _render_watch_snap(snap) -> None: WELCOME_TAGLINE = "Throttle: stop renting your AI. Cut the bill." WELCOME_STEPS = ( + ("Find the AI calls in your code, offline, no keys:", None, ( + "throttle scan .", + )), ("See the whole loop on made-up data:", "offline, no keys, every number labelled DEMO.", ( "throttle demo", )), - ("Capture your OpenAI / Anthropic traffic:", None, ( + ("Capture your traffic, see what each job costs:", None, ( "throttle capture", - )), - ("See each kind of AI job and what it costs:", None, ( "throttle workloads", )), ("Test a cheaper model on it, then eval it:", None, ( diff --git a/src/throttle/console/docs/getting-started/installation.md b/src/throttle/console/docs/getting-started/installation.md index 1b6dd64..172976f 100644 --- a/src/throttle/console/docs/getting-started/installation.md +++ b/src/throttle/console/docs/getting-started/installation.md @@ -4,15 +4,15 @@ Throttle needs **Python 3.11 or newer**. The command is `throttle`. -Install v0.7.0 from PyPI. Pin the version so what you install never changes under you. +Install v0.8.0 from PyPI. Pin the version so what you install never changes under you. ```bash pip -python3 -m pip install "throttle-pro==0.7.0" +python3 -m pip install "throttle-pro==0.8.0" throttle demo ``` ```bash pipx -pipx install "throttle-pro==0.7.0" +pipx install "throttle-pro==0.8.0" throttle --version ``` @@ -21,7 +21,7 @@ That gives you the whole loop (`capture`, `workloads`, `shadow`, `eval`, `report To install straight from the source you can read, use the release tag: ```bash -pip install "git+https://github.com/KushagraKanaujia/throttle@v0.7.0" +pip install "git+https://github.com/KushagraKanaujia/throttle@v0.8.0" ``` @@ -29,7 +29,7 @@ pip install "git+https://github.com/KushagraKanaujia/throttle@v0.7.0" | Extra | Install | What it adds | | --- | --- | --- | -| `embeddings` | `pip install "throttle-pro[embeddings]==0.7.0"` | `throttle workloads --embed` (similarity clusters) and the opt-in semantic tier of the caching proxy. | +| `embeddings` | `pip install "throttle-pro[embeddings]==0.8.0"` | `throttle workloads --embed` (similarity clusters) and the opt-in semantic tier of the caching proxy. | ## Check your setup diff --git a/src/throttle/console/docs/getting-started/quickstart.md b/src/throttle/console/docs/getting-started/quickstart.md index c82b356..934cbcf 100644 --- a/src/throttle/console/docs/getting-started/quickstart.md +++ b/src/throttle/console/docs/getting-started/quickstart.md @@ -5,12 +5,20 @@ ## 1. Install ```bash -pip install "throttle-pro==0.7.0" +pip install "throttle-pro==0.8.0" ``` -This pins v0.7.0. To install from the source tag instead, see Installation. +This pins v0.8.0. To install from the source tag instead, see Installation. -## 2. Run the demo +## 2. Scan your code + +```bash +throttle scan . +``` + +It finds the OpenAI and Anthropic SDK calls in the folder, prices each one at list price, and names up to two cheaper models to test. Offline, no keys, in seconds. Every dollar is an ESTIMATE and every cheaper model is UNTESTED. See [Scan your code](scan.md). + +## 3. Run the demo ```bash throttle demo @@ -18,7 +26,7 @@ throttle demo It runs capture, workloads, shadow, eval and report on made-up traffic, offline, with no keys and no GPU. Open `throttle-demo/report.html`. Every number is labelled DEMO. See [Demo](../loop/demo.md). -## 3. Run it on your traffic +## 4. Run it on your traffic ```bash # capture: a local pass-through proxy diff --git a/src/throttle/console/docs/getting-started/scan.md b/src/throttle/console/docs/getting-started/scan.md new file mode 100644 index 0000000..9dcee48 --- /dev/null +++ b/src/throttle/console/docs/getting-started/scan.md @@ -0,0 +1,70 @@ +# Scan your code + +*Find the AI calls in your code, what each costs at list price, and which cheaper models to test. Offline, in seconds.* + +```bash +throttle scan . +``` + +Needs Throttle 0.8.0 or later. It needs no API key, no network, no GPU and no traffic, and it writes nothing. It reads the Python, JavaScript and TypeScript files in the folder (skipping `node_modules`, virtualenvs, `dist`, `build` and similar) and finds the direct OpenAI and Anthropic SDK calls it can read, the same way `throttle jobs scan` does. + +For each call it shows: + +- **Where it is**: the job name (a `# throttle-job: name` tag, or the function) and `file:line`. +- **The model**: the literal name in the code. When the code sets it at runtime, the default written next to it (`os.getenv("MODEL", "gpt-4o")`, `process.env.MODEL ?? "gpt-4o"`, `def f(model="gpt-4o")`), marked "(default)", or the model you name with `--model NAME`, marked "(--model)". +- **What it costs**: per 1,000 calls at list price, from the bundled price table. With `--calls-per-day N` it shows dollars per month instead (N calls a day for each job, x 30 days). +- **Up to two cheaper models to test**, each marked UNTESTED. + +The table lists the priced calls, most expensive first, up to 15 (`--all` lists every one). It ends with one summary line, for example "6 AI calls found, 4 priced. Moving the priced ones to the first candidate could cut about 61% of their list-price cost, if every call site runs equally often (ESTIMATE)", and the commands that test the first candidate on your own traffic. + +Calls in tests, specs, examples, evals and benchmarks (`tests/`, `__tests__/`, `*.test.ts`, `test_*.py`, `examples/`, `evals/` and similar) are left out of the totals and counted on one line. `--all` puts them back. When every call is in one of them, they are all counted. + +## What the numbers are + +Every dollar figure is an **ESTIMATE**: list price times a token count per call. + +| Input | Default | Change it with | +| --- | --- | --- | +| Input tokens per call | 1,000 (ASSUMED) | `--tokens-in N` | +| Output tokens per call | 300 (ASSUMED) | `--tokens-out N` | +| Calls per day, per job | none: shows $ per 1,000 calls | `--calls-per-day N` | +| Prices | bundled table, LIST PRICE with its date | `--prices FILE` (ASSUMED) | + +When the code passes a literal output cap (`max_tokens`, `max_completion_tokens`, `max_output_tokens`, `maxTokens` or `maxOutputTokens`) that is smaller than the output tokens above, scan uses the cap and marks the model with `*`. Cached input is not counted. + +## How the cheaper models are picked + +All comparisons are list cost per call at the token counts above. A model must cost at least 10% less to be listed. Dated snapshots and models built only for code are left out. + +1. **First pick, same provider.** One tier down in the same family: `gpt-4o` to `gpt-4o-mini`, `claude-sonnet-4-5` to `claude-haiku-4-5`. When the family has no cheaper tier, the same provider's next price step down. +2. **Second pick, other provider.** The model whose cost per call is closest to the first pick's, so it sits in the same price class. + +> **Note:** A cheaper model is only cheaper. Scan does not test quality, so every pick is UNTESTED. A model with a lower list price can still cost more in real use if it writes longer answers or thinks longer. Prove it with the loop below, which uses measured tokens. + +## What it cannot price + +- **A model set at runtime with no default in the code** (`model=os.environ["CHAT_MODEL"]`, `model: config.model`, `create(**params)`). It is listed with the code that sets it. Rerun with `--model NAME` to price those calls as the model you run. +- **A model that is not in the price table**, such as a fine-tune, a deployment name, another provider's model or an older model the price page no longer lists. It is listed by name. Add a price with `--prices FILE`. +- **Calls through your own wrapper, another SDK or an agent framework.** They are not found. `throttle capture` sees them on real traffic. + +When nothing is found, scan says what it looks for and exits 0. + +## Prove it on your own traffic + +```bash +throttle capture +export OPENAI_BASE_URL=http://127.0.0.1:8790/v1 +export ANTHROPIC_BASE_URL=http://127.0.0.1:8790 + +# after real traffic has gone through +throttle shadow --candidate-url https://api.openai.com/v1 \ + --api-key-env OPENAI_API_KEY --candidate-model gpt-4o-mini --label gpt-4o-mini +throttle eval --candidate gpt-4o-mini --judge-model gpt-4.1 --judge-key-env OPENAI_API_KEY +throttle report --candidate gpt-4o-mini --candidate-model gpt-4o-mini +``` + +Scan prints these commands with its own first pick filled in. When the pick is a Claude model, the shadow step uses `https://api.anthropic.com/v1` (Anthropic's OpenAI-compatible endpoint, offered for testing) with `ANTHROPIC_API_KEY`, and when the code has no OpenAI calls the judge is a Claude model (`--judge-provider anthropic`). Each step has its own page under [The loop](../loop/overview.md). + +## For tools + +`throttle scan . --json` prints the same data as JSON with `"schema": "throttle.own.scan/1"`: the assumptions, a summary, one entry per call, tests and examples included with `"counted": false` (with `status` `priced`, `model_set_at_runtime`, `model_not_in_price_table` or `not_priced_at_this_size`) and each candidate with `"status": "UNTESTED"`. diff --git a/src/throttle/console/docs/index.md b/src/throttle/console/docs/index.md index f462753..f30cfb4 100644 --- a/src/throttle/console/docs/index.md +++ b/src/throttle/console/docs/index.md @@ -14,6 +14,7 @@ capture ──► workloads ──► shadow ──► eval ──► report ─ ## Start here +- **[Scan your code](getting-started/scan.md)**: `throttle scan .` finds the AI calls in your code, what each costs at list price and which cheaper models to test. Offline, no keys. - **[Quickstart](getting-started/quickstart.md)**: Run the whole loop on made-up data in a minute, then on your own traffic. - **[The loop](loop/overview.md)**: One page per stage, with every command and flag. - **[Honest numbers](loop/honest-numbers.md)**: The labels, and the rules that keep a savings number defensible. diff --git a/src/throttle/console/server.py b/src/throttle/console/server.py index 06eebee..30e6371 100644 --- a/src/throttle/console/server.py +++ b/src/throttle/console/server.py @@ -48,6 +48,7 @@ {"tab": "docs", "group": "Getting Started", "pages": [ ("index", "Introduction"), ("getting-started/quickstart", "Quickstart"), + ("getting-started/scan", "Scan your code"), ("getting-started/installation", "Installation"), ("getting-started/first-check", "Your first cost check"), ("getting-started/free-gpu", "Try on a free GPU"), diff --git a/src/throttle/own/cli.py b/src/throttle/own/cli.py index 9609a5d..20e45b6 100644 --- a/src/throttle/own/cli.py +++ b/src/throttle/own/cli.py @@ -18,6 +18,7 @@ def handle(args) -> int # runs it; dispatched on args.command # command name -> module path. Order is the order shown in --help. STAGES: dict[str, str] = { + "scan": "throttle.own.scan_cost", "capture": "throttle.own.capture", "workloads": "throttle.own.workloads", "shadow": "throttle.own.shadow", diff --git a/src/throttle/own/scan.py b/src/throttle/own/scan.py index 1234fc6..1ba0254 100644 --- a/src/throttle/own/scan.py +++ b/src/throttle/own/scan.py @@ -49,6 +49,7 @@ import hashlib import json import logging +import os import re import subprocess from dataclasses import dataclass, field @@ -75,7 +76,14 @@ MIN_MATCH_CHARS = 12 TAG_HEADER = "x-throttle-job" _TAG_COMMENT = re.compile(r"(?:#|//)\s*throttle-job:\s*([A-Za-z0-9._:/@-]+)") +_MAX_TOKEN_KEYS = ("max_tokens", "max_completion_tokens", "max_output_tokens", "maxTokens", "maxOutputTokens") _GREP_PATTERN = r"completions|responses\.|messages\.|ChatCompletion|litellm|ChatOpenAI|ChatAnthropic|generateText|streamText|generateObject|streamObject" +# Every call the scanner can classify contains one of these words, so a file +# without any of them is skipped before it is parsed (most files in a repo). +_PREFILTER = re.compile(_GREP_PATTERN) +# SDK chain segments that wrap the same call: client.messages.with_raw_response.create +_PASSTHROUGH = frozenset({"with_raw_response", "with_streaming_response"}) +_ANTHROPIC_IMPORT_JS = re.compile(r"""['"]@anthropic-ai/sdk|['"]anthropic['"]""") # --------------------------------------------------------------------------- @@ -209,6 +217,17 @@ class CallSite: # sha256 (first 16 hex) of the call's arguments as written (tools, schemas, # few-shot messages, temperature ...); None when not known. call_digest: str | None = None + # The output token cap when the code passes a literal one (max_tokens, + # max_completion_tokens, max_output_tokens, maxTokens, maxOutputTokens). + # Used only by `throttle scan` as an upper bound for output tokens; it is + # not part of the job's identity, its approval key or its saved JSON. + max_tokens: int | None = field(default=None, compare=False) + # When the model is set at run time: a literal default written next to it + # (os.getenv("MODEL", "gpt-4o"), process.env.MODEL ?? "gpt-4o", a + # parameter default), and the ``**kwargs`` the model comes in through. + # Used only by `throttle scan`; not part of identity or saved JSON. + model_default: str | None = field(default=None, compare=False) + model_spread: str | None = field(default=None, compare=False) @property def job(self) -> str: @@ -347,7 +366,11 @@ def _chain(node: ast.expr) -> list[str]: return list(reversed(names)) -def _classify_py(chain: list[str], kwargs: Mapping[str, ast.expr], litellm_names: set[str]) -> tuple[str, str] | None: +def _classify_py(chain: list[str], kwargs: Mapping[str, ast.expr], litellm_names: set[str], *, + spread: bool = False, anthropic_module: bool = False) -> tuple[str, str] | None: + """``spread``: the call passes ``**kwargs``, so the model may come in + through them. ``anthropic_module``: the file imports ``anthropic``.""" + chain = [part for part in chain if part not in _PASSTHROUGH] if not chain: return None last = chain[-1] @@ -361,6 +384,8 @@ def _classify_py(chain: list[str], kwargs: Mapping[str, ast.expr], litellm_names if len(chain) >= 2 and chain[-2] == "messages" and last in _CHAT_ENDINGS: if "model" in kwargs and ("max_tokens" in kwargs or "messages" in kwargs): return PROVIDER_ANTHROPIC, "messages" + if spread and "model" not in kwargs and anthropic_module: + return PROVIDER_ANTHROPIC, "messages" # client.messages.create(**params) return None if len(chain) >= 2 and chain[-2] == "completions" and last in _CHAT_ENDINGS and "prompt" in kwargs: return PROVIDER_OPENAI, "completions" @@ -600,6 +625,67 @@ def _py_shape(node: ast.expr) -> str: return ast.unparse(_Anonymize().visit(copy.deepcopy(node))) +def _py_default(node: ast.expr | None, names: Mapping[str, ast.expr], params: Mapping[str, ast.expr], + depth: int = 0) -> str | None: + """A literal default for a model set at run time: ``os.getenv("M", "x")``, + ``os.environ.get("M", "x")``, ``cfg.get("model", "x")``, ``m or "x"``, a + parameter default ``def f(model="x")``, following plain names.""" + if node is None or depth > 4: + return None + + def literal(value: ast.expr | None) -> str | None: + if isinstance(value, ast.Constant) and isinstance(value.value, str): + text = value.value.strip() + if text and len(text) <= 100 and not any(ch.isspace() for ch in text): + return text + return None + + if isinstance(node, ast.Name): + if node.id in names and names[node.id] is not node: + return _py_default(names[node.id], names, params, depth + 1) + if node.id in params: + return literal(params[node.id]) + return None + if isinstance(node, ast.Call) and isinstance(node.func, (ast.Attribute, ast.Name)): + fname = node.func.attr if isinstance(node.func, ast.Attribute) else node.func.id + if fname in ("getenv", "get") and len(node.args) == 2: + return literal(node.args[1]) + if fname in ("getenv", "get"): + for kw in node.keywords: + if kw.arg == "default": + return literal(kw.value) + return None + if isinstance(node, ast.BoolOp) and isinstance(node.op, ast.Or): + return literal(node.values[-1]) + return None + + +def _py_params(func: ast.AST | None) -> dict[str, ast.expr]: + """Parameter name -> default expression for one function.""" + if not isinstance(func, (ast.FunctionDef, ast.AsyncFunctionDef)): + return {} + args = func.args + positional = args.posonlyargs + args.args + out: dict[str, ast.expr] = {} + for arg, default in zip(positional[len(positional) - len(args.defaults):], args.defaults): + out[arg.arg] = default + for arg, default in zip(args.kwonlyargs, args.kw_defaults): + if default is not None: + out[arg.arg] = default + return out + + +def _py_max_tokens(kwargs: Mapping[str, ast.expr], names: Mapping[str, ast.expr]) -> int | None: + """A literal positive output-token cap, following one name assigned a literal.""" + for key in _MAX_TOKEN_KEYS: + node = kwargs.get(key) + if isinstance(node, ast.Name): + node = names.get(node.id) + if isinstance(node, ast.Constant) and type(node.value) is int and node.value > 0: + return node.value + return None + + def scan_python(path: str, source: str) -> list[CallSite]: try: tree = ast.parse(source) @@ -616,9 +702,14 @@ def _scan_python_tree(path: str, source: str, tree: ast.Module) -> list[CallSite env = _PyEnv(tree) lines = source.splitlines() litellm_names: set[str] = set() + anthropic_module = False for node in ast.walk(tree): if isinstance(node, ast.ImportFrom) and node.module == "litellm": litellm_names.update(alias.asname or alias.name for alias in node.names) + if isinstance(node, ast.ImportFrom) and (node.module or "").split(".")[0] == "anthropic": + anthropic_module = True + if isinstance(node, ast.Import) and any(a.name.split(".")[0] == "anthropic" for a in node.names): + anthropic_module = True sites: list[CallSite] = [] def visit(node: ast.AST, scope: list[str], func: ast.AST | None) -> None: @@ -637,17 +728,30 @@ def visit(node: ast.AST, scope: list[str], func: ast.AST | None) -> None: def _py_call(call: ast.Call, scope: list[str], func: ast.AST | None) -> CallSite | None: kwargs = {k.arg: k.value for k in call.keywords if k.arg} - found = _classify_py(_chain(call.func), kwargs, litellm_names) + spreads = [k.value for k in call.keywords if k.arg is None] + names = env.for_function(func) + # **params where params is a dict literal in this function or module: + # read its keys like keyword arguments (the call's own keywords win). + for value in spreads: + target = names.get(value.id) if isinstance(value, ast.Name) else value + if isinstance(target, ast.Dict): + for key, item in _dict_fields(target).items(): + kwargs.setdefault(key, item) + found = _classify_py(_chain(call.func), kwargs, litellm_names, + spread=bool(spreads), anthropic_module=anthropic_module) if found is None: return None provider, api = found - names = env.for_function(func) model_node = kwargs.get("model") if model_node is None and api == "langchain": model_node = kwargs.get("model_name") model_t = _py_template(model_node, names) if model_node is not None else None model = model_t.parts[0] if model_t is not None and len(model_t.parts) == 1 else None model_expr = None if model or model_node is None else ast.unparse(model_node) + model_default = None if model else _py_default(model_node, names, _py_params(func)) + model_spread = None + if model_node is None and spreads: + model_spread = "**" + ast.unparse(spreads[0]) system = user = None dynamic = False if api in ("chat.completions", "litellm"): @@ -684,6 +788,8 @@ def _py_call(call: ast.Call, scope: list[str], func: ast.AST | None) -> CallSite tag=(_valid_tag(_py_tag(kwargs), path, call.lineno) or _valid_tag(_comment_tag(lines, call.lineno), path, call.lineno)), call_digest=_py_call_digest(call, names), + max_tokens=_py_max_tokens(kwargs, names), + model_default=model_default, model_spread=model_spread, ) visit(tree, [], None) @@ -1099,7 +1205,11 @@ def messages(self, start: int, end: int) -> tuple[Template | None, Template | No return system, user, dynamic -def _js_classify(chain: list[str], fields: Mapping[str, Any]) -> tuple[str, str] | None: +def _js_classify(chain: list[str], fields: Mapping[str, Any], *, opaque: bool = False, + anthropic_module: bool = False) -> tuple[str, str] | None: + """``opaque``: the request is not an object literal with a ``model`` key + (``create(params)``, ``create({...params})``).""" + chain = [part for part in chain if part not in _PASSTHROUGH] last = chain[-1] if len(chain) >= 3 and chain[-3] == "chat" and chain[-2] == "completions" and last in _CHAT_ENDINGS: return PROVIDER_OPENAI, "chat.completions" @@ -1108,6 +1218,8 @@ def _js_classify(chain: list[str], fields: Mapping[str, Any]) -> tuple[str, str] if len(chain) >= 2 and chain[-2] == "messages" and last in _CHAT_ENDINGS: if "model" in fields and ("max_tokens" in fields or "messages" in fields): return PROVIDER_ANTHROPIC, "messages" + if opaque and anthropic_module: + return PROVIDER_ANTHROPIC, "messages" # anthropic.messages.create(params) return None if len(chain) >= 2 and chain[-2] == "completions" and last in _CHAT_ENDINGS and "prompt" in fields: return PROVIDER_OPENAI, "completions" @@ -1119,6 +1231,7 @@ def _js_classify(chain: list[str], fields: Mapping[str, Any]) -> tuple[str, str] def scan_javascript(path: str, source: str) -> list[CallSite]: js = _JsFile(path, source) toks = js.tokens + anthropic_module = bool(_ANTHROPIC_IMPORT_JS.search(source)) sites: list[CallSite] = [] for i, tok in enumerate(toks): if tok.kind != "punct" or tok.text != "(" or i == 0 or toks[i - 1].kind != "ident": @@ -1138,11 +1251,28 @@ def scan_javascript(path: str, source: str) -> list[CallSite]: fields: dict[str, tuple[int, int]] = {} if i + 1 < close and toks[i + 1].text == "{": fields = _object_fields(toks, i + 1, _match_close(toks, i + 1)) - found = _js_classify(chain, fields) + first_arg = "" + if i + 1 < close: + k = i + 1 + while k < close and toks[k].text != ",": + if toks[k].kind == "punct" and toks[k].text in _OPEN: + k = _match_close(toks, k) + k += 1 + first_arg = source[toks[i + 1].start: toks[k - 1].start + len(toks[k - 1].text)] + opaque = "model" not in fields and bool(first_arg) + # an AI SDK call takes an options object; streamText(part.text) is + # someone's own function of the same name + if (len(chain) == 1 and chain[0] in _AI_SDK and first_arg + and not (first_arg.startswith("{") or re.fullmatch(r"[A-Za-z_$][\w$]*", first_arg))): + continue + found = _js_classify(chain, fields, opaque=opaque, anthropic_module=anthropic_module) if found is None: continue provider, api = found - model = model_expr = None + model = model_expr = model_default = None + # create(params) or create({ ...params }): the model comes in through them + model_spread = (" ".join(first_arg.split()) + if opaque and (not first_arg.lstrip().startswith("{") or "..." in first_arg) else None) if "model" in fields: ms, me = js.resolve_span(*fields["model"]) model = js.literal(ms, me) @@ -1153,6 +1283,11 @@ def scan_javascript(path: str, source: str) -> list[CallSite]: model = toks[ms + 2].value if model is None: model_expr = source[toks[ms].start: toks[me].start + len(toks[me].text)] + # process.env.MODEL ?? "gpt-4o" / || "gpt-4o" + if me > ms and toks[me].kind == "str" and toks[me - 1].text in ("??", "||"): + value = str(toks[me].value or "").strip() + if value and not any(ch.isspace() for ch in value): + model_default = value system = user = None dynamic = False if api in ("chat.completions",) or (api in ("messages", "ai-sdk") and "messages" in fields): @@ -1204,6 +1339,16 @@ def scan_javascript(path: str, source: str) -> list[CallSite]: elif nxt.kind == "ident" and nxt.text in ("throttleJob", "throttle_job"): if k + 3 < close and toks[k + 3].kind == "str": tag = toks[k + 3].value + max_tokens = None + for key in _MAX_TOKEN_KEYS: + if key in fields and max_tokens is None: + s, e = js.resolve_span(*fields[key]) + if s == e and toks[s].kind == "num": + try: + value = int(toks[s].text.replace("_", "")) + except ValueError: + value = 0 + max_tokens = value if value > 0 else None line = js.line_of(i) if api == "ai-sdk" and provider == PROVIDER_UNKNOWN: provider = _provider_from_model(model) @@ -1214,6 +1359,7 @@ def scan_javascript(path: str, source: str) -> list[CallSite]: wants_json="response_format" in fields or (api == "ai-sdk" and chain[-1].endswith("Object")), tag=_valid_tag(tag, path, line) or _valid_tag(_comment_tag(js.lines, line), path, line), call_digest=_digest(" ".join(t.text for t in toks[i + 1:close])), + max_tokens=max_tokens, model_default=model_default, model_spread=model_spread, )) return sites @@ -1231,6 +1377,8 @@ def wanted(path: str) -> bool: def scan_source(path: str, source: str) -> list[CallSite]: + if not _PREFILTER.search(source): + return [] # no AI call this scanner can read; skip the parse if path.endswith(PY_SUFFIXES): return scan_python(path, source) if path.endswith(JS_SUFFIXES): @@ -1247,10 +1395,20 @@ def scan_files(files: Mapping[str, str]) -> list[CallSite]: return _number(sites) +def _walk(root: Path) -> Iterable[Path]: + """Every file under ``root``, never entering a :data:`SKIP_DIRS` folder or + a symlinked folder. Pruning here (not after listing) keeps a scan fast on + a repo with a large ``node_modules`` or virtualenv.""" + for dirpath, dirnames, filenames in os.walk(root): + dirnames[:] = sorted(d for d in dirnames if d not in SKIP_DIRS) + for name in sorted(filenames): + yield Path(dirpath) / name + + def scan_dir(root: Path) -> list[CallSite]: root = Path(root) files: dict[str, str] = {} - for path in sorted(root.rglob("*")): + for path in _walk(root): rel = path.relative_to(root).as_posix() if not path.is_file() or not wanted(rel): continue diff --git a/src/throttle/own/scan_cost.py b/src/throttle/own/scan_cost.py new file mode 100644 index 0000000..fd9a7be --- /dev/null +++ b/src/throttle/own/scan_cost.py @@ -0,0 +1,880 @@ +"""``throttle scan``: the 30-second first run. + +It reads the code in a folder, finds the direct OpenAI and Anthropic SDK +calls it can read (with :mod:`throttle.own.scan`, the code behind ``throttle +jobs scan``), prices each one at list price, and names up to two cheaper +models to test. It needs no API key, no network, no GPU and no captured +traffic, and it writes nothing. Calls through a wrapper of the user's own, +another SDK or an agent framework are not found; ``throttle capture`` sees +them on real traffic. + +What it can and cannot say: + +* Every dollar figure is an ESTIMATE: list price (from the bundled + ``prices.json``, via :mod:`throttle.own.pricing`) times an assumed token + shape per call (default 1,000 tokens in and 300 out, labelled ASSUMED, or + the user's ``--tokens-in`` / ``--tokens-out``). When the code passes a + literal output cap (``max_tokens`` and friends) smaller than the assumed + output, the cap is used, because a call cannot write more than that. +* A cheaper model is only cheaper. Nothing here tests quality, so every + candidate is UNTESTED. A model with a lower list price can still cost more + in real use if it writes longer answers or thinks longer; the loop + (``capture`` -> ``shadow`` -> ``eval`` -> ``report``) measures that. +* ``--calls-per-day`` turns per-call cost into $/month (x 30 days), with the + same daily call count assumed for every job. Without it the output is per + 1,000 calls, and the headline cut weights every call site the same. +* A model set at run time is priced only from a literal default written next + to it (``os.getenv("MODEL", "gpt-4o")``, labelled "default") or from the + user's ``--model`` (labelled "--model"). Otherwise it is listed, not priced. +* Tests, examples, evals and benchmarks are left out of the totals (``--all`` + includes them), unless every call is in one of them. + +How candidates are picked is in :func:`pick_candidates`. +""" + +from __future__ import annotations + +import argparse +import json +import math +import re +import sys +from dataclasses import dataclass, field +from pathlib import Path, PurePosixPath +from typing import Any, Iterable, Sequence + +from . import pricing as pricing_mod +from . import records +from . import scan as scan_mod + +SCHEMA = "throttle.own.scan/1" +# The first release with `throttle scan`. Install steps that pin an older +# release must not run it (tests/test_own_scan_cost.py checks the docs). +FIRST_RELEASE = "0.8.0" +DEFAULT_TOKENS_IN = 1_000 +DEFAULT_TOKENS_OUT = 300 +MAX_TOKENS_FLAG = 10_000_000 +DAYS_PER_MONTH = 30 +MAX_CANDIDATES = 2 +# A candidate must be at least this much cheaper per call, so the list never +# suggests a switch that saves a rounding error. +MIN_CUT = 0.10 +# Models built for one kind of work (code). Never suggested for a general job. +SPECIALIZED = ("codex",) + +ESTIMATE = "ESTIMATE" +ASSUMED = "ASSUMED" +UNTESTED = "UNTESTED" + +STATUS_PRICED = "priced" +STATUS_RUNTIME = "model_set_at_runtime" +STATUS_UNKNOWN = "model_not_in_price_table" +STATUS_UNPRICED = "not_priced_at_this_size" # over a long-context threshold the price page does not price + +WHY_FAMILY = "same provider, smaller tier of the same family" +WHY_PROVIDER = "same provider, next price step down" +WHY_OTHER = "other provider, similar price to the first pick" +WHY_OTHER_ALONE = "other provider, closest cheaper price" + +EXIT_OK = 0 +EXIT_USAGE = 2 + +_DATED = re.compile(r"[-@](\d{4}-\d{2}-\d{2}|\d{8})$") + + +# --------------------------------------------------------------------------- +# Pricing and candidates +# --------------------------------------------------------------------------- + + +def per_call_usd(price: pricing_mod.Price, tokens_in: int, tokens_out: int) -> float | None: + """List-price cost of one call of this shape (no cached input), or + ``None`` when the price page does not price a prompt this long.""" + usage = records.Usage(input_tokens=tokens_in, output_tokens=tokens_out, cached_input_tokens=0) + return pricing_mod.cost_usd(price, usage) + + +def family(price: pricing_mod.Price) -> tuple[str, str]: + """Which models are tiers of one family, read from the name only. + + * Anthropic names end in the version: ``claude-haiku-4-5`` and + ``claude-opus-4-5`` are both family ``4-5``. + * Other names start with it: everything up to and including the first + part that has a digit, so ``gpt-5-mini`` and ``gpt-5-pro`` are family + ``gpt-5`` and ``o3-mini`` is ``o3``. + """ + parts = price.model.lower().split("-") + if price.provider == records.PROVIDER_ANTHROPIC: + tail: list[str] = [] + while parts and parts[-1].isdigit(): + tail.insert(0, parts.pop()) + return price.provider, "-".join(tail) or price.model.lower() + for i, part in enumerate(parts): + if any(ch.isdigit() for ch in part): + return price.provider, "-".join(parts[: i + 1]) + return price.provider, price.model.lower() + + +def _suggestable(price: pricing_mod.Price) -> bool: + name = price.model.lower() + return not _DATED.search(name) and not any(word in name for word in SPECIALIZED) + + +@dataclass(frozen=True) +class Candidate: + price: pricing_mod.Price + per_call_usd: float + why: str + + +def pick_candidates(current: pricing_mod.Price, table: Iterable[pricing_mod.Price], tokens_in: int, + tokens_out: int, *, limit: int = MAX_CANDIDATES) -> list[Candidate]: + """Up to ``limit`` cheaper models to test instead of ``current``. + + All comparisons are list cost per call at this token shape. + + 1. The pool: every other model in the price table that costs at least + ``MIN_CUT`` (10%) less per call, leaving out dated snapshots (pinned + old versions) and models built for code only (``SPECIALIZED``). + 2. First pick, same provider: one tier down in the same family (the most + expensive cheaper model of that family, e.g. ``gpt-4o`` -> + ``gpt-4o-mini``, ``claude-sonnet-4-5`` -> ``claude-haiku-4-5``). With + no cheaper tier in the family, the same provider's next price step + down (the most expensive model in the pool). + 3. Second pick, other provider: the model whose cost per call is closest + (as a ratio) to the first pick's, so it sits in the same price class. + With no first pick, the closest to the current model's. + + Ties go to the entry listed first in ``prices.json``. The picks say + nothing about quality: they are what to test, and are UNTESTED. + """ + if limit <= 0: + return [] + now = per_call_usd(current, tokens_in, tokens_out) + if now is None or now <= 0: + return [] + pool: list[Candidate] = [] + for price in table: + if price is current or (price.provider, price.model.lower()) == (current.provider, current.model.lower()): + continue + if not _suggestable(price): + continue + cost = per_call_usd(price, tokens_in, tokens_out) + if cost is None or cost > now * (1 - MIN_CUT) * (1 + 1e-9): # 1e-9: float noise at the edge + continue + pool.append(Candidate(price, cost, "")) + + def most_expensive(items: list[Candidate]) -> Candidate | None: + best: Candidate | None = None + for item in items: # table order, so the first listed wins a tie + if best is None or item.per_call_usd > best.per_call_usd: + best = item + return best + + picks: list[Candidate] = [] + same = [c for c in pool if c.price.provider == current.provider] + in_family = [c for c in same if family(c.price) == family(current)] + first = most_expensive(in_family) + if first is not None: + picks.append(Candidate(first.price, first.per_call_usd, WHY_FAMILY)) + else: + first = most_expensive(same) + if first is not None: + picks.append(Candidate(first.price, first.per_call_usd, WHY_PROVIDER)) + + target = first.per_call_usd if first is not None else now + best_other: Candidate | None = None + best_gap = math.inf + for item in pool: + if item.price.provider == current.provider or item.per_call_usd <= 0: + continue + gap = abs(math.log(item.per_call_usd / target)) + if gap < best_gap - 1e-12: + best_other, best_gap = item, gap + if best_other is not None: + why = WHY_OTHER if first is not None else WHY_OTHER_ALONE + picks.append(Candidate(best_other.price, best_other.per_call_usd, why)) + return picks[:limit] + + +# --------------------------------------------------------------------------- +# The scan +# --------------------------------------------------------------------------- + + + + +# How the model of a job was found. +SOURCE_LITERAL = "literal" # written in the call +SOURCE_DEFAULT = "default_in_code" # set at run time, with a literal default next to it +SOURCE_FLAG = "flag" # set at run time; priced as the user's --model +SOURCE_RUNTIME = "runtime" # set at run time; not priced + +# Where a call lives. Tests, examples and evals are not counted by default. +WHERE_APP = "app" +WHERE_TEST = "test" +WHERE_EXAMPLE = "example" + +MAX_ROWS = 15 # priced jobs in the table before "--all" +MAX_NOT_PRICED = 8 # lines in the "Not priced" list before "--all" +EXPR_WIDTH = 40 + +_TEST_DIRS = frozenset({"test", "tests", "__tests__", "spec", "specs", "__mocks__", "mocks", "fixtures", + "e2e", "testing"}) +_EXAMPLE_DIRS = frozenset({"example", "examples", "sample", "samples", "eval", "evals", "evaluation", + "evaluations", "benchmark", "benchmarks", "bench", "cookbook", "cookbooks"}) +_TEST_FILE = re.compile(r"(^test_.*\.py$|_test\.py$|^conftest\.py$|\.(test|spec)\.[cm]?[jt]sx?$" + r"|_spec\.[cm]?[jt]sx?$)", re.IGNORECASE) +_EXAMPLE_FILE = re.compile(r"(^|[_.-])(benchmark|bench|eval|evals|example|examples)([_.-][^/]*)?\.[a-z]+$", + re.IGNORECASE) +# Model names from other providers (the SDK call can point at any +# OpenAI-compatible server), and older OpenAI / Anthropic models that the +# bundled price table does not carry. +_OTHER_PROVIDER = re.compile(r"^(gemini|gemma|llama|meta-llama|mistral|mixtral|codestral|command|grok|deepseek|" + r"qwen|phi|ollama|groq|together|accounts/)", re.IGNORECASE) +_OLDER = re.compile(r"^(gpt-3\.5|gpt-4$|gpt-4-|text-|davinci|babbage|claude-instant|claude-[123]([-.]|$)|" + r"claude-(opus|sonnet)-4(-[01])?(-\d{8})?$)", re.IGNORECASE) +_HTTP_VERBS = frozenset({"GET", "POST", "PUT", "PATCH", "DELETE"}) +_GENERIC_STEMS = frozenset({"index", "route", "main", "__init__", "app", "server", "handler"}) + + +def where(path: str) -> str: + """``test`` for test and spec files, ``example`` for examples, evals and + benchmarks, else ``app``.""" + parts = PurePosixPath(path).parts + dirs = [part.lower() for part in parts[:-1]] + if any(d in _TEST_DIRS for d in dirs) or (parts and _TEST_FILE.search(parts[-1])): + return WHERE_TEST + if any(d in _EXAMPLE_DIRS for d in dirs) or (parts and _EXAMPLE_FILE.search(parts[-1])): + return WHERE_EXAMPLE + return WHERE_APP + + +@dataclass(frozen=True) +class Shape: + tokens_in: int = DEFAULT_TOKENS_IN + tokens_out: int = DEFAULT_TOKENS_OUT + calls_per_day: int | None = None + tokens_in_given: bool = False + tokens_out_given: bool = False + # --model: price every job whose model is set at run time as this model + runtime_model: str | None = None + + @property + def calls_per_month(self) -> int | None: + return None if self.calls_per_day is None else self.calls_per_day * DAYS_PER_MONTH + + +@dataclass +class JobCost: + site: scan_mod.CallSite + status: str + tokens_out: int + capped: bool = False + price: pricing_mod.Price | None = None + per_call_usd: float | None = None + candidates: list[Candidate] = field(default_factory=list) + model: str | None = None # the model priced (or looked up) + model_source: str = SOURCE_RUNTIME + where: str = WHERE_APP + counted: bool = True # False: in tests or examples, left out of the totals + + @property + def priced(self) -> bool: + return self.status == STATUS_PRICED + + +def _lookup(table: pricing_mod.PriceTable, model: str, provider: str) -> pricing_mod.Price | None: + """The price for ``model``: the call's provider first, then the other one + (an OpenAI client can call a Claude model through a compatible endpoint).""" + model = model.strip() + for prefix, named in (("openai/", records.PROVIDER_OPENAI), ("anthropic/", records.PROVIDER_ANTHROPIC)): + if model.lower().startswith(prefix): # litellm style "anthropic/claude-..." + model, provider = model[len(prefix):], named + order = [provider] if provider in records.PROVIDERS else [] + order += [name for name in records.PROVIDERS if name not in order] + for name in order: + found = table.lookup(name, model) + if found is not None: + return found + return None + + +def cost_job(site: scan_mod.CallSite, table: pricing_mod.PriceTable, shape: Shape) -> JobCost: + tokens_out = shape.tokens_out + capped = False + if site.max_tokens is not None and site.max_tokens < tokens_out: + tokens_out, capped = site.max_tokens, True + place = where(site.path) + if site.model: + model, source = site.model, SOURCE_LITERAL + elif shape.runtime_model: + model, source = shape.runtime_model, SOURCE_FLAG + elif site.model_default: + model, source = site.model_default, SOURCE_DEFAULT + else: + return JobCost(site, STATUS_RUNTIME, tokens_out, capped, where=place) + price = _lookup(table, model, site.provider) + if price is None: + return JobCost(site, STATUS_UNKNOWN, tokens_out, capped, model=model, model_source=source, where=place) + cost = per_call_usd(price, shape.tokens_in, tokens_out) + if cost is None: + return JobCost(site, STATUS_UNPRICED, tokens_out, capped, price, model=model, model_source=source, + where=place) + candidates = pick_candidates(price, table, shape.tokens_in, tokens_out) + return JobCost(site, STATUS_PRICED, tokens_out, capped, price, cost, candidates, model=model, + model_source=source, where=place) + + +def mark_counted(jobs: Sequence[JobCost], include_all: bool) -> None: + """Leave tests, examples and evals out of the totals, unless ``include_all`` + or every call is in one of them (a repo of examples is still worth a look).""" + in_app = any(j.where == WHERE_APP for j in jobs) + for j in jobs: + j.counted = include_all or not in_app or j.where == WHERE_APP + + +def _per_1k(value: float | None) -> float | None: + return None if value is None else value * 1000 + + +def _monthly(value: float | None, shape: Shape) -> float | None: + if value is None or shape.calls_per_month is None: + return None + return value * shape.calls_per_month + + +def _cut(now: float | None, then: float | None) -> float | None: + if now is None or then is None or now <= 0: + return None + return 1 - then / now + + +def summarize(jobs: Sequence[JobCost], shape: Shape) -> dict[str, Any]: + """Totals over the counted jobs. Each call site counts once: the code says + nothing about how often a call runs.""" + counted = [j for j in jobs if j.counted] + priced = [j for j in counted if j.priced] + now = sum(j.per_call_usd or 0.0 for j in priced) + first = sum((j.candidates[0].per_call_usd if j.candidates else j.per_call_usd or 0.0) for j in priced) + cut = _cut(now, first) if priced else None + return { + "calls_found": len(jobs), + "counted": len(counted), + "not_counted_tests_examples": len(jobs) - len(counted), + "priced": len(priced), + "with_candidate": sum(1 for j in priced if j.candidates), + "not_priced": len(counted) - len(priced), + "per_1k_calls_usd": _per_1k(now) if priced else None, + "first_candidates_per_1k_calls_usd": _per_1k(first) if priced else None, + "monthly_usd": _monthly(now, shape) if priced else None, + "first_candidates_monthly_usd": _monthly(first, shape) if priced else None, + "cut_pct": None if cut is None else round(cut * 100, 1), + "volume": "the same for every call site", + "label": ESTIMATE, + } + + +def run_scan(root: Path, shape: Shape, table: pricing_mod.PriceTable, *, include_all: bool = False) -> list[JobCost]: + jobs = [cost_job(site, table, shape) for site in scan_mod.scan_dir(root)] + mark_counted(jobs, include_all) + return jobs + + +JUDGE_OPENAI = ("gpt-4.1", "gpt-4o") +JUDGE_ANTHROPIC = ("claude-sonnet-4-6", "claude-opus-4-8") +CANDIDATE_URL = { + records.PROVIDER_OPENAI: ("https://api.openai.com/v1", "OPENAI_API_KEY"), + # Anthropic's OpenAI-compatible endpoint, which `throttle shadow` speaks + records.PROVIDER_ANTHROPIC: ("https://api.anthropic.com/v1", "ANTHROPIC_API_KEY"), +} + + +def next_steps(jobs: Sequence[JobCost]) -> list[str]: + """Copyable commands that prove the biggest suggestion on real traffic. + + The candidate is the first pick of the costliest counted job, served at + its provider's OpenAI-compatible URL. The judge is an OpenAI model when the + code calls OpenAI (so the user has that key), else a Claude model.""" + counted = [j for j in jobs if j.counted] + with_pick = [j for j in counted if j.priced and j.candidates] + if not with_pick: + return [] + top = max(with_pick, key=lambda j: j.per_call_usd or 0.0) + pick = top.candidates[0].price + url, key = CANDIDATE_URL.get(pick.provider, ("URL", "KEY_VAR")) + uses_openai = any(j.site.provider == records.PROVIDER_OPENAI + or (j.price is not None and j.price.provider == records.PROVIDER_OPENAI) for j in counted) + if uses_openai: + judge = next(m for m in JUDGE_OPENAI if m != pick.model) + judge_flags = f"--judge-model {judge} --judge-key-env OPENAI_API_KEY" + else: + judge = next(m for m in JUDGE_ANTHROPIC if m != pick.model) + judge_flags = f"--judge-provider anthropic --judge-model {judge} --judge-key-env ANTHROPIC_API_KEY" + return [ + "throttle capture", + "export OPENAI_BASE_URL=http://127.0.0.1:8790/v1", + "export ANTHROPIC_BASE_URL=http://127.0.0.1:8790", + f"throttle shadow --candidate-url {url} --api-key-env {key} --candidate-model {pick.model} " + f"--label {pick.model}", + f"throttle eval --candidate {pick.model} {judge_flags}", + f"throttle report --candidate {pick.model} --candidate-model {pick.model}", + ] + + +# --------------------------------------------------------------------------- +# JSON +# --------------------------------------------------------------------------- + + +def _round(value: float | None) -> float | None: + return None if value is None else round(value, 8) + + +def _price_json(price: pricing_mod.Price | None) -> dict[str, Any] | None: + if price is None: + return None + data = price.to_json() + keep = ("provider", "model", "input_per_m", "output_per_m", "as_of", "source", "label", "note") + return {key: data[key] for key in keep if key in data} + + +def _labels(jobs: Sequence[JobCost], table: pricing_mod.PriceTable) -> list[str]: + counted = [j for j in jobs if j.counted] + return sorted({j.price.label for j in counted if j.price} + | {c.price.label for j in counted for c in j.candidates}) or table.labels + + +def to_json(path: str, jobs: Sequence[JobCost], shape: Shape, table: pricing_mod.PriceTable) -> dict[str, Any]: + out_jobs = [] + for j in jobs: + site = j.site + candidates = [] + for c in j.candidates: + candidates.append({ + "model": c.price.model, "provider": c.price.provider, "status": UNTESTED, "why": c.why, + "price": _price_json(c.price), + "per_call_usd": _round(c.per_call_usd), + "per_1k_calls_usd": _round(_per_1k(c.per_call_usd)), + "monthly_usd": _round(_monthly(c.per_call_usd, shape)), + "cut_pct": round(_cut(j.per_call_usd, c.per_call_usd) * 100, 1), # type: ignore[operator] + }) + out_jobs.append({ + "job": site.job, "path": site.path, "line": site.line, "function": site.function, + "language": site.language, "provider": site.provider, "api": site.api, + "model": site.model, "model_expr": site.model_expr, + "model_default": site.model_default, "model_spread": site.model_spread, + "priced_model": j.model, "model_source": j.model_source, + "where": j.where, "counted": j.counted, + "status": j.status, + "tokens_in": shape.tokens_in, "tokens_out": j.tokens_out, + "max_tokens": site.max_tokens, "tokens_out_capped_by_max_tokens": j.capped, + "price": _price_json(j.price), + "per_call_usd": _round(j.per_call_usd), + "per_1k_calls_usd": _round(_per_1k(j.per_call_usd)), + "monthly_usd": _round(_monthly(j.per_call_usd, shape)), + "candidates": candidates, + }) + summary = summarize(jobs, shape) + for key in ("per_1k_calls_usd", "first_candidates_per_1k_calls_usd", "monthly_usd", + "first_candidates_monthly_usd"): + summary[key] = _round(summary[key]) + return { + "schema": SCHEMA, + "path": path, + "label": ESTIMATE, + "assumptions": { + "tokens_in": shape.tokens_in, + "tokens_in_source": "flag" if shape.tokens_in_given else ASSUMED, + "tokens_out": shape.tokens_out, + "tokens_out_source": "flag" if shape.tokens_out_given else ASSUMED, + "cached_input_tokens": 0, + "calls_per_day": shape.calls_per_day, + "calls_per_day_source": None if shape.calls_per_day is None else "flag", + "days_per_month": DAYS_PER_MONTH, + "runtime_model": shape.runtime_model, + "runtime_model_source": None if shape.runtime_model is None else "flag", + "price_labels": _labels(jobs, table), + "quality": UNTESTED, + }, + "summary": summary, + "jobs": out_jobs, + "next_steps": next_steps(jobs), + } + + +# --------------------------------------------------------------------------- +# Terminal +# --------------------------------------------------------------------------- + + +def money(value: float | None) -> str: + if value is None: + return "-" + if value >= 100: + return f"${value:,.0f}" + if value >= 0.01: + return f"${value:,.2f}" + if value == 0: + return "$0" + return f"${value:.4f}".rstrip("0") if value >= 0.0001 else "under $0.0001" + + +def _fit(text: str, width: int) -> str: + if len(text) <= width: + return text + return text[: max(0, width - 2)] + ".." + + +def _fit_left(text: str, width: int) -> str: + """Cut from the left, so a location keeps its file name and line.""" + if len(text) <= width: + return text + return ".." + text[len(text) - max(0, width - 2):] + + +_TS_CAST = re.compile(r"\s+as\s+[\w.$<>\[\]|'\"\s]+$|!$") + + +def _short(expr: str | None, width: int = EXPR_WIDTH) -> str: + """Code on one line, whitespace runs collapsed, a TypeScript cast + (``x as T``) dropped, at most ``width`` long.""" + text = " ".join((expr or "").split()) + return _fit(_TS_CAST.sub("", text) or text, width) + + +def _display(path: str) -> str: + """The folder name for '.', so the title says what was scanned.""" + if path in (".", "./"): + return Path.cwd().name or path + return path + + +def _continued(command: str, *flags: str) -> list[str]: + """Break a long shell command before each of ``flags`` with a backslash, + so each printed line stays short and the whole still pastes as one.""" + lines = [command] + for flag in flags: + head, sep, tail = lines[-1].partition(" " + flag) + if sep: + lines[-1:] = [head + " \\", " " + flag + tail] + return lines + + +def _int(value: int) -> str: + return f"{value:,}" + + +def _plural(n: int, word: str) -> str: + return f"{n} {word}{'' if n == 1 else 's'}" + + +def job_name(site: scan_mod.CallSite) -> str: + """A short name for the table: the tag, else the function, else the file.""" + if site.tag: + return site.tag + path = PurePosixPath(site.path) + place = path.stem if path.stem not in _GENERIC_STEMS else f"{path.parent.name or '.'}/{path.stem}" + if site.function == "": + return place + if site.function in _HTTP_VERBS: + return f"{path.parent.name or path.stem} {site.function}" + return site.function + + +def _model_label(j: JobCost, width: int) -> str: + suffix = "*" if j.capped else "" + suffix += {SOURCE_DEFAULT: " (default)", SOURCE_FLAG: " (--model)"}.get(j.model_source, "") + return _fit(j.model or "", max(4, width - len(suffix))) + suffix + + +def not_priced_reason(j: JobCost, table: pricing_mod.PriceTable) -> str: + site = j.site + if j.status == STATUS_RUNTIME: + expr = _short(site.model_expr) or _short(site.model_spread) + return f"model set at runtime ({expr})" if expr else "no model named in the call" + model = j.model or "" + via = {SOURCE_DEFAULT: " (default in code)", SOURCE_FLAG: " (your --model)"}.get(j.model_source, "") + if j.status == STATUS_UNKNOWN: + if _OTHER_PROVIDER.search(model): + return f"{model}{via}: not an OpenAI or Anthropic model; add its price with --prices FILE" + if _OLDER.search(model): + dates = sorted({p.as_of for p in table if p.origin == pricing_mod.ORIGIN_BUNDLED}) + checked = f", prices checked {dates[-1]}" if dates else "" + return f"{model}{via}: older model, not in the price table{checked}; add its price with --prices FILE" + return f"{model}{via} is not in the price table; add it with --prices FILE" + return f"{model}{via}: the price page does not price a prompt this long" + + +def _not_priced_lines(jobs: Sequence[JobCost], table: pricing_mod.PriceTable) -> list[str]: + """One line per (file, reason): many calls of one kind in one file fold.""" + groups: dict[tuple[str, str], list[int]] = {} + for j in jobs: + groups.setdefault((j.site.path, not_priced_reason(j, table)), []).append(j.site.line) + out = [] + for (path, reason), lines in groups.items(): + if len(lines) == 1: + out.append(f"- {path}:{lines[0]} {reason}") + else: + out.append(f"- {path}: {len(lines)} calls, lines {', '.join(map(str, lines))} {reason}") + return out + + +LOOKS_FOR = ( + "Python: client.chat.completions.create, client.responses.create, client.messages.create " + "(Anthropic), openai.ChatCompletion.create, litellm.completion, ChatOpenAI / ChatAnthropic.", + "JavaScript / TypeScript: the same SDK calls, plus the Vercel AI SDK (generateText, streamText, " + "generateObject, streamObject).", + f"It skips these folders: {', '.join(sorted(scan_mod.SKIP_DIRS))}.", +) +NOT_FOUND = ("Not found by scan: calls through your own wrapper, another SDK or an agent framework. " + "'throttle capture' records those from real traffic.") + + +def render(path: str, jobs: Sequence[JobCost], shape: Shape, table: pricing_mod.PriceTable, *, + show_all: bool = False) -> str: + from .. import style as style_mod + + st = style_mod.get() + inner = st.columns if st.narrow else st.width - 4 + arrow = "↳ " if st.unicode else "> " + monthly = shape.calls_per_day is not None + cost_head = "$ / MONTH" if monthly else "$ / 1K CALLS" + show_cut = inner >= 50 + cost_w = 13 + cut_w = 7 if show_cut else 0 + rest = max(20, inner - cost_w - (cut_w + 2 if show_cut else 1)) + model_w = min(26, max(18, rest // 2)) + job_w = max(8, rest - model_w) + + def row(job: str, model: str, cost: str, cut: str) -> str: + line = f"{_fit(job, job_w):<{job_w}} {_fit(model, model_w):<{model_w}}{cost:>{cost_w}}" + return line + (f" {cut:>{cut_w - 1}}" if show_cut else "") + + def amount(per_call: float | None) -> str: + return money(_monthly(per_call, shape) if monthly else _per_1k(per_call)) + + if not jobs: + rows: list[Any] = [ + (f"No OpenAI or Anthropic calls found in {_display(path)}.", ()), + "", + ("What scan looks for:", ()), + *[(f"- {text}", (), " ") for text in LOOKS_FOR], + "", + ("Calls through your own wrapper, another SDK or an agent framework are not found. " + "'throttle capture' records them from real traffic.", ()), + ] + return "\n".join([ + st.panel(rows, title=f"Throttle scan: {_display(path)}"), + "", + "No code to scan yet? See the whole loop on made-up data: throttle demo", + ]) + + counted = [j for j in jobs if j.counted] + hidden = len(jobs) - len(counted) + priced = sorted((j for j in counted if j.priced), key=lambda j: -(j.per_call_usd or 0.0)) + shown = priced if show_all else priced[:MAX_ROWS] + + rows = [] + if shown: + rows += [st.bold(row("JOB", "MODEL", cost_head, "CHANGE" if show_cut else "")), ""] + for j in shown: + site = j.site + rows.append(row(job_name(site), _model_label(j, model_w), amount(j.per_call_usd), "")) + where_text = " " + _fit_left(site.location, job_w - 2) + if not show_cut: # narrow: the location gets a line of its own + rows.append(st.dim(" " + _fit_left(site.location, inner - 2))) + where_text = "" + if not j.candidates: + rows.append(st.dim(row(where_text, " no cheaper model listed", "", "").rstrip())) + for i, c in enumerate(j.candidates): + cut = _cut(j.per_call_usd, c.per_call_usd) + cut_text = f"-{cut * 100:.0f}%" if cut is not None else "" + line = row(where_text if i == 0 else "", arrow + c.price.model, amount(c.per_call_usd), cut_text) + head, tail = line[:job_w], line[job_w:] + if show_cut: + rows.append(st.dim(head) + tail[: -cut_w + 1] + st.good(tail[-cut_w + 1:])) + else: + rows.append(st.dim(head) + tail) + rows.append("") + if len(shown) < len(priced): + rows.append(st.dim(f"{_plural(len(priced) - len(shown), 'more priced call')}, each cheaper per call " + "than these. --all lists them.")) + if any(j.candidates for j in shown): + rows.append(st.dim(f"{arrow}cheaper at list price, {UNTESTED} on your traffic.")) + if any(j.capped for j in shown): + rows.append(st.dim("* output capped by a literal max_tokens in the code.")) + if any(j.model_source == SOURCE_DEFAULT for j in shown): + rows.append(st.dim("(default): model set at runtime, priced at the default written in the code.")) + if any(j.model_source == SOURCE_FLAG for j in shown): + rows.append(st.dim(f"(--model): model set at runtime, priced as {shape.runtime_model} (your flag).")) + if shown: + rows.append("") + + summary = summarize(jobs, shape) + head = f"{_plural(summary['counted'], 'AI call')} found, {summary['priced']} priced" + if summary["cut_pct"] is not None and summary["with_candidate"]: + weight = "" if monthly else ", if every call site runs equally often" + text = (f"{head}. Moving the priced ones to the first candidate could cut about " + f"{summary['cut_pct']:.0f}% of their list-price cost{weight} ({ESTIMATE}).") + elif summary["priced"]: + text = f"{head}. No cheaper model in the price table for them." + else: + text = f"{head}." + rows.append((text, ("1",))) + if monthly and summary["monthly_usd"] is not None: + rows.append((f"At {_int(shape.calls_per_day or 0)} calls a day per job: about " + f"{money(summary['monthly_usd'])}/mo now, about " + f"{money(summary['first_candidates_monthly_usd'])}/mo on the first candidates " + f"({ESTIMATE}).", ())) + if hidden: + rows.append((f"Not counted: {hidden} more in tests, examples or evals. --all includes them.", ())) + elif not show_all and any(j.where != WHERE_APP for j in jobs): + rows.append((f"All {summary['counted']} are in tests, examples or evals, so they are counted.", ())) + runtime = [j for j in counted if j.status == STATUS_RUNTIME] + if runtime and not shape.runtime_model: + providers = [j.site.provider for j in runtime] + example = ("claude-sonnet-4-6" if providers.count(records.PROVIDER_ANTHROPIC) + > providers.count(records.PROVIDER_OPENAI) else "gpt-4o") + rows.append((f"{len(runtime)} of {summary['counted']} set the model at runtime. To price them, rerun " + f"with the model you use, for example --model {example}.", ())) + rows.append("") + if shape.tokens_in_given == shape.tokens_out_given: + source = "your flags" if shape.tokens_in_given else ASSUMED + shape_text = f"{_int(shape.tokens_in)} tokens in, {_int(shape.tokens_out)} out, {source}." + else: + tin = "your flag" if shape.tokens_in_given else ASSUMED + tout = "your flag" if shape.tokens_out_given else ASSUMED + shape_text = (f"{_int(shape.tokens_in)} tokens in ({tin}), " + f"{_int(shape.tokens_out)} out ({tout}).") + rows.append((f"Per call: {shape_text} Change with --tokens-in / --tokens-out.", ())) + if not monthly: + rows.append(("Add --calls-per-day N for a monthly figure.", ())) + else: + rows.append((f"Per month: --calls-per-day {_int(shape.calls_per_day or 0)} x {DAYS_PER_MONTH} days, " + "the same for every job.", ())) + rows.append((f"Prices: {', '.join(_labels(jobs, table))}, standard tier, no cached input.", ())) + rows.append(("A lower list price can still cost more if the model writes longer answers or thinks " + "longer. Quality is not tested here.", ())) + rows.append((NOT_FOUND, ())) + + unpriced = [j for j in counted if not j.priced] + if unpriced: + lines = _not_priced_lines(unpriced, table) + rows.append("") + rows.append((f"Not priced ({len(unpriced)}):", ("1",))) + for line in lines if show_all else lines[:MAX_NOT_PRICED]: + rows.append((line, (), " ")) + if not show_all and len(lines) > MAX_NOT_PRICED: + rows.append((f"... {len(lines) - MAX_NOT_PRICED} more. --all lists them.", ())) + + out = [st.panel(rows, title=f"Throttle scan: {_display(path)}")] + steps = next_steps(jobs) + if steps: + out.append("") + out.append(st.bold("Prove it on your own traffic") + " (see README \"The loop\"):") + out.append(st.dim(" 1. Capture: a local proxy in front of the API, then point your app at it")) + out.extend(f" {st.accent(cmd) if i == 0 else cmd}" for i, cmd in enumerate(steps[:3])) + out.append(st.dim(" 2. Shadow: replay captured requests on the candidate (never served)")) + out.extend(f" {piece}" for piece in _continued(steps[3], "--api-key-env", "--candidate-model")) + if "api.anthropic.com" in steps[3]: + out.append(st.dim(" That is Anthropic's OpenAI-compatible endpoint, offered for testing.")) + out.append(st.dim(" 3. Eval and report: judge the answers, then price it on measured tokens")) + out.extend(f" {piece}" for piece in _continued(steps[4], "--judge-provider", "--judge-model")) + out.extend(f" {piece}" for piece in _continued(steps[5], "--candidate-model")) + else: + out.append("") + out.append("Next: capture real traffic with 'throttle capture' (see README \"The loop\"), " + "or try the whole loop on made-up data with 'throttle demo'.") + return "\n".join(out) + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + + +def _token_count(value: str) -> int: + try: + parsed = int(value.replace(",", "").replace("_", "")) + except ValueError as exc: + raise argparse.ArgumentTypeError("must be a whole number of tokens") from exc + if parsed <= 0: + raise argparse.ArgumentTypeError("must be greater than zero") + if parsed > MAX_TOKENS_FLAG: + raise argparse.ArgumentTypeError(f"must be at most {MAX_TOKENS_FLAG:,}") + return parsed + + +def _calls_per_day(value: str) -> int: + try: + parsed = int(value.replace(",", "").replace("_", "")) + except ValueError as exc: + raise argparse.ArgumentTypeError("must be a whole number of calls") from exc + if parsed <= 0: + raise argparse.ArgumentTypeError("must be greater than zero") + return parsed + + +def _model_name(value: str) -> str: + name = value.strip() + if not name or len(name) > 100 or any(ch.isspace() for ch in name): + raise argparse.ArgumentTypeError("must be one model name, e.g. gpt-4o") + return name + + +def add_parser(subparsers: argparse._SubParsersAction) -> None: + parser = subparsers.add_parser( + "scan", + help="start here: find the AI calls in your code, their list-price cost and cheaper models to test", + description=( + "Find the direct OpenAI and Anthropic SDK calls in the code, price each one at list price, and " + "name up to two cheaper models to test. No API key, no network, no traffic, and it writes " + "nothing. Every dollar figure is an ESTIMATE from list prices and an assumed token shape per " + "call. Candidates are UNTESTED: scan says they are cheaper, not that they are as good. Tests, " + "examples and evals are left out of the totals unless you pass --all." + ), + ) + parser.add_argument("path", nargs="?", default=".", help="the code to scan (default: current folder)") + parser.add_argument("--tokens-in", type=_token_count, default=None, metavar="N", + help=f"input tokens per call (default {DEFAULT_TOKENS_IN:,}, ASSUMED)") + parser.add_argument("--tokens-out", type=_token_count, default=None, metavar="N", + help=f"output tokens per call (default {DEFAULT_TOKENS_OUT}, ASSUMED; a literal " + "max_tokens in the code caps it)") + parser.add_argument("--calls-per-day", type=_calls_per_day, default=None, metavar="N", + help=f"calls a day for each job, to show $/month (x {DAYS_PER_MONTH} days)") + parser.add_argument("--model", type=_model_name, default=None, metavar="NAME", + help="price every call whose model is set at runtime as NAME (overrides a default " + "written in the code)") + parser.add_argument("--all", action="store_true", + help="include tests, examples and evals, and list every row") + parser.add_argument("--prices", type=Path, default=None, metavar="FILE", + help="JSON price file that adds or replaces prices (labelled ASSUMED)") + parser.add_argument("--json", action="store_true", help=f"print JSON ({SCHEMA}) instead of the table") + + +def handle(args: argparse.Namespace) -> int: + root = Path(args.path).expanduser() + if not root.is_dir(): + print(f"throttle scan: {args.path} is not a folder", file=sys.stderr) + return EXIT_USAGE + try: + table = pricing_mod.load_prices(args.prices) + except ValueError as exc: + print(f"throttle scan: {exc}", file=sys.stderr) + return EXIT_USAGE + shape = Shape( + tokens_in=args.tokens_in or DEFAULT_TOKENS_IN, + tokens_out=args.tokens_out or DEFAULT_TOKENS_OUT, + calls_per_day=args.calls_per_day, + tokens_in_given=args.tokens_in is not None, + tokens_out_given=args.tokens_out is not None, + runtime_model=getattr(args, "model", None), + ) + show_all = bool(getattr(args, "all", False)) + jobs = run_scan(root, shape, table, include_all=show_all) + if args.json: + print(json.dumps(to_json(args.path, jobs, shape, table), indent=2)) + else: + print(render(args.path, jobs, shape, table, show_all=show_all)) + return EXIT_OK diff --git a/tests/test_cli.py b/tests/test_cli.py index f148f08..aac7221 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -394,6 +394,7 @@ def test_parser_subcommands_match_expected(self) -> None: self.assertEqual( set(subparser_action.choices), { + "scan", "plan", "smoke", "benchmark", @@ -2378,7 +2379,7 @@ def test_loop_is_listed_before_the_serving_tools(self) -> None: listing = text.split("positional arguments:", 1)[1].split("options:", 1)[0] names = [line.split()[0] for line in listing.splitlines() if line.startswith(" ") and line.split() and not line.startswith(" ")] - loop = ["demo", "capture", "workloads", "shadow", "eval", "report", "route", "refresh", "jobs", "pr-check"] + loop = ["scan", "demo", "capture", "workloads", "shadow", "eval", "report", "route", "refresh", "jobs", "pr-check"] self.assertEqual(names[:len(loop)], loop) for serving in ("check", "plan", "smoke", "benchmark", "golden", "autotune", "proxy", "savings"): self.assertGreater(names.index(serving), names.index("pr-check"), serving) diff --git a/tests/test_first_run.py b/tests/test_first_run.py index 7a490e1..fea64ac 100644 --- a/tests/test_first_run.py +++ b/tests/test_first_run.py @@ -291,8 +291,9 @@ def test_no_args_prints_the_loop_and_exits_zero(capsys, monkeypatch): assert len(lines) <= 20 assert "Throttle " + __version__ in out assert "Throttle: stop renting your AI. Cut the bill." in out - # The loop, in order: demo -> capture -> workloads -> shadow -> eval -> report. - order = ["1. See the whole loop", "throttle demo", "2. Capture", "throttle capture", + # The loop, in order: scan -> demo -> capture -> workloads -> shadow -> eval -> report. + order = ["1. Find the AI calls", "throttle scan .", "2. See the whole loop", "throttle demo", + "3. Capture", "throttle capture", "throttle workloads", "throttle shadow", "throttle eval", "throttle report", "throttle route approve"] positions = [out.index(text) for text in order] diff --git a/tests/test_own_scan_cost.py b/tests/test_own_scan_cost.py new file mode 100644 index 0000000..cd7d3fe --- /dev/null +++ b/tests/test_own_scan_cost.py @@ -0,0 +1,835 @@ +"""`throttle scan`: the offline first run (find, price, suggest, never claim).""" + +from __future__ import annotations + +import json +import math +import re +import socket +import tomllib +from pathlib import Path + +import pytest + +from throttle.cli import GETTING_STARTED, HELP_ORDER, build_parser, main +from throttle.own import pricing, scan, scan_cost +from throttle.own.scan_cost import ( + MIN_CUT, SCHEMA, STATUS_PRICED, STATUS_RUNTIME, STATUS_UNKNOWN, Shape, WHY_FAMILY, WHY_OTHER, + WHY_OTHER_ALONE, WHY_PROVIDER, family, per_call_usd, pick_candidates, +) + +REPO = Path(__file__).resolve().parents[1] + +# A small app with Python and JS/TS calls: two priced OpenAI calls (one with a +# literal max_tokens under the assumed output), two priced Anthropic calls, a +# model set at runtime and a model missing from the price table. +FIXTURE = { + "app/summarize.py": '''from openai import OpenAI + +client = OpenAI() + + +def summarize(text: str) -> str: + resp = client.chat.completions.create( + model="gpt-4o", + max_tokens=200, + messages=[ + {"role": "system", "content": "Summarize the support ticket in two sentences."}, + {"role": "user", "content": text}, + ], + ) + return resp.choices[0].message.content +''', + "app/classify.py": '''import anthropic + +client = anthropic.Anthropic() + + +def classify(ticket: str) -> str: + # throttle-job: ticket-triage + msg = client.messages.create( + model="claude-sonnet-4-5", + max_tokens=1024, + system="Label the ticket as billing, bug or other. Reply with the label only.", + messages=[{"role": "user", "content": ticket}], + ) + return msg.content[0].text +''', + "app/router.py": '''import os + +from openai import OpenAI + +client = OpenAI() + + +def answer(question: str) -> str: + resp = client.chat.completions.create( + model=os.environ["CHAT_MODEL"], + messages=[{"role": "user", "content": question}], + ) + return resp.choices[0].message.content +''', + "web/src/extract.ts": '''import OpenAI from "openai"; + +const client = new OpenAI(); + +export async function extractInvoice(text: string) { + const res = await client.responses.create({ + model: "gpt-4.1", + instructions: "Extract the invoice number, total and due date as JSON.", + input: text, + }); + return res.output_text; +} + +export async function draftReply(text: string) { + const res = await client.chat.completions.create({ + model: "acme-finetune-v2", + messages: [{ role: "user", content: text }], + }); + return res.choices[0].message.content; +} +''', + "web/src/chat.js": '''import Anthropic from "@anthropic-ai/sdk"; + +const anthropic = new Anthropic(); + +export async function chat(history) { + const msg = await anthropic.messages.create({ + model: "claude-opus-4-5", + max_tokens: 2048, + messages: history, + }); + return msg.content[0].text; +} +''', +} + + +def write_fixture(root: Path) -> Path: + for rel, source in FIXTURE.items(): + path = root / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(source, encoding="utf-8") + return root + + +def price(model: str, provider: str = "openai", inp: float = 1.0, out: float = 4.0, **extra) -> pricing.Price: + return pricing.Price(provider=provider, model=model, input_per_m=inp, output_per_m=out, + cached_input_per_m=None, as_of="2026-10-27", source="test", **extra) + + +@pytest.fixture +def plain(monkeypatch): + monkeypatch.setenv("NO_COLOR", "1") + monkeypatch.setenv("COLUMNS", "80") + monkeypatch.delenv("FORCE_COLOR", raising=False) + + +def flatten(out: str) -> str: + """The output as one line of words, without the panel frame.""" + lines = [] + for line in out.splitlines(): + if line[:1] in "│|" and line[-1:] in "│|": + line = line[1:-1] + lines.append(line) + return " ".join(" ".join(lines).split()) + + +def run(capsys, *argv: str) -> tuple[int, str, str]: + code = main(["scan", *argv]) + out = capsys.readouterr() + return code, out.out, out.err + + +# --------------------------------------------------------------------------- +# Candidate selection +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("model,provider,expected", [ + ("gpt-4o-mini", "openai", "gpt-4o"), + ("gpt-5-pro", "openai", "gpt-5"), + ("gpt-5.4-nano", "openai", "gpt-5.4"), + ("o3-mini", "openai", "o3"), + ("gpt-6-astra", "openai", "gpt-6"), + ("claude-haiku-4-5", "anthropic", "4-5"), + ("claude-opus-4-5", "anthropic", "4-5"), + ("claude-sonnet-5", "anthropic", "5"), + ("plain", "openai", "plain"), +]) +def test_family_is_read_from_the_name(model, provider, expected) -> None: + assert family(price(model, provider)) == (provider, expected) + + +def test_first_pick_is_one_tier_down_in_the_family_then_other_provider_at_a_similar_price() -> None: + table = [ + price("big-9", inp=10, out=40), + price("big-9-mid", inp=3, out=12), + price("big-9-small", inp=0.5, out=2), + price("other-1", inp=2, out=8), # same provider, not in the family + price("claude-top-9", "anthropic", inp=20, out=80), # dearer: never suggested + price("claude-mid-9", "anthropic", inp=3, out=12), + price("claude-low-9", "anthropic", inp=0.4, out=1.6), + ] + picks = pick_candidates(table[0], table, 1000, 300) + assert [(c.price.model, c.why) for c in picks] == [("big-9-mid", WHY_FAMILY), ("claude-mid-9", WHY_OTHER)] + assert all(c.per_call_usd < per_call_usd(table[0], 1000, 300) for c in picks) + + +def test_no_cheaper_tier_in_family_falls_back_to_next_price_step_of_the_provider() -> None: + table = [ + price("solo-3", inp=4, out=16), + price("other-2", inp=3, out=12), + price("other-1", inp=1, out=4), + price("claude-mid-9", "anthropic", inp=3, out=12), + ] + picks = pick_candidates(table[0], table, 1000, 300) + assert [(c.price.model, c.why) for c in picks] == [("other-2", WHY_PROVIDER), ("claude-mid-9", WHY_OTHER)] + + +def test_cheapest_model_gets_only_a_cheaper_other_provider_model_or_nothing() -> None: + table = [price("tiny-1", inp=1, out=4), price("claude-x-1", "anthropic", inp=0.5, out=2)] + picks = pick_candidates(table[0], table, 1000, 300) + assert [(c.price.model, c.why) for c in picks] == [("claude-x-1", WHY_OTHER_ALONE)] + assert pick_candidates(table[1], table, 1000, 300) == [] + + +def test_pool_skips_snapshots_specialized_models_and_tiny_savings() -> None: + now = price("big-9", inp=10, out=40) + table = [ + now, + price("big-9-2024-05-13", inp=1, out=4), # dated snapshot + price("big-9-codex", inp=1, out=4), # built for code only + price("big-9-nearly", inp=9.5, out=38), # only 5% cheaper + ] + assert pick_candidates(now, table, 1000, 300) == [] + cheaper_enough = price("big-9-ok", inp=10 * (1 - MIN_CUT), out=40 * (1 - MIN_CUT)) + assert [c.price.model for c in pick_candidates(now, [*table, cheaper_enough], 1000, 300)] == ["big-9-ok"] + + +def test_ties_go_to_the_entry_listed_first_and_limit_is_respected() -> None: + now = price("claude-top-9", "anthropic", inp=10, out=40) + table = [now, price("claude-a-9", "anthropic", inp=2, out=8), price("claude-b-9", "anthropic", inp=2, out=8), + price("x-1", inp=2, out=8), price("y-1", inp=2, out=8)] + picks = pick_candidates(now, table, 1000, 300) + assert [c.price.model for c in picks] == ["claude-a-9", "x-1"] + assert [c.price.model for c in pick_candidates(now, table, 1000, 300, limit=1)] == ["claude-a-9"] + assert pick_candidates(now, table, 1000, 300, limit=0) == [] + + +def test_picks_depend_on_the_token_shape() -> None: + # cheap input, dear output vs the reverse: which is cheaper flips with the shape + now = price("base-1", inp=5, out=5) + out_heavy = price("base-1-a", inp=1, out=6) + in_heavy = price("base-1-b", inp=6, out=1) + table = [now, out_heavy, in_heavy] + assert [c.price.model for c in pick_candidates(now, table, 10_000, 10)] == ["base-1-a"] + assert [c.price.model for c in pick_candidates(now, table, 10, 10_000)] == ["base-1-b"] + + +def test_bundled_table_picks_are_cheaper_and_same_provider_first() -> None: + table = pricing.load_prices() + for entry in table: + picks = pick_candidates(entry, table, 1000, 300) + now = per_call_usd(entry, 1000, 300) + assert len(picks) <= 2 + for pick in picks: + assert pick.per_call_usd <= now * (1 - MIN_CUT) + 1e-15, (entry.model, pick.price.model) + assert pick.price.model != entry.model + if picks and picks[0].why in (WHY_FAMILY, WHY_PROVIDER): + assert picks[0].price.provider == entry.provider + gpt4o = table.lookup("openai", "gpt-4o") + assert pick_candidates(gpt4o, table, 1000, 300)[0].price.model == "gpt-4o-mini" + sonnet = table.lookup("anthropic", "claude-sonnet-4-5") + assert pick_candidates(sonnet, table, 1000, 300)[0].price.model == "claude-haiku-4-5" + + +# --------------------------------------------------------------------------- +# Pricing math +# --------------------------------------------------------------------------- + + +def test_per_call_cost_is_list_price_times_tokens() -> None: + p = price("m-1", inp=2.5, out=10) + assert per_call_usd(p, 1000, 300) == pytest.approx((2.5 * 1000 + 10 * 300) / 1_000_000) + + +def test_long_prompt_over_an_unpriced_threshold_is_not_priced() -> None: + p = price("m-1", inp=2.5, out=10, long_context=pricing.LongContext(above_input_tokens=5000)) + assert per_call_usd(p, 6000, 300) is None + site = scan.CallSite(path="a.py", line=1, function="f", language="python", provider="openai", + api="chat.completions", model="m-1") + job = scan_cost.cost_job(site, pricing.PriceTable([p]), Shape(tokens_in=6000)) + assert job.status == scan_cost.STATUS_UNPRICED and job.per_call_usd is None + + +def test_max_tokens_caps_output_and_monthly_uses_calls_per_day() -> None: + table = pricing.PriceTable([price("m-1", inp=2, out=10), price("m-1-mini", inp=0.2, out=1)]) + site = scan.CallSite(path="a.py", line=3, function="f", language="python", provider="openai", + api="chat.completions", model="m-1", max_tokens=100) + shape = Shape(tokens_in=1000, tokens_out=300, calls_per_day=1000) + job = scan_cost.cost_job(site, table, shape) + assert job.capped and job.tokens_out == 100 + assert job.per_call_usd == pytest.approx((2 * 1000 + 10 * 100) / 1e6) + summary = scan_cost.summarize([job], shape) + assert summary["monthly_usd"] == pytest.approx(job.per_call_usd * 1000 * 30) + assert summary["per_1k_calls_usd"] == pytest.approx(job.per_call_usd * 1000) + assert summary["cut_pct"] == pytest.approx(90.0) + # a cap above the assumed output changes nothing + site.max_tokens = 4096 + assert not scan_cost.cost_job(site, table, shape).capped + + +def test_summary_cut_counts_jobs_without_a_candidate_at_their_own_cost() -> None: + table = pricing.PriceTable([price("m-1", inp=2, out=10), price("m-1-mini", inp=0.2, out=1), + price("solo", inp=0.1, out=0.5)]) + sites = [scan.CallSite(path="a.py", line=i, function=f"f{i}", language="python", provider="openai", + api="chat.completions", model=m) for i, m in enumerate(["m-1", "solo"], 1)] + shape = Shape() + jobs = [scan_cost.cost_job(s, table, shape) for s in sites] + assert jobs[1].candidates == [] + now = jobs[0].per_call_usd + jobs[1].per_call_usd + after = jobs[0].candidates[0].per_call_usd + jobs[1].per_call_usd + assert scan_cost.summarize(jobs, shape)["cut_pct"] == round((1 - after / now) * 100, 1) + + +def test_litellm_style_and_unknown_provider_names_are_looked_up() -> None: + table = pricing.load_prices() + for provider, model in (("unknown", "claude-haiku-4-5"), ("openai", "anthropic/claude-haiku-4-5"), + ("unknown", "gpt-4o-mini")): + site = scan.CallSite(path="a.py", line=1, function="f", language="python", provider=provider, + api="litellm", model=model) + assert scan_cost.cost_job(site, table, Shape()).status == STATUS_PRICED, model + + +# --------------------------------------------------------------------------- +# Scanner additions: literal max_tokens, pruned walk +# --------------------------------------------------------------------------- + + +def test_scanner_reads_literal_max_tokens_in_python_and_js_but_not_into_identity() -> None: + py = scan.scan_python("a.py", 'LIMIT = 64\nclient.chat.completions.create(model="gpt-4o", messages=[], ' + 'max_tokens=LIMIT)\nclient.chat.completions.create(model="gpt-4o", messages=[], ' + 'max_tokens=n)\n') + assert [s.max_tokens for s in py] == [64, None] + js = scan.scan_javascript("a.ts", 'await client.messages.create({ model: "claude-haiku-4-5", ' + 'max_tokens: 1_024, messages: [] });\n' + 'await generateText({ model: openai("gpt-4o"), maxOutputTokens: 50, prompt: "hi" });\n') + assert [s.max_tokens for s in js] == [1024, 50] + assert "max_tokens" not in py[0].to_json() + assert scan.CallSite.from_json(py[0].to_json()) == py[0] + + +def test_scan_dir_never_enters_skipped_folders(tmp_path) -> None: + write_fixture(tmp_path) + for skipped in ("node_modules/pkg", ".venv/lib", "dist"): + target = tmp_path / skipped / "x.py" + target.parent.mkdir(parents=True) + target.write_text('client.chat.completions.create(model="gpt-4o", messages=[])\n') + found = {s.path for s in scan.scan_dir(tmp_path)} + assert found == {"app/summarize.py", "app/classify.py", "app/router.py", "web/src/extract.ts", + "web/src/chat.js"} + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("argv,needle", [ + (["--tokens-in", "0"], "greater than zero"), + (["--tokens-in", "-5"], "greater than zero"), + (["--tokens-out", "lots"], "whole number"), + (["--tokens-out", "20000001"], "at most"), + (["--calls-per-day", "0"], "greater than zero"), + (["--calls-per-day", "1.5"], "whole number"), +]) +def test_bad_flags_are_usage_errors(capsys, argv, needle) -> None: + with pytest.raises(SystemExit) as exc: + build_parser().parse_args(["scan", *argv]) + assert exc.value.code == 2 + assert needle in capsys.readouterr().err + + +def test_good_flags_parse_with_commas() -> None: + args = build_parser().parse_args(["scan", "src", "--tokens-in", "1,500", "--tokens-out", "200", + "--calls-per-day", "10_000", "--json"]) + assert (args.path, args.tokens_in, args.tokens_out, args.calls_per_day, args.json) == ("src", 1500, 200, + 10000, True) + + +def test_missing_folder_is_a_usage_error(tmp_path, capsys, plain) -> None: + code, out, err = run(capsys, str(tmp_path / "nope")) + assert code == 2 and "is not a folder" in err and out == "" + + +def test_empty_repo_says_what_scan_looks_for(tmp_path, capsys, plain, monkeypatch) -> None: + (tmp_path / "main.py").write_text("print('no AI here')\n") + monkeypatch.chdir(tmp_path) + code, out, _ = run(capsys, ".") + assert code == 0 + flat = flatten(out) + assert f"No OpenAI or Anthropic calls found in {tmp_path.name}." in flat + assert "chat.completions.create" in flat and "generateText" in flat and "node_modules" in flat + assert "throttle demo" in flat + code, out, _ = run(capsys, str(tmp_path), "--json") + data = json.loads(out) + assert data["summary"]["calls_found"] == 0 and data["jobs"] == [] and data["next_steps"] == [] + + +def test_runtime_and_unknown_models_are_listed_not_priced(tmp_path, capsys, plain) -> None: + write_fixture(tmp_path) + code, out, _ = run(capsys, str(tmp_path), "--json") + assert code == 0 + jobs = {j["function"]: j for j in json.loads(out)["jobs"]} + runtime = jobs["answer"] + assert runtime["status"] == STATUS_RUNTIME and runtime["model"] is None + assert runtime["model_source"] == "runtime" and runtime["model_expr"] == "os.environ['CHAT_MODEL']" + assert runtime["priced_model"] is None and runtime["where"] == "app" and runtime["counted"] is True + assert runtime["per_call_usd"] is None and runtime["candidates"] == [] + unknown = jobs["draftReply"] + assert unknown["status"] == STATUS_UNKNOWN and unknown["model"] == "acme-finetune-v2" + assert unknown["price"] is None and unknown["candidates"] == [] + code, out, _ = run(capsys, str(tmp_path)) + flat = flatten(out) + assert "Not priced (2):" in flat + assert "app/router.py:9 model set at runtime (os.environ['CHAT_MODEL'])" in flat + assert "acme-finetune-v2 is not in the price table; add it with --prices FILE" in flat + + +def test_prices_file_prices_an_unknown_model_as_assumed(tmp_path, capsys, plain) -> None: + app = write_fixture(tmp_path / "app") + prices = tmp_path / "prices.json" + prices.write_text(json.dumps([{"provider": "openai", "model": "acme-finetune-v2", + "input_per_m": 3, "output_per_m": 12}])) + code, out, _ = run(capsys, str(app), "--json", "--prices", str(prices)) + assert code == 0 + job = next(j for j in json.loads(out)["jobs"] if j["model"] == "acme-finetune-v2") + assert job["status"] == STATUS_PRICED and job["price"]["label"].startswith("ASSUMED") + bad = tmp_path / "bad.json" + bad.write_text("{") + code, _, err = run(capsys, str(app), "--prices", str(bad)) + assert code == 2 and "not valid JSON" in err + + +def test_json_schema_is_stable(tmp_path, capsys, plain) -> None: + write_fixture(tmp_path) + code, out, _ = run(capsys, str(tmp_path), "--json", "--calls-per-day", "2000") + assert code == 0 + data = json.loads(out) + assert data["schema"] == SCHEMA == "throttle.own.scan/1" + assert set(data) == {"schema", "path", "label", "assumptions", "summary", "jobs", "next_steps"} + assert data["label"] == "ESTIMATE" and data["summary"]["label"] == "ESTIMATE" + a = data["assumptions"] + assert (a["tokens_in"], a["tokens_in_source"], a["tokens_out"], a["tokens_out_source"]) == ( + 1000, "ASSUMED", 300, "ASSUMED") + assert (a["calls_per_day"], a["calls_per_day_source"], a["days_per_month"], a["quality"]) == ( + 2000, "flag", 30, "UNTESTED") + assert a["price_labels"] and all(label.startswith("LIST PRICE (") for label in a["price_labels"]) + assert (a["runtime_model"], a["runtime_model_source"]) == (None, None) + s = data["summary"] + assert (s["calls_found"], s["counted"], s["not_counted_tests_examples"], s["priced"], s["not_priced"], + s["with_candidate"]) == (6, 6, 0, 4, 2, 4) + assert s["volume"] == "the same for every call site" + job_keys = {"job", "path", "line", "function", "language", "provider", "api", "model", "model_expr", + "model_default", "model_spread", "priced_model", "where", "counted", + "model_source", "status", "tokens_in", "tokens_out", "max_tokens", + "tokens_out_capped_by_max_tokens", "price", "per_call_usd", "per_1k_calls_usd", "monthly_usd", + "candidates"} + for job in data["jobs"]: + assert set(job) == job_keys + for c in job["candidates"]: + assert set(c) == {"model", "provider", "status", "why", "price", "per_call_usd", "per_1k_calls_usd", + "monthly_usd", "cut_pct"} + assert c["status"] == "UNTESTED" and c["per_call_usd"] < job["per_call_usd"] + assert c["monthly_usd"] == pytest.approx(c["per_call_usd"] * 2000 * 30, rel=1e-6) + summarize = next(j for j in data["jobs"] if j["function"] == "summarize") + assert summarize["job"] == "app/summarize.py::summarize" and summarize["max_tokens"] == 200 + assert summarize["tokens_out"] == 200 and summarize["tokens_out_capped_by_max_tokens"] is True + assert summarize["per_1k_calls_usd"] == pytest.approx((2.5 * 1000 + 10 * 200) / 1000) + assert next(j for j in data["jobs"] if j["function"] == "classify")["job"] == "ticket-triage" + total = sum(j["monthly_usd"] for j in data["jobs"] if j["monthly_usd"] is not None) + assert s["monthly_usd"] == pytest.approx(total) + assert math.isclose(s["cut_pct"], round((1 - s["first_candidates_per_1k_calls_usd"] + / s["per_1k_calls_usd"]) * 100, 1), abs_tol=0.05) + + +def test_end_to_end_on_a_python_and_js_repo_offline_and_writes_nothing(tmp_path, capsys, plain, + monkeypatch) -> None: + repo = write_fixture(tmp_path / "repo") + data_dir = tmp_path / "throttle-data" + monkeypatch.setenv("THROTTLE_OWN_DIR", str(data_dir)) + monkeypatch.setenv("HOME", str(tmp_path / "home")) + monkeypatch.chdir(repo) + + def no_network(*_a, **_k): + raise AssertionError("throttle scan must not use the network") + + monkeypatch.setattr(socket.socket, "connect", no_network) + monkeypatch.setattr(socket, "create_connection", no_network) + before = sorted(p.relative_to(tmp_path).as_posix() for p in tmp_path.rglob("*")) + code, out, err = run(capsys, ".") + after = sorted(p.relative_to(tmp_path).as_posix() for p in tmp_path.rglob("*")) + assert code == 0 and err == "" + assert before == after and not data_dir.exists() + + lines = out.splitlines() + assert lines[0].startswith("┌─ Throttle scan: repo ") or lines[0].startswith("+- Throttle scan: repo ") + assert all(len(line) <= 80 for line in lines), [line for line in lines if len(line) > 80] + flat = flatten(out) + for needed in ("JOB", "MODEL", "$ / 1K CALLS", "CHANGE", + "ticket-triage", "app/classify.py:8", "claude-sonnet-4-5", + "summarize", "app/summarize.py:7", "gpt-4o*", "gpt-4o-mini", + "extractInvoice", "web/src/extract.ts:6", "gpt-4.1", + "chat", "web/src/chat.js:6", "claude-opus-4-5", + "cheaper at list price, UNTESTED on your traffic.", + "* output capped by a literal max_tokens in the code.", + "6 AI calls found, 4 priced. Moving the priced ones to the first candidate could cut about", + "of their list-price cost, if every call site runs equally often (ESTIMATE).", + "1 of 6 set the model at runtime. To price them, rerun with the model you use, for example " + "--model gpt-4o.", + "Not found by scan: calls through your own wrapper, another SDK or an agent framework.", + "Per call: 1,000 tokens in, 300 out, ASSUMED.", + "Add --calls-per-day N for a monthly figure.", + "Quality is not tested here.", + 'Prove it on your own traffic (see README "The loop")', + "throttle capture", "export OPENAI_BASE_URL=http://127.0.0.1:8790/v1", + # the costliest job is chat (claude-opus-4-5); its first pick is a Claude model + "throttle shadow --candidate-url https://api.anthropic.com/v1 \\ " + "--api-key-env ANTHROPIC_API_KEY \\ --candidate-model claude-sonnet-4-5 --label claude-sonnet-4-5", + "That is Anthropic's OpenAI-compatible endpoint, offered for testing.", + # the code calls OpenAI too, so the judge is the OpenAI one + "throttle eval --candidate claude-sonnet-4-5 \\ --judge-model gpt-4.1 --judge-key-env OPENAI_API_KEY", + "throttle report --candidate claude-sonnet-4-5 \\ --candidate-model claude-sonnet-4-5"): + assert needed in flat, needed + # never a quality claim, never an em dash + lowered = flat.lower() + assert "as good" not in lowered and "just as well" not in lowered and "\u2014" not in out + # priced rows come most expensive first + order = [flat.index(name) for name in ("claude-opus-4-5 $", "claude-sonnet-4-5 $", "gpt-4o* $", "gpt-4.1 $")] + assert order == sorted(order) + + +def test_monthly_output_and_given_tokens_are_labelled(tmp_path, capsys, plain) -> None: + write_fixture(tmp_path) + code, out, _ = run(capsys, str(tmp_path), "--calls-per-day", "2000", "--tokens-in", "500") + assert code == 0 + flat = flatten(out) + assert "$ / MONTH" in flat + assert "At 2,000 calls a day per job: about $" in flat and "/mo on the first candidates (ESTIMATE)" in flat + assert "Per call: 500 tokens in (your flag), 300 out (ASSUMED)." in flat + assert "--calls-per-day 2,000 x 30 days, the same for every job." in flat + assert "Add --calls-per-day" not in flat + + +def test_scan_is_first_in_help_welcome_and_readme() -> None: + assert HELP_ORDER[0] == "scan" + assert GETTING_STARTED.index("throttle scan .") < GETTING_STARTED.index("throttle demo") + from throttle.cli import WELCOME_STEPS + assert WELCOME_STEPS[0][2] == ("throttle scan .",) + readme = (REPO / "README.md").read_text(encoding="utf-8") + assert "throttle.own.scan/1" in readme and "throttle scan ." in readme + + +def _version(text: str) -> tuple[int, ...]: + return tuple(int(part) for part in text.split(".")) + + +def _pinned_blocks(text: str) -> list[tuple[str, str]]: + """(pinned version, code block) for each code block that pins throttle-pro.""" + out = [] + for block in re.findall(r"```(?:sh|bash)\n(.*?)```", text, re.S): + found = re.search(r'throttle-pro==([0-9.]+)"', block) + if found: + out.append((found.group(1), block)) + return out + + +def test_install_steps_never_run_scan_on_a_release_without_it() -> None: + """`throttle scan` first ships in FIRST_RELEASE. A pinned install block may + only run it once its pin is at least that, and once the package version + reaches it the README's first block must pin it and run scan first.""" + first = _version(scan_cost.FIRST_RELEASE) + version = tomllib.loads((REPO / "pyproject.toml").read_text(encoding="utf-8"))["project"]["version"] + docs = REPO / "src/throttle/console/docs/getting-started" + for path in (REPO / "README.md", docs / "quickstart.md", docs / "scan.md", docs / "installation.md"): + text = path.read_text(encoding="utf-8") + for pin, block in _pinned_blocks(text): + if "throttle scan" in block: + assert _version(pin) >= first, (path.name, pin) + if "throttle scan" in text and any(_version(pin) < first for pin, _ in _pinned_blocks(text)): + assert scan_cost.FIRST_RELEASE in text, path.name # says which release has it + readme = (REPO / "README.md").read_text(encoding="utf-8") + start = readme.index("## Try it in a minute") + pin, block = _pinned_blocks(readme[start:])[0] + if _version(version) >= first: + assert _version(pin) >= first and block.splitlines()[1] == "throttle scan .", (version, pin) + else: + assert block.splitlines() == [f'pip install "throttle-pro=={pin}"', "throttle demo"] + + +# --------------------------------------------------------------------------- +# Real-repo regressions: calls the first version missed or miscounted +# --------------------------------------------------------------------------- + + +def scan_tree(root: Path, files: dict[str, str]) -> list[scan.CallSite]: + for rel, source in files.items(): + path = root / rel + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(source, encoding="utf-8") + return scan.scan_dir(root) + + +AGENT_PY = """from anthropic import Anthropic + + +class Agent: + def call(self, params): + return self.client.messages.create( + **params, + extra_headers={"anthropic-beta": "x"}, + ) +""" + +LOOP_PY = """import anthropic + + +def sample(client, model, messages, betas): + api_kwargs = {"max_tokens": 4096, "messages": messages, "model": model, "system": ["x"]} + if betas: + return client.beta.messages.create(**api_kwargs) + return client.messages.create(**api_kwargs) + + +def raw(client, messages): + return client.beta.messages.with_raw_response.create( + max_tokens=1024, messages=messages, model="claude-sonnet-4-5", + ) +""" + +# Twilio has client.messages.create too: no anthropic import, not an AI call +SMS_PY = """from twilio.rest import Client + + +def send(client, **kwargs): + return client.messages.create(**kwargs) +""" + +LLM_TS = """import Anthropic from "@anthropic-ai/sdk"; + +export async function run(client: Anthropic, params: any) { + return client.messages.create(params); +} +""" + + +def test_anthropic_calls_through_kwargs_and_raw_response_are_found(tmp_path) -> None: + # anthropic-quickstarts: agents/agent.py:113, browser-use-demo loop.py:137/140, + # computer-use-demo loop.py:168 were missed before + sites = scan_tree(tmp_path, {"agent.py": AGENT_PY, "loop.py": LOOP_PY, "sms.py": SMS_PY, "llm.ts": LLM_TS}) + found = {(s.path, s.line): s for s in sites} + assert set(found) == {("agent.py", 6), ("loop.py", 7), ("loop.py", 8), ("loop.py", 12), ("llm.ts", 4)} + assert all(s.provider == "anthropic" and s.api == "messages" for s in sites) + assert found[("agent.py", 6)].model is None and found[("agent.py", 6)].model_spread == "**params" + assert found[("loop.py", 7)].model_expr == "model" # read from the api_kwargs dict + assert found[("loop.py", 12)].model == "claude-sonnet-4-5" + assert found[("llm.ts", 4)].model_spread == "params" + + +def test_ai_sdk_name_with_a_non_object_argument_is_not_a_call(tmp_path) -> None: + # LibreChat: a local streamText(part.text) helper + sites = scan_tree(tmp_path, {"a.js": ( + "const streamText = (t) => res.write(t);\n" + "for (const part of content) { streamText(part.text); }\n" + 'await streamText({ model: openai("gpt-4o"), prompt: "hi" });\n' + "await generateText(options);\n")}) + assert [(s.line, s.model) for s in sites] == [(3, "gpt-4o"), (4, None)] + + +def test_a_file_without_any_ai_call_word_is_not_parsed(monkeypatch) -> None: + calls = [] + monkeypatch.setattr(scan, "scan_javascript", lambda *a: calls.append(a) or []) + assert scan.scan_source("big.ts", "export const x = 1;\n" * 1000) == [] + assert calls == [] + scan.scan_source("ai.ts", "await client.chat.completions.create({ model: 'gpt-4o' })") + assert len(calls) == 1 + + +@pytest.mark.parametrize("source,default", [ + ('import os\nclient.chat.completions.create(model=os.getenv("CHAT_MODEL", "gpt-4o"), messages=[])\n', "gpt-4o"), + ('import os\nM = os.environ.get("M", "gpt-4o-mini")\nclient.chat.completions.create(model=M, messages=[])\n', + "gpt-4o-mini"), + ('def f(model="gpt-4.1"):\n client.chat.completions.create(model=model, messages=[])\n', "gpt-4.1"), + ('def f(*, model: str = "o3"):\n client.chat.completions.create(model=model, messages=[])\n', "o3"), + ('client.chat.completions.create(model=cfg.model or "gpt-4o", messages=[])\n', "gpt-4o"), + ('client.chat.completions.create(model=cfg.get("model", "gpt-4o"), messages=[])\n', "gpt-4o"), + ('client.chat.completions.create(model=cfg.model, messages=[])\n', None), + ('client.chat.completions.create(model=os.getenv("M", ""), messages=[])\n', None), +]) +def test_python_literal_defaults_of_a_runtime_model(source, default) -> None: + [site] = scan.scan_python("a.py", source) + assert site.model is None and site.model_default == default + + +@pytest.mark.parametrize("expr,default", [ + ('process.env.CHAT_MODEL ?? "gpt-4o"', "gpt-4o"), + ("process.env.CHAT_MODEL || 'claude-haiku-4-5'", "claude-haiku-4-5"), + ("MODEL", "gpt-4.1"), # const MODEL = process.env.X ?? "gpt-4.1" + ("settings.model", None), +]) +def test_js_literal_defaults_of_a_runtime_model(expr, default) -> None: + source = ('const MODEL = process.env.X ?? "gpt-4.1";\n' + f'await client.chat.completions.create({{ model: {expr}, messages: [] }});\n') + [site] = scan.scan_javascript("a.ts", source) + assert site.model is None and site.model_default == default + + +RUNTIME_REPO = { + "app/chat.py": ( + "import os\n" + "from openai import OpenAI\n\n" + "client = OpenAI()\n\n\n" + "def chat(q):\n" + ' return client.chat.completions.create(model=os.getenv("CHAT_MODEL", "gpt-4o"), messages=[])\n\n\n' + "def route(q, model):\n" + " return client.chat.completions.create(model=model, messages=[])\n" + ), +} + + +def test_default_in_code_is_priced_and_labelled(tmp_path, capsys, plain) -> None: + scan_tree(tmp_path, RUNTIME_REPO) + code, out, _ = run(capsys, str(tmp_path), "--json") + jobs = {j["function"]: j for j in json.loads(out)["jobs"]} + assert jobs["chat"]["status"] == STATUS_PRICED and jobs["chat"]["model"] is None + assert (jobs["chat"]["priced_model"], jobs["chat"]["model_source"]) == ("gpt-4o", "default_in_code") + assert jobs["route"]["status"] == STATUS_RUNTIME + code, out, _ = run(capsys, str(tmp_path)) + flat = flatten(out) + assert "gpt-4o (default)" in flat + assert "(default): model set at runtime, priced at the default written in the code." in flat + assert "1 of 2 set the model at runtime. To price them, rerun with the model you use" in flat + + +def test_model_flag_prices_runtime_jobs_and_wins_over_a_default(tmp_path, capsys, plain) -> None: + scan_tree(tmp_path, RUNTIME_REPO) + code, out, _ = run(capsys, str(tmp_path), "--json", "--model", "gpt-4.1") + data = json.loads(out) + assert data["assumptions"]["runtime_model"] == "gpt-4.1" + assert data["assumptions"]["runtime_model_source"] == "flag" + for job in data["jobs"]: + assert (job["status"], job["priced_model"], job["model_source"]) == (STATUS_PRICED, "gpt-4.1", "flag") + code, out, _ = run(capsys, str(tmp_path), "--model", "gpt-4.1") + flat = flatten(out) + assert "gpt-4.1 (--model)" in flat + assert "(--model): model set at runtime, priced as gpt-4.1 (your flag)." in flat + assert "set the model at runtime" not in flat + with pytest.raises(SystemExit): + build_parser().parse_args(["scan", "--model", "two words"]) + + +def test_tests_examples_and_evals_are_not_counted_unless_all(tmp_path, capsys, plain) -> None: + call = 'client.chat.completions.create(model="gpt-4o", messages=[])\n' + js_call = 'await client.chat.completions.create({ model: "gpt-4o", messages: [] });\n' + scan_tree(tmp_path, {"src/app.py": call, "tests/test_app.py": call, "src/__tests__/x.spec.ts": js_call, + "web/chat.test.ts": js_call, "examples/demo.py": call, "evals/run.py": call, + "scripts/latency_benchmark.py": call, "conftest.py": call}) + code, out, _ = run(capsys, str(tmp_path), "--json") + data = json.loads(out) + assert (data["summary"]["calls_found"], data["summary"]["counted"], + data["summary"]["not_counted_tests_examples"]) == (8, 1, 7) + assert {j["path"]: j["where"] for j in data["jobs"]}["examples/demo.py"] == "example" + assert [j["path"] for j in data["jobs"] if j["counted"]] == ["src/app.py"] + flat = flatten(run(capsys, str(tmp_path))[1]) + assert "1 AI call found, 1 priced." in flat + assert "Not counted: 7 more in tests, examples or evals. --all includes them." in flat + flat = flatten(run(capsys, str(tmp_path), "--all")[1]) + assert "8 AI calls found, 8 priced." in flat and "Not counted" not in flat + # a repo that is only examples still gets a result + only = tmp_path / "only" + scan_tree(only, {"examples/a.py": call}) + flat = flatten(run(capsys, str(only))[1]) + assert "1 AI call found, 1 priced." in flat and "All 1 are in tests, examples or evals" in flat + + +def test_table_keeps_file_and_line_sorts_by_cost_and_caps_rows(tmp_path, capsys, plain) -> None: + deep = "src/khoj/processor/conversation/openai/utils.py" + models = ["gpt-4o-mini"] * 16 + ["gpt-4o", "o3"] + body = "".join(f'def f{i}():\n client.chat.completions.create(model="{m}", messages=[])\n\n' + for i, m in enumerate(models)) + scan_tree(tmp_path, {deep: body}) + out = run(capsys, str(tmp_path))[1] + flat = flatten(out) + assert "/utils.py:2 " in flat and "src/khoj/processor/conve.." not in flat # cut from the left + # f16 is gpt-4o ($5.50 per 1K), f17 is o3 ($4.40), f0 to f15 are gpt-4o-mini + assert flat.index("f16 gpt-4o ") < flat.index("f17 o3 ") < flat.index("f0 gpt-4o-mini ") + assert "3 more priced calls, each cheaper per call than these. --all lists them." in flat + assert out.count("gpt-4o-mini ") >= 13 + flat = flatten(run(capsys, str(tmp_path), "--all")[1]) + assert "more priced calls" not in flat + + +def test_cap_marker_survives_a_long_model_name(tmp_path, capsys, plain) -> None: + scan_tree(tmp_path, {"a.py": 'import anthropic\nclient.messages.create(model="claude-sonnet-4-5-20250929", ' + 'max_tokens=50, messages=[])\n'}) + flat = flatten(run(capsys, str(tmp_path))[1]) + assert re.search(r"claude-sonnet-4-5-20\S*\.\.\* \$", flat), flat + assert "* output capped by a literal max_tokens in the code." in flat + + +def test_not_priced_list_folds_by_file_and_reason_and_names_the_cause(tmp_path, capsys, plain) -> None: + scan_tree(tmp_path, { + "llm.py": "".join(f"def f{i}(model_name):\n client.chat.completions.create(model=model_name, " + "messages=[])\n" for i in range(3)), + "old.py": ('client.chat.completions.create(model="gpt-3.5-turbo", messages=[])\n' + 'client.messages.create(model="claude-3-haiku-20240307", max_tokens=5, messages=[])\n'), + "other.ts": ('await generateText({ model: google("gemini-1.5-flash"), prompt: "x" });\n' + "await client.chat.completions.create({\n model: pick(\n a,\n b,\n ),\n" + " messages: [] });\n"), + }) + flat = flatten(run(capsys, str(tmp_path))[1]) + assert "- llm.py: 3 calls, lines 2, 4, 6 model set at runtime (model_name)" in flat + assert "gpt-3.5-turbo: older model, not in the price table, prices checked 2026-10-27" in flat + assert "claude-3-haiku-20240307: older model, not in the price table" in flat + assert "gemini-1.5-flash: not an OpenAI or Anthropic model; add its price with --prices FILE" in flat + assert "model set at runtime (pick( a, b, ))" in flat # whitespace collapsed + # nothing priced: no table, no legend, a concrete next line + assert "JOB" not in flat and "cheaper at list price" not in flat + assert "set the model at runtime. To price them, rerun with the model you use, for example --model" in flat + + +def test_anthropic_only_repo_gets_claude_judge_and_anthropic_endpoint(tmp_path, capsys, plain) -> None: + scan_tree(tmp_path, {"a.py": 'import anthropic\nclient.messages.create(model="claude-sonnet-4-5", ' + 'max_tokens=1024, messages=[])\n'}) + code, out, _ = run(capsys, str(tmp_path), "--json") + steps = json.loads(out)["next_steps"] + assert steps[3] == ("throttle shadow --candidate-url https://api.anthropic.com/v1 --api-key-env " + "ANTHROPIC_API_KEY --candidate-model claude-haiku-4-5 --label claude-haiku-4-5") + assert steps[4] == ("throttle eval --candidate claude-haiku-4-5 --judge-provider anthropic " + "--judge-model claude-sonnet-4-6 --judge-key-env ANTHROPIC_API_KEY") + assert not any("URL " in s or "KEY_VAR" in s or "OPENAI_API_KEY" in s for s in steps) + + +def test_the_judge_is_never_the_candidate() -> None: + table = pricing.load_prices() + for model, provider in (("claude-opus-4-8", "anthropic"), ("gpt-5.2", "openai"), ("gpt-4.1", "openai")): + job = scan_cost.cost_job(scan.CallSite(path="a.py", line=1, function="f", language="python", + provider=provider, api="messages", model=model), table, Shape()) + steps = scan_cost.next_steps([job]) + if steps: + pick = job.candidates[0].price.model + assert f"--judge-model {pick} " not in steps[4], (model, steps[4]) + + +def test_narrow_terminal_drops_the_change_column(tmp_path, capsys, monkeypatch) -> None: + monkeypatch.setenv("NO_COLOR", "1") + monkeypatch.setenv("COLUMNS", "40") + monkeypatch.delenv("FORCE_COLOR", raising=False) + write_fixture(tmp_path) + out = run(capsys, str(tmp_path))[1] + head = next(line for line in out.splitlines() if line.startswith("JOB")) + assert "CHANGE" not in head and len(head) <= 40 + rows = [line for line in out.splitlines() if "$" in line and "↳" in line] + assert rows and all("%" not in line and len(line) <= 40 for line in rows) diff --git a/tests/test_style_and_panels.py b/tests/test_style_and_panels.py index a275af3..ab20631 100644 --- a/tests/test_style_and_panels.py +++ b/tests/test_style_and_panels.py @@ -174,7 +174,7 @@ def test_narrow_terminal_gets_panels_without_a_frame(monkeypatch) -> None: assert box.splitlines()[0] == "Verdict" screen = welcome_screen() assert not any(ch in screen for ch in "┌┐└┘") - assert "1. See the whole loop" in screen + assert "1. Find the AI calls" in screen and "2. See the whole loop" in screen # 60 columns and up keep the frame. monkeypatch.setenv("COLUMNS", "60") assert Style(colour=False, unicode=True).panel(["x"]).startswith("┌") @@ -433,7 +433,7 @@ def readme_section(title: str) -> str: def test_readme_install_and_action_are_pinned() -> None: text = (REPO / "README.md").read_text(encoding="utf-8") - assert 'pip install "throttle-pro==0.7.0"' in text + assert 'pip install "throttle-pro==0.8.0"' in text assert "PyPI is behind" not in text assert f"uses: KushagraKanaujia/throttle/.github/actions/throttle-pr@{PINNED_SHA} # v0.7.0" in text assert "Pin the action by its full commit SHA" in text @@ -441,7 +441,7 @@ def test_readme_install_and_action_are_pinned() -> None: assert not re.search(r"pip install git\+https://github\.com/KushagraKanaujia/throttle\s*$", text, re.M) for doc in ("getting-started/installation.md", "getting-started/quickstart.md"): body = (REPO / "src/throttle/console/docs" / doc).read_text(encoding="utf-8") - assert 'throttle-pro==0.7.0' in body, doc + assert 'throttle-pro==0.8.0' in body, doc assert "behind until" not in body, doc assert "own/mvp" not in body and "pipx upgrade" not in body, doc