Repository navigation
fix(pricing): price Qwen on Bedrock, and stop a cacheless model costing zero - #186
Conversation
…ng zero
A codex run against qwen.qwen3-coder-30b-a3b on Bedrock recorded
total_cost_usd: null, which plots as a free model. Two separate causes.
The model had no price row. Added it from the Bedrock pricing page, Standard
tier: $0.1545 per 1M input, $0.6180 per 1M output. Flex and Batch are both
$0.0773/$0.3090 and Priority is $0.2704/$1.0815 if a run ever needs another
tier. qwen.qwen3-coder-480b-a35b is deliberately absent: its cells on that page
render client-side and did not resolve, so there is no number to cite yet.
A row alone would not have fixed it. _rates stripped only the us./global./eu./ap.
inference-profile prefixes, so neither spelling that reaches this module
resolved: the Bedrock id carries a version suffix (qwen.qwen3-coder-30b-a3b-v1:0)
and the harness records the LiteLLM alias (qwen.qwen3-coder-30b-a3b-instruct).
_normalize now strips the prefix, a -vN:M version suffix and a trailing
-instruct, so all three spellings price the same.
Qwen does not support prompt caching on Bedrock. A Converse call carrying a
cachePoint block is rejected with AccessDeniedException ("You invoked an
unsupported model or your request did not allow prompt caching") while the same
request against claude-haiku-4-5 caches 5,204 tokens, and the pricing page
publishes no cache columns for the family. So the row carries no cache_read or
cache_write rate, and cost_usd now returns None when a caller reports cached
tokens against a row that has none. Pricing them at zero would understate the
bill and read exactly like a genuinely cheap run.
The hello-world task that recorded null now prices at $0.117154 from its
726,017 fresh input and 8,065 output tokens.
test_price_table_rows_are_complete asserted all four rate keys on every row,
which is no longer the contract, so it now requires input and output and checks
that whatever rates a row does carry are non-negative. Two tests added: all
three spellings price identically, and cached tokens against an uncached model
return None. 470 tests pass.
…teway-registry-v2 The run that proves the price row in the previous commit. 19 of 21 tasks scored, mean 42.22, at $0.6204 per task and $13.17 for the run. Two model failures excluded from the mean: cli-custom-egress-oauth-provider-flags filed no artifacts after 187 turns and 3.5M input tokens, and registration-admission-control-gate is missing review.md. Route: codex exec (Responses API) -> a LiteLLM proxy -> Bedrock Converse in us-east-2. Bedrock serves the Responses API only for the openai.* family, so Qwen rejects it with "does not support the '/openai/v1/responses' API" and codex 0.153.4 refuses wire_api = "chat"; a translating proxy is therefore mandatory, and Converse is the only surface of the two Qwen does serve whose usage block can report cache tokens at all. cache_read is 0 on every task across 83,488,690 input tokens, and that is the truth rather than a plumbing gap. Bedrock rejects a Converse cachePoint for this family, verified against claude-haiku-4-5 caching 5,204 tokens on the same request shape. So every token here is billed fresh, which is what separates $13.17 metered from the $5.56 a comparable self-hosted codex run costs on hardware-derived GPU-seconds at a 96.4% prefix-cache hit rate. Not charted. The harness is codex, which no published chart scans, and the cost is a metered Bedrock bill that does not compare as raw dollars with the hardware-derived self-hosted points. run-summary.md is not committed: .gitignore keeps run-summary.json, metrics.json and eval.json per run and excludes the markdown.
|
Added the run that proves the price row: codex on Bedrock-hosted Two model failures, excluded from the mean:
Route, for the record: Not charted. The harness is codex, which no published chart scans, and this is a metered Bedrock bill that does not compare as raw dollars with the hardware-derived self-hosted points. |
…0 notes (#187) The 0.2.0 notes described the release as it stood when they were written, but #186 merged eleven seconds earlier and carried two things they never mentioned: Qwen3-Coder-30B priced on Bedrock, and a new benchmarked model. The notes now cover both, and record the pricing fix under config changes and bug fixes. codex had no setup page. omp and kiro-cli each have one; codex's wiring lived scattered across the harness reference, the benchmark skill and a run-swe-headless docstring, so anyone pointing it at a model had to rediscover it. Two new documents, split so no fact lives in both: - docs/codex-setup.md installs codex, wires it to Bedrock, states the constraint that decides everything else (it speaks only the Responses API, which Bedrock serves for openai.* models alone), and tabulates the failure modes that cost real debugging time: the stdin hang, the ignored OPENAI_BASE_URL, the tool parsers that abort on Responses-shaped tools, the null cost of an unpriced model. - docs/faq/ holds one file per question with an index. Its first entry answers how to wire codex to an open-weight model on Bedrock (the LiteLLM Responses-to-Converse bridge, and why the committed litellm-mantle.yaml cannot serve that path), to an open-weight model on vLLM (the provider block, the Responses-safe parsers, the context window), and to an OpenAI model on Bedrock (nothing to bridge). Both are linked from the README documentation map, the AGENTS.md setup list and the harness reference's agent entry. Also commits the Hello-World smoke run that validated the codex-to-Bedrock route. Its metrics.json carries total_cost_usd: null because it ran before the Qwen price row existed; the 21-task run committed in #186 has real per-task costs. The pre-commit whitespace hooks trimmed trailing spaces in three of its generated markdown artifacts and added a final newline to run-summary.md; the content is otherwise the model's output as written. Every relative link in the touched documents resolves, and every file in docs/faq/ appears in its index.
A codex run against
qwen.qwen3-coder-30b-a3bon Bedrock recordedtotal_cost_usd: null, which plots as a free model on the cost/quality frontier. Two separate causes, both fixed here.The model had no price row
Added from the Bedrock pricing page, Standard tier:
$0.1545per 1M input,$0.6180per 1M output. Other tiers for the same model, if a run ever needs one: Flex and Batch both$0.0773 / $0.3090, Priority$0.2704 / $1.0815.qwen.qwen3-coder-480b-a35bis deliberately absent. Its cells on the pricing page render client-side ({priceOf!bedrock/...}placeholders) and did not resolve, so there is no number to cite.pricing:GetProductsis denied for this instance role, so the API could not confirm it either.A row alone would not have fixed it
_ratesstripped only theus./global./eu./ap.inference-profile prefixes. Neither spelling that actually reaches the module resolved:qwen.qwen3-coder-30b-a3b-v1:0--model:qwen.qwen3-coder-30b-a3b-instruct_normalizenow strips the prefix, a-vN:Mversion suffix, and a trailing-instruct, so all three spellings resolve to one row.Qwen does not support prompt caching on Bedrock
Measured, not assumed. Identical Converse requests, same region, same
cachePointblock, only the model changed:qwen.qwen3-coder-30b-a3b-v1:0AccessDeniedException: You invoked an unsupported model or your request did not allow prompt cachingus.anthropic.claude-haiku-4-5cacheWriteInputTokens: 5204; warm:cacheReadInputTokens: 5204The pricing page also publishes no cache columns for the Qwen family. So the row carries no
cache_read/cache_writerate, andcost_usdreturnsNonewhen a caller reports cached tokens against a row that has none. Pricing them at zero would understate the bill and be indistinguishable from a genuinely cheap run.Result
The hello-world task that recorded
nullnow prices at $0.117154 from its 726,017 fresh input and 8,065 output tokens.Tests
test_price_table_rows_are_completeasserted all four rate keys on every row. That is no longer the contract, so re-pinning it would have been wrong: it now requiresinputandoutputand checks that whatever rates a row does carry are non-negative. Two added:None, not a silent zero470 tests pass(was 468).ruff formatandruff checkclean,banditclean.