Skip to content

fix(pricing): price Qwen on Bedrock, and stop a cacheless model costing zero - #186

Merged
aarora79 merged 2 commits into
mainfrom
fix/bedrock-pricing-qwen
Sep 11, 2026
Merged

aarora79 merged 2 commits into
mainfrom
fix/bedrock-pricing-qwen

Conversation

@aarora79

Copy link
Copy Markdown
Owner

A codex run against qwen.qwen3-coder-30b-a3b on Bedrock recorded total_cost_usd: null, which plots as a free model on the cost/quality frontier. Two separate causes, both fixed here.

The model had no price row

Added from the Bedrock pricing page, Standard tier: $0.1545 per 1M input, $0.6180 per 1M output. Other tiers for the same model, if a run ever needs one: Flex and Batch both $0.0773 / $0.3090, Priority $0.2704 / $1.0815.

qwen.qwen3-coder-480b-a35b is deliberately absent. Its cells on the pricing page render client-side ({priceOf!bedrock/...} placeholders) and did not resolve, so there is no number to cite. pricing:GetProducts is denied for this instance role, so the API could not confirm it either.

A row alone would not have fixed it

_rates stripped only the us. / global. / eu. / ap. inference-profile prefixes. Neither spelling that actually reaches the module resolved:

  • the Bedrock id carries a version suffix: qwen.qwen3-coder-30b-a3b-v1:0
  • the harness records the LiteLLM alias as --model: qwen.qwen3-coder-30b-a3b-instruct

_normalize now strips the prefix, a -vN:M version suffix, and a trailing -instruct, so all three spellings resolve to one row.

Qwen does not support prompt caching on Bedrock

Measured, not assumed. Identical Converse requests, same region, same cachePoint block, only the model changed:

Model Result
qwen.qwen3-coder-30b-a3b-v1:0 AccessDeniedException: You invoked an unsupported model or your request did not allow prompt caching
us.anthropic.claude-haiku-4-5 cold: cacheWriteInputTokens: 5204; warm: cacheReadInputTokens: 5204

The pricing page also publishes no cache columns for the Qwen family. So the row carries no cache_read / cache_write rate, and cost_usd returns None when a caller reports cached tokens against a row that has none. Pricing them at zero would understate the bill and be indistinguishable from a genuinely cheap run.

Result

The hello-world task that recorded null now prices at $0.117154 from its 726,017 fresh input and 8,065 output tokens.

Tests

test_price_table_rows_are_complete asserted all four rate keys on every row. That is no longer the contract, so re-pinning it would have been wrong: it now requires input and output and checks that whatever rates a row does carry are non-negative. Two added:

  • all three spellings of the Qwen id price identically
  • cached tokens against an uncached model return None, not a silent zero

470 tests pass (was 468). ruff format and ruff check clean, bandit clean.

…ng zero

A codex run against qwen.qwen3-coder-30b-a3b on Bedrock recorded
total_cost_usd: null, which plots as a free model. Two separate causes.

The model had no price row. Added it from the Bedrock pricing page, Standard
tier: $0.1545 per 1M input, $0.6180 per 1M output. Flex and Batch are both
$0.0773/$0.3090 and Priority is $0.2704/$1.0815 if a run ever needs another
tier. qwen.qwen3-coder-480b-a35b is deliberately absent: its cells on that page
render client-side and did not resolve, so there is no number to cite yet.

A row alone would not have fixed it. _rates stripped only the us./global./eu./ap.
inference-profile prefixes, so neither spelling that reaches this module
resolved: the Bedrock id carries a version suffix (qwen.qwen3-coder-30b-a3b-v1:0)
and the harness records the LiteLLM alias (qwen.qwen3-coder-30b-a3b-instruct).
_normalize now strips the prefix, a -vN:M version suffix and a trailing
-instruct, so all three spellings price the same.

Qwen does not support prompt caching on Bedrock. A Converse call carrying a
cachePoint block is rejected with AccessDeniedException ("You invoked an
unsupported model or your request did not allow prompt caching") while the same
request against claude-haiku-4-5 caches 5,204 tokens, and the pricing page
publishes no cache columns for the family. So the row carries no cache_read or
cache_write rate, and cost_usd now returns None when a caller reports cached
tokens against a row that has none. Pricing them at zero would understate the
bill and read exactly like a genuinely cheap run.

The hello-world task that recorded null now prices at $0.117154 from its
726,017 fresh input and 8,065 output tokens.

test_price_table_rows_are_complete asserted all four rate keys on every row,
which is no longer the contract, so it now requires input and output and checks
that whatever rates a row does carry are non-negative. Two tests added: all
three spellings price identically, and cached tokens against an uncached model
return None. 470 tests pass.
…teway-registry-v2

The run that proves the price row in the previous commit. 19 of 21 tasks
scored, mean 42.22, at $0.6204 per task and $13.17 for the run. Two model
failures excluded from the mean: cli-custom-egress-oauth-provider-flags filed
no artifacts after 187 turns and 3.5M input tokens, and
registration-admission-control-gate is missing review.md.

Route: codex exec (Responses API) -> a LiteLLM proxy -> Bedrock Converse in
us-east-2. Bedrock serves the Responses API only for the openai.* family, so
Qwen rejects it with "does not support the '/openai/v1/responses' API" and codex
0.153.4 refuses wire_api = "chat"; a translating proxy is therefore mandatory,
and Converse is the only surface of the two Qwen does serve whose usage block
can report cache tokens at all.

cache_read is 0 on every task across 83,488,690 input tokens, and that is the
truth rather than a plumbing gap. Bedrock rejects a Converse cachePoint for this
family, verified against claude-haiku-4-5 caching 5,204 tokens on the same
request shape. So every token here is billed fresh, which is what separates
$13.17 metered from the $5.56 a comparable self-hosted codex run costs on
hardware-derived GPU-seconds at a 96.4% prefix-cache hit rate.

Not charted. The harness is codex, which no published chart scans, and the cost
is a metered Bedrock bill that does not compare as raw dollars with the
hardware-derived self-hosted points.

run-summary.md is not committed: .gitignore keeps run-summary.json, metrics.json
and eval.json per run and excludes the markdown.
@aarora79

Copy link
Copy Markdown
Owner Author

Added the run that proves the price row: aa829a0.

codex on Bedrock-hosted qwen.qwen3-coder-30b-a3b-instruct, /swe3 over mcp-gateway-registry-v2. 19 of 21 tasks scored, mean 42.22, $0.6204 per task, $13.17 for the run. Every per-task cost now populates where this model previously recorded null, which is the fix working end to end.

Two model failures, excluded from the mean: cli-custom-egress-oauth-provider-flags filed no artifacts after 187 turns and 3.5M input tokens, and registration-admission-control-gate is missing review.md.

cache_read is 0 on every task across 83,488,690 input tokens. That is the truth, not a gap: Bedrock rejects a Converse cachePoint for the Qwen family, verified against claude-haiku-4-5 caching 5,204 tokens on the same request shape. Every token here is billed fresh, which is the whole difference between $13.17 metered and the $5.56 a comparable self-hosted codex run costs on hardware-derived GPU-seconds at a 96.4% prefix-cache hit rate.

Route, for the record: codex exec (Responses API) to a LiteLLM proxy to Bedrock Converse in us-east-2. Bedrock serves Responses only for the openai.* family, and codex 0.153.4 refuses wire_api = "chat", so a translating proxy is mandatory for any open-weight Bedrock model under this harness.

Not charted. The harness is codex, which no published chart scans, and this is a metered Bedrock bill that does not compare as raw dollars with the hardware-derived self-hosted points.

@aarora79
aarora79 merged commit 844d724 into main Sep 11, 2026
5 checks passed
aarora79 added a commit that referenced this pull request Sep 11, 2026
…0 notes (#187)

The 0.2.0 notes described the release as it stood when they were written, but
#186 merged eleven seconds earlier and carried two things they never mentioned:
Qwen3-Coder-30B priced on Bedrock, and a new benchmarked model. The notes now
cover both, and record the pricing fix under config changes and bug fixes.

codex had no setup page. omp and kiro-cli each have one; codex's wiring lived
scattered across the harness reference, the benchmark skill and a
run-swe-headless docstring, so anyone pointing it at a model had to rediscover
it. Two new documents, split so no fact lives in both:

- docs/codex-setup.md installs codex, wires it to Bedrock, states the constraint
  that decides everything else (it speaks only the Responses API, which Bedrock
  serves for openai.* models alone), and tabulates the failure modes that cost
  real debugging time: the stdin hang, the ignored OPENAI_BASE_URL, the tool
  parsers that abort on Responses-shaped tools, the null cost of an unpriced
  model.
- docs/faq/ holds one file per question with an index. Its first entry answers
  how to wire codex to an open-weight model on Bedrock (the LiteLLM
  Responses-to-Converse bridge, and why the committed litellm-mantle.yaml cannot
  serve that path), to an open-weight model on vLLM (the provider block, the
  Responses-safe parsers, the context window), and to an OpenAI model on Bedrock
  (nothing to bridge).

Both are linked from the README documentation map, the AGENTS.md setup list and
the harness reference's agent entry.

Also commits the Hello-World smoke run that validated the codex-to-Bedrock
route. Its metrics.json carries total_cost_usd: null because it ran before the
Qwen price row existed; the 21-task run committed in #186 has real per-task
costs. The pre-commit whitespace hooks trimmed trailing spaces in three of its
generated markdown artifacts and added a final newline to run-summary.md; the
content is otherwise the model's output as written.

Every relative link in the touched documents resolves, and every file in
docs/faq/ appears in its index.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant