Cost optimisation: token/cost observability, reasoning-effort tiers, pricing override - #85
Merged
Merged
Conversation
Capture the API usage object that providers were discarding and surface it, so cost can be measured (and demoed) rather than guessed at. - Providers gain invoke_verbose() returning text + normalised token usage; invoke() delegates to it, so custom/Codex/third-party providers are unaffected (usage reported as None). - OpenAI (chat + responses) and Anthropic capture their usage objects; oas_cli/usage.py normalises the differing shapes to one canonical dict and attaches a best-effort estimated_cost_usd for models in a small price table (None when unknown — never guessed). - invoke_intelligence records usage on a ContextVar; the runner reads it and adds a usage key to the result envelope. oa run shows "<n> tok · ~$<cost>". - Tests for normalisation, cost estimation, the invoke_verbose fallback, and envelope wiring. REFERENCE.md documents the new field. Single-call path only; the multi-turn tool loop is a documented follow-up.
Declare one portable intelligence.config.reasoning_effort and map it to each engine's native reasoning control, so spend can track task difficulty without per-provider spec changes. - oas_cli/reasoning.py centralises the mapping and validation. - OpenAI: reasoning_effort (Chat Completions) / reasoning.effort (Responses). - Codex: -c model_reasoning_effort=<tier> CLI override. - Anthropic: extended-thinking token budget per tier, plus the API-required temperature=1 and max_tokens-above-budget adjustments. Response text extraction now skips leading thinking blocks. - Schema enum so `oa validate` rejects bad tiers before any model call. - Tests for the mapping, each provider payload, and the Codex command. REFERENCE.md documents it as experimental. Spike values (esp. Anthropic budgets) are starting points to validate against real Opus 4.x / Codex, not tuned figures.
Checking the spike against the current Claude API reference showed the Anthropic branch 400s on the target models: thinking.budget_tokens is rejected on Opus 4.6+, and the forced temperature=1 is rejected on Opus 4.7/4.8/Fable 5. - Anthropic now maps reasoning_effort -> output_config.effort (a near 1:1 tier mapping, GA, no beta header) + adaptive thinking, and drops the temperature field. Simpler than the old budget/temperature/max_tokens hack. - Refresh the cost table to current models (Opus 4.x $5/$25, Sonnet 4.6 $3/$15, Haiku 4.5 $1/$5, Fable 5 $10/$50); drop stale claude-3-* entries. Entries are model-specific so older Opus (4.0/4.1) isn't mispriced. - Add scripts/verify_reasoning.py: live smoke-test firing low vs high per engine, printing token deltas (uses the new usage capture) so the mapping can be validated against real models before relying on it. - Update reasoning tests and REFERENCE.md accordingly. Codex mapping (-c model_reasoning_effort=) confirmed against codex-cli 0.114.0.
_build_intelligence_config hardcoded https://api.openai.com/v1 as the endpoint default for every engine. The provider registry merges this over the per-engine defaults (`{**defaults, **config}`, config wins), so it clobbered the correct endpoint for anthropic / grok / local / cortex — only OpenAI worked by accident. A no-endpoint `engine: anthropic` spec (the README's own example) routed to api.openai.com and 404'd. Surfaced by the live reasoning validation. - Only carry an explicit endpoint; when absent, each provider applies its own default (and the registry supplies grok/local/cortex defaults as before). - Harden the two custom-engine codegen paths to config.get("endpoint", ""). - Regression test: no-endpoint anthropic/grok/local specs don't inherit the OpenAI URL. - verify_reasoning.py: drop the endpoint workaround (now exercises the fixed default path) and add --repeat to average out adaptive-thinking noise. Validated live: Opus 4.8 via the default endpoint, reasoning_effort low vs high over 3 samples — mean 583 vs 838 output tokens (+255), ranges non-overlapping.
A task that declares tools but runs on a provider without native tool support (e.g. Codex) falls back to a single invoke_intelligence call, which records usage in the registry ContextVar — but _run_single_task only popped it in the no-tools branch, so these tasks returned "usage": null even when the call reported usage (flagged in code review). - Pop usage once after the tool/no-tool branch so both paths capture it. The native multi-turn loop still records no per-call usage (unchanged follow-up). Popping on every task also prevents stale ContextVar leakage across tasks. - Regression test for the tool-fallback branch. - REFERENCE.md / CHANGELOG clarified: only the *native* multi-turn loop is null; the text-only tool fallback is captured.
Closes the last usage-capture gap. The native tool-calling loop made several provider calls but recorded no usage, so tool-using tasks on OpenAI/Anthropic returned "usage": null. - InvokeResult carries per-turn usage; OpenAI and Anthropic invoke_with_tools populate it from the API response. - _invoke_with_tools sums prompt/completion across every turn (each turn re-sends the growing history, so the sum is what's actually billed) and records it via the shared registry.record_usage helper — the runner's existing pop_last_usage picks it up, no envelope changes. - record_usage extracted in the registry and reused by invoke_intelligence. - Tests: multi-turn summation + cost enrichment, and None when the provider omits usage. REFERENCE.md / CHANGELOG updated — multi-turn is now covered.
Live testing (code review) showed real OpenAI reasoning calls 400 on o4-mini: "Unsupported parameter: 'max_tokens' ... Use 'max_completion_tokens' instead" — and those models also reject a non-default temperature. The provider always sent max_tokens + temperature, so the reasoning path (the o4-mini used by verify_reasoning.py) broke end to end. - When reasoning_effort is set, OpenAI Chat Completions uses max_completion_tokens and omits temperature (chat + tools paths), via a shared _apply_sampling_and_reasoning helper. Standard models and OpenAI-compatible servers (grok/local/cortex) keep max_tokens + temperature — unchanged. - verify_reasoning.py: add --sleep to space calls for low-RPM accounts (o4-mini free tier is 3 RPM). - Tests for the reasoning-model vs standard-model payload shapes. Validated live: o4-mini low vs high → 1000 vs 1473 output tokens (+473), no 400. Both engines (Opus 4.8 + o4-mini) now confirmed end to end. (o4-mini cost is null — not in the price table — by design, not guessed.)
- Add current OpenAI GPT-5.x rates (gpt-5.5, 5.5-pro, 5.4, 5.4-mini, 5.4-nano, 5.4-pro, 5.3-codex) and legacy o-series (o4-mini, o3) to the price table, sourced from OpenAI's pricing page. o3-mini omitted — output rate unconfirmed (table never guesses). - REFERENCE: estimated_cost_usd is explicitly a pay-as-you-go API list-price estimate. It does not reflect subscriptions (ChatGPT/Claude Max), committed/ negotiated rates, Bedrock/Vertex, or local models. Token counts are always accurate and are the figure to track against any plan.
The default estimated_cost_usd is a list price, which doesn't match
subscriptions, committed-use, negotiated, or local pricing. Let callers
substitute their own rate — or turn cost off where it isn't meaningful.
Rates resolve first-match-wins across three layers:
- per-spec: intelligence.config.pricing = {input_per_1m, output_per_1m} | "none"
- global: OA_PRICING env = {model: {input, output}} (extends table) | "none"
- built-in: the hand-maintained table
"none" at any layer disables cost from there down; a more specific layer can
re-enable with explicit rates. Token counts are always reported regardless.
- estimate_cost_usd grows a `pricing=` kwarg; rates threaded through
record_usage from both the single-call and tool-loop paths.
- Schema: intelligence.config.pricing (oneOf "none" | {input_per_1m,
output_per_1m}); oa validate enforces it (rejects negative rates).
- Tests for every layer + precedence + malformed-env fallthrough.
- REFERENCE / CHANGELOG document the override.
Code review (Codex): an invalid pricing override could quietly misstate cost — _extract_rate accepted negative rates from OA_PRICING, and a malformed per-spec or env override fell through to _NOT_SET, silently reverting to the built-in list price. Since run_task_from_spec doesn't schema-validate, a Python caller or bad OA_PRICING could produce a negative estimate, or a list-price estimate when the operator thought they'd overridden/disabled cost. - A pricing override that is present but invalid now raises InvalidPricingError: negative rates, missing input/output, a pricing string other than "none", and malformed/non-object OA_PRICING all fail loud. - Absent overrides, and models simply not listed in a valid OA_PRICING map, still fall through to the built-in table (unchanged). - Tests for each fail-closed case; REFERENCE / CHANGELOG note the behaviour.
…_ERROR Code review (Codex): InvalidPricingError was caught by the runner's broad `except Exception` and rewritten to RUN_ERROR, so callers couldn't tell operator cost-misconfiguration apart from an arbitrary runtime failure. - _run_single_task now catches InvalidPricingError explicitly and raises OARunError(code="PRICING_CONFIG_ERROR", stage="cost"). - Test asserts a bad config.pricing surfaces that code end to end via run_task_from_spec (not RUN_ERROR). - REFERENCE error-code table + cost section and CHANGELOG document it.
…l spec
Promote the cost/usage layer from implementation-only into the normative spec
and canonical schema (1.5 is Draft, so these additive, backward-compatible
features amend in place):
- §5.5 Reasoning Effort — config.reasoning_effort (low|medium|high), per-engine
mapping, and the allowance to adjust incompatible request params.
- §8.3 Token Usage & Cost — the `usage` envelope block (normalised counts,
null-when-unreported, multi-call summation) and best-effort estimated_cost_usd
(list-price, never guessed); the config.pricing override ("none" | rates) with
fail-closed semantics.
- §11.2 — PRICING_CONFIG_ERROR (stage `cost`).
- spec/schema/oas-schema-1.5.json — reasoning_effort + pricing on
intelligence.config (matches the runtime schema).
int() was called on object-typed values from the parsed usage dict, which mypy rejects (call-overload). Add a small _as_int helper that isinstance-narrows to int/float (anything else → 0), keeping behaviour identical. mypy oas_cli tests: Success, no issues.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
sgriffiths
force-pushed
the
feat/cost-observability
branch
from
June 27, 2026 23:30
9ea5ae3 to
c73cddb
Compare
sgriffiths
marked this pull request as ready for review
June 27, 2026 23:34
….8, Fable 5) Opus 4.7/4.8 and Fable 5 reject sampling params, but the Anthropic provider always sent `temperature` (only dropped on the reasoning path), so a plain non-reasoning call to claude-opus-4-8 returned HTTP 400: "`temperature` is deprecated for this model." Surfaced while building the cost-cascade demo, which couldn't use Opus as its frontier model. - _supports_temperature(model) gates the field by model-id prefix; both the single-call and tool paths omit temperature for those models. - Tests: gate logic + payload capture (omitted for opus-4-8, sent for sonnet). Validated live: plain claude-opus-4-8 call now returns output + usage/cost.
aswhitehouse
reviewed
Jul 3, 2026
aswhitehouse
reviewed
Jul 3, 2026
aswhitehouse
reviewed
Jul 3, 2026
aswhitehouse
reviewed
Jul 3, 2026
aswhitehouse
reviewed
Jul 3, 2026
aswhitehouse
approved these changes
Jul 3, 2026
aswhitehouse
left a comment
aswhitehouse
left a comment
Collaborator
There was a problem hiding this comment.
Review summary
Verdict: Approve with minor nits. Strong, well-tested feature work — CI green, backward-compatible design, and the endpoint / Anthropic reasoning fixes are genuinely important.
Highlights
invoke_verbose()+ContextVaris a clean way to add observability without breaking custom/Codex providers.- Fail-closed pricing (
PRICING_CONFIG_ERROR, layered overrides, never guess rates) is the right semantics for anything involving money. - Endpoint fix (stop hardcoding OpenAI URL in
_build_intelligence_config) fixes a real regression foranthropic/grok/localspecs with no explicit endpoint. - Test coverage (~580 new lines) matches the scope: normalisation, cost layering, reasoning payloads, endpoint regression, tool-fallback usage, multi-turn summation.
- Spec formalisation (§5.5, §8.3,
PRICING_CONFIG_ERROR) keeps the 1.5 draft aligned with the runtime.
Non-blocking notes
- Malformed
OA_PRICINGis intentionally a global kill switch for cost estimation — worth calling out operationally. - Hand-maintained price table will drift; the override layers and "never guess" policy mitigate this.
- Invalid
reasoning_effortvia the Python API (bypassingoa validate) surfaces as genericRUN_ERROR— low priority since the CLI path is covered.
Left inline nits on stale comments/docs and one edge case. Happy to merge once those are addressed (or tracked as follow-ups).
- runner: flush accumulated tool-loop usage via _record() before raising on _MAX_TOOL_ITERATIONS, so spend telemetry survives loop exhaustion (+test) - runner/ui: drop stale "multi-turn tool path doesn't record usage" notes — the native loop now sums and records usage across turns - schema: align reasoning_effort description with the canonical schema wording - spec: reorder §5.4 Custom Engine before §5.5 Reasoning Effort
The tool loop already flushed its per-turn usage before raising, but the value died in the context var — the caller only pops usage on the success path, so a maxed-out loop reported no spend at all. - OARunError carries an optional usage field, emitted in to_dict() - the max-iterations raise attaches the flushed usage and consumes the context var (so nothing leaks into the next task) - print_error_panel renders a token/cost suffix when usage is present - oa run passes err.usage through on both the panel and --quiet JSON paths Covered by tests for the error envelope, to_dict(), and panel rendering.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cost-optimisation work for the agent runtime: full token/cost observability, experimental reasoning-effort tiers, a flexible cost-rate override, and two real bug fixes surfaced along the way — all formalised into the spec. Tokens are captured on every execution path; reasoning tiers are validated live against Opus 4.8 and OpenAI
o4-mini.What's in here
Features
usageobject; now every envelope carries normalisedprompt/completion/total_tokensplus best-effortestimated_cost_usd(nullfor unknown models — never guessed).oa runshows<total> tok · ~$<cost>. Covers single-call, the text-only-provider tool fallback, and the native multi-turn tool loop (summed across turns).intelligence.config.reasoning_effort: low|medium|high, mapped per engine: OpenAIreasoning_effort/reasoning.effort, Codex-c model_reasoning_effort=, Anthropicoutput_config.effort+ adaptive thinking.oa validateenforces the enum.config.pricing→OA_PRICINGenv → built-in table (first match wins);"none"disables cost at any layer. Invalid overrides fail closed withPRICING_CONFIG_ERROR(stagecost). Built-in table refreshed (GPT-5.x, o-series, Claude).Fixes (found via live testing / review)
anthropic/grok/local/cortexspec inherited the OpenAI URL and 404'd — each provider now applies its own default.output_config.effort(the oldbudget_tokens+temperature400'd on current Opus).max_completion_tokens(notmax_tokens) and droptemperature.Spec
config.pricing, §11.2PRICING_CONFIG_ERROR. Additive, backward-compatible (1.5 is Draft).Validation
Automated (CI): 495 passed, 3 skipped; mypy clean; ruff lint + format clean. Mocked-provider tests cover usage normalisation, cost layering + fail-closed (
PRICING_CONFIG_ERROR), reasoning payload shapes, the endpoint regression, and multi-turn usage summation.Manual live runs — not run in CI (needs API keys and a few cents); reproduce with
python scripts/verify_reasoning.py. These exercise the path mocks can't: that the reasoning-effort mapping is accepted by a real model and actually moves tokens, with captured usage/cost matching expectations. Indicative sample runs, not benchmarks:claude-opus-4-8(--repeat 3mean)o4-mini