Skip to content

harness: make agent=codex runnable on the endpoint path, and fix its token accounting - #184

Merged
aarora79 merged 3 commits into
mainfrom
fix/codex-harness-vllm-and-token-accounting
Sep 11, 2026
Merged

aarora79 merged 3 commits into
mainfrom
fix/codex-harness-vllm-and-token-accounting

Conversation

@aarora79

Copy link
Copy Markdown
Owner

Fixes #183.

The harness already implemented a codex agent that no orchestrator would run, and two accounting paths reported wrong token counts for it on the self-hosted path. This makes --agent codex work end to end and fixes the counting.

Routing: an endpoint run reached Amazon Bedrock instead of the endpoint

_build_codex_env exported OPENAI_BASE_URL, and -c model_provider=... was passed only for provider=bedrock. codex 0.153.4 ignores OPENAI_BASE_URL and takes its base URL from the provider its config selects, so an endpoint run went wherever ~/.codex/config.toml pointed. On a machine configured for the judge that is Amazon Bedrock, which answered 404 The model 'minicpm5-2b' does not exist. Pointing codex at model_provider=openai instead reaches wss://api.openai.com and 401s.

_build_codex_cmd now declares its own provider block for provider=endpoint:

-c model_provider=harness_endpoint
-c model_providers.harness_endpoint.base_url=<endpoint>/v1
-c model_providers.harness_endpoint.wire_api=responses
-c model_providers.harness_endpoint.env_key=OPENAI_API_KEY
-c model_context_window=<served window>      # when the config knows it

wire_api = "responses" is not a choice: codex 0.153.4 removed the chat-completions wire and rejects wire_api = "chat". The context-window override silences the Model metadata not found. Defaulting to fallback metadata warning, which otherwise leaves codex sizing its context from a guess.

_codex_base_url validates the endpoint before it reaches argv: it requires http or https and refuses a URL that embeds credentials. The provider block is visible in ps and printed by --dry-run, so a https://user:pass@host endpoint would leak that credential to any local user. The API key stays in OPENAI_API_KEY, and only the variable's name travels on the command line.

Token totals: the cache read could vanish

_codex_result_from_events subtracts codex's cached and cache-written portions out of input_tokens, because codex reports the total prompt with those as subsets. The fields it hands on are therefore disjoint. compute_total_tokens_processed then re-tested them for a partition, and at a cache hit rate near 50% fresh input equals the cache sum, which is exactly its partition signature: it dropped the whole cache read, halved the token count, and halved the cost per task derived from it, while logging that it had prevented a double count.

Callers now declare the shape through cache_partition_for_agent(agent) rather than letting it be guessed. Every recompute site that knows the agent passes it: run-swe-headless.py, summarize_run.py, gen_agent_report.py, plot_cost_quality.py, plot_cost_accuracy_bubble.py and eval_swe_router.py. Detection stays the default for every other agent, so no existing run changes.

The normalizer also sums across turn.completed events instead of overwriting. One exec emits one turn today, verified across seven probes, but a build that splits an exec into several turns (resume, compaction) would otherwise keep only the last.

Retried requests are now visible

codex counts the attempt it accepted and ignores the ones it abandoned, so a flaky endpoint bills the GPU for prefill that never reaches the metrics. Measured on a retrying path: vLLM counted 26,022 prompt tokens across three requests while codex reported 8,700. metrics.json now carries prompt_token_reconciliation for an endpoint run at concurrency 1 -- the agent's prompt tokens against the vllm:prompt_tokens_total window delta, with any shortfall named and logged.

vLLM parser constraint, documented

codex sends flat Responses-shaped tools ({"type": "function", "name": "exec_command", ...}) with stream: true on every request. Seven of vLLM 0.29.0's 51 tool parsers read the nested chat-completions shape (tool.function.name) and abort tool extraction: minicpm5xml, dots, hy_v3, hy_v4, rust, step3, step3p5. Non-streaming returns HTTP 500; streaming returns 200, drops the tool call, and ends with no response.completed, which codex reads as stream disconnected before completion and retries five times before failing the turn. qwen3_coder and hermes never touch those objects and work. The benchmark skill and the FP8 model guide now name both sets.

The model is not the problem: MiniCPM5-2B returns a clean tool call on /v1/chat/completions (finish_reason: tool_calls, {"command": ["ls"]}).

Verification

A real run on qwen3.6-35b-fp8 (vLLM 0.29.0, 1x L40S, 262,144-token window, --tool-call-parser qwen3_coder):

OK: 6/6 artifacts, 32 turns, 32,679 in / 12,434 out tokens, 113.6s,
prefix cache 93.0% hit (435,968/468,647 tokens)
[token-accounting] no partition (DECLARED by the caller) --
total_tokens = 32,679 + 12,434 + 435,968 = 481,081

481,081 equals the server's own 468,647 prompt plus 12,434 generation tokens, and prompt_token_reconciliation reported 0 unaccounted. total_cost_usd stays null on the self-hosted path, as it should.

Also run: uv run python -m unittest discover -s tests (468 tests; the one failure, test_committed_models_json_matches_the_generator, predates this branch -- the committed vend/swe-router/models.json pins commit c559a44 while main is at 5f32859), ruff check, ruff format, bandit -r scripts/ (clean), and mypy scripts/run-swe-headless.py (14 errors, the same 14 the file has on main).

New tests cover the 50%-hit-rate case that used to drop the cache, the declared-versus-detected verdict, multi-turn usage accumulation, the endpoint provider block, and the endpoint validator (credentials, scheme, missing value), plus the reconciliation helper's healthy, retrying, concurrent and counter-missing paths.

…g the cache

The harness implemented a codex agent that no orchestrator would run, and two
accounting paths reported wrong numbers for it on the self-hosted path. Fixes
issue #183.

Routing. codex 0.153.4 ignores OPENAI_BASE_URL and resolves its base URL from
the provider its config selects, so an endpoint run reached whatever
~/.codex/config.toml pointed at -- on a judge-configured machine, Amazon
Bedrock, which answered "404 The model 'minicpm5-2b' does not exist".
_build_codex_cmd now declares its own provider block (base_url, wire_api =
responses, env_key) plus model_context_window when the config knows the served
window. _codex_base_url rejects a non-http scheme and refuses a URL that embeds
credentials, because the provider block travels in argv where ps exposes it;
the key stays in OPENAI_API_KEY.

Token totals. _codex_result_from_events subtracts the cached and cache-written
portions out of codex's input_tokens, so the fields it returns are disjoint.
compute_total_tokens_processed then re-tested them for a partition and, at a
cache hit rate near 50%, matched its own signature and dropped the entire cache
read -- halving the token count and the derived cost per task. Callers now
declare the shape through cache_partition_for_agent instead of having it
guessed. The normalizer also sums across turn.completed events rather than
overwriting, so a build that splits an exec into several turns cannot lose all
but the last.

Retried requests. codex counts the attempt it accepted, so prefill for requests
it abandoned never reaches the metrics. metrics.json now carries
prompt_token_reconciliation for an endpoint run: the agent's prompt tokens
against vllm:prompt_tokens_total, with the shortfall named.

Verified on qwen3.6-35b-fp8 (vLLM 0.29.0, 1x L40S, 262,144-token window):
32 turns, 6/6 artifacts, total_tokens 481,081 = 468,647 prompt + 12,434 output,
matching the server's own counters, reconciliation 0 unaccounted.

vLLM's Responses API needs a tool parser that accepts flat Responses-shaped
tools. Seven of its 51 parsers read tool.function.name and abort tool
extraction, which truncates the stream and makes codex retry every request; the
benchmark skill and the model guide now name the safe and unsafe parsers.
@aarora79 aarora79 added the bug Something isn't working label Sep 11, 2026
…ot-include

Adds the codex /swe3 run over mcp-gateway-registry-v2 for Qwen3.6-35B-A3B FP8
on a single L40S (g6e.4xlarge), the first run of the agent=codex path this
branch fixes. 21 of 21 tasks scored, no failures, 126/126 artifacts, mean
61.45 against the omp run's 59.23.

It lands under do-not-include/qwen3.6-35b-fp8/codex so both harnesses for this
model sit side by side and neither reaches the charts. The exclusion reasons are
unchanged: the model's only throughput sweep is on g6e.4xlarge rather than the
canonical p5en arm, and its token counts are confounded with a 262144 context
window against the BF16 run's 200000. Verified that plot_cost_quality's shallow
scan still finds 19 models for omp and zero scorable codex runs.

Priced on the sweep's cheapest blended rate ($0.0545/1M at concurrency 10,
$1.298/hr), codex costs $0.2645 per task against omp's $0.5555, because it
processes 4.85M tokens per task against 10.19M. The score gap sits inside the
noise: paired delta +2.22, sd 8.27, 95% CI -1.32 to +5.75.

The README entry now covers both runs, including the token-accounting
convention each harness uses, so the input_tokens fields are not compared
across them by mistake.
…mmit

The Test job failed on test_committed_models_json_matches_the_generator:
vend/swe-router/models.json recorded source_commit c559a44 while the generator
computes 5f32859.

build_vended_models._git_commit stamps the commit that last changed
docs/metrics/pareto-frontier-omp-swe3.json. The committed copy was generated on
branch benchmark/minicpm5-2b-omp-swe3, where that commit was c559a44. GitHub
squashed the branch into main as 5f32859 (#182), so the sha the generator reads
changed and the committed file went stale. The failure predates this branch's
results commit -- the same job failed on 8dd4b15.

Regenerating moves only the provenance line: all 19 models keep their scores,
costs and tier breakdowns, which confirms the codex run added under
do-not-include stays invisible to the generator.
@aarora79
aarora79 merged commit 266ea67 into main Sep 11, 2026
5 checks passed
@aarora79 aarora79 mentioned this pull request Sep 11, 2026
aarora79 added a commit that referenced this pull request Sep 29, 2026
Main added codex as a fifth coding agent (#162, #184), which needed reconciling
beyond the textual conflicts.

Textual conflicts, all in .claude/skills/benchmark/SKILL.md:

- The agent input. Kept this branch's table, added a `codex` row, and kept
  main's Responses-API tool-parser warning, which is hard-won and still true.
  Did NOT keep main's claim that pi cannot run --provider bedrock: pi has a
  PI_PROVIDER_BEDROCK constant and two committed scripts that do exactly that,
  which is one of the drift fixes this branch makes.
- The CLI pre-flight. Kept this branch's structure, added `codex exec --json`
  to the list of agent binaries.
- The orchestrator command line. Took both additions: codex in the agent list
  and the --skill flag.

One semantic conflict git could not see. Main's _build_codex_env sets
OPENAI_API_KEY from config.api_key directly, which bypasses this branch's
api_key_env and would send the "local" placeholder to a gateway that needs a
real credential. Routed it through resolved_api_key(), so all six call sites
that send the key now honour the environment variable.

Also updated the onboard-benchmark skill, which said codex was only the judge:
the harness-selection table now has a Responses-API row for it, and both that
step and the judge step note that running --agent codex puts one tool on both
sides of the run, so the judge model must differ from the model under test.
aarora79 added a commit that referenced this pull request Oct 13, 2026
build_vended_models.py stamped models.json with the commit that last
touched the omp frontier JSON and that commit's date. A squash merge
replaces the branch commit, so the stamp made on a PR no longer matched
main and test_committed_models_json_matches_the_generator failed with
every score still correct. It happened after #184, #210 and #219.

Both fields now come from committed data, which a merge does not
rewrite:

- source_commit becomes source_sha256: the first 12 hex digits of the
  frontier JSON's SHA-256, the value `sha256sum <file> | cut -c1-12`
  prints. It changes only when the data does.
- measured_on is the latest run_date among the run summaries of the
  models in the frontier, rather than a commit date in the merger's
  timezone. For the current data that is 2026-10-11, Sonnet 5.5's run.

schema_version goes to 1.1 for the renamed field. route.py and the
skill check only the major version, so both keep working. The generator
no longer runs git, which removes its two subprocess calls.

Tests: the git-based provenance test is replaced by checks that the hash
matches the file bytes, changes with the data and ignores where the file
lives, and that measured_on takes the latest frontier model's run date
and is null when no summary records one. A simulated squash merge onto
main, with a different commit, date and timezone, passes --check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

codex harness: agent is unreachable, endpoint runs hit Bedrock, and token totals can drop the cache read

1 participant