Repository navigation
harness: make agent=codex runnable on the endpoint path, and fix its token accounting - #184
Merged
Merged
Conversation
…g the cache The harness implemented a codex agent that no orchestrator would run, and two accounting paths reported wrong numbers for it on the self-hosted path. Fixes issue #183. Routing. codex 0.153.4 ignores OPENAI_BASE_URL and resolves its base URL from the provider its config selects, so an endpoint run reached whatever ~/.codex/config.toml pointed at -- on a judge-configured machine, Amazon Bedrock, which answered "404 The model 'minicpm5-2b' does not exist". _build_codex_cmd now declares its own provider block (base_url, wire_api = responses, env_key) plus model_context_window when the config knows the served window. _codex_base_url rejects a non-http scheme and refuses a URL that embeds credentials, because the provider block travels in argv where ps exposes it; the key stays in OPENAI_API_KEY. Token totals. _codex_result_from_events subtracts the cached and cache-written portions out of codex's input_tokens, so the fields it returns are disjoint. compute_total_tokens_processed then re-tested them for a partition and, at a cache hit rate near 50%, matched its own signature and dropped the entire cache read -- halving the token count and the derived cost per task. Callers now declare the shape through cache_partition_for_agent instead of having it guessed. The normalizer also sums across turn.completed events rather than overwriting, so a build that splits an exec into several turns cannot lose all but the last. Retried requests. codex counts the attempt it accepted, so prefill for requests it abandoned never reaches the metrics. metrics.json now carries prompt_token_reconciliation for an endpoint run: the agent's prompt tokens against vllm:prompt_tokens_total, with the shortfall named. Verified on qwen3.6-35b-fp8 (vLLM 0.29.0, 1x L40S, 262,144-token window): 32 turns, 6/6 artifacts, total_tokens 481,081 = 468,647 prompt + 12,434 output, matching the server's own counters, reconciliation 0 unaccounted. vLLM's Responses API needs a tool parser that accepts flat Responses-shaped tools. Seven of its 51 parsers read tool.function.name and abort tool extraction, which truncates the stream and makes codex retry every request; the benchmark skill and the model guide now name the safe and unsafe parsers.
…ot-include Adds the codex /swe3 run over mcp-gateway-registry-v2 for Qwen3.6-35B-A3B FP8 on a single L40S (g6e.4xlarge), the first run of the agent=codex path this branch fixes. 21 of 21 tasks scored, no failures, 126/126 artifacts, mean 61.45 against the omp run's 59.23. It lands under do-not-include/qwen3.6-35b-fp8/codex so both harnesses for this model sit side by side and neither reaches the charts. The exclusion reasons are unchanged: the model's only throughput sweep is on g6e.4xlarge rather than the canonical p5en arm, and its token counts are confounded with a 262144 context window against the BF16 run's 200000. Verified that plot_cost_quality's shallow scan still finds 19 models for omp and zero scorable codex runs. Priced on the sweep's cheapest blended rate ($0.0545/1M at concurrency 10, $1.298/hr), codex costs $0.2645 per task against omp's $0.5555, because it processes 4.85M tokens per task against 10.19M. The score gap sits inside the noise: paired delta +2.22, sd 8.27, 95% CI -1.32 to +5.75. The README entry now covers both runs, including the token-accounting convention each harness uses, so the input_tokens fields are not compared across them by mistake.
…mmit The Test job failed on test_committed_models_json_matches_the_generator: vend/swe-router/models.json recorded source_commit c559a44 while the generator computes 5f32859. build_vended_models._git_commit stamps the commit that last changed docs/metrics/pareto-frontier-omp-swe3.json. The committed copy was generated on branch benchmark/minicpm5-2b-omp-swe3, where that commit was c559a44. GitHub squashed the branch into main as 5f32859 (#182), so the sha the generator reads changed and the committed file went stale. The failure predates this branch's results commit -- the same job failed on 8dd4b15. Regenerating moves only the provenance line: all 19 models keep their scores, costs and tier breakdowns, which confirms the codex run added under do-not-include stays invisible to the generator.
Merged
aarora79
added a commit
that referenced
this pull request
Sep 29, 2026
Main added codex as a fifth coding agent (#162, #184), which needed reconciling beyond the textual conflicts. Textual conflicts, all in .claude/skills/benchmark/SKILL.md: - The agent input. Kept this branch's table, added a `codex` row, and kept main's Responses-API tool-parser warning, which is hard-won and still true. Did NOT keep main's claim that pi cannot run --provider bedrock: pi has a PI_PROVIDER_BEDROCK constant and two committed scripts that do exactly that, which is one of the drift fixes this branch makes. - The CLI pre-flight. Kept this branch's structure, added `codex exec --json` to the list of agent binaries. - The orchestrator command line. Took both additions: codex in the agent list and the --skill flag. One semantic conflict git could not see. Main's _build_codex_env sets OPENAI_API_KEY from config.api_key directly, which bypasses this branch's api_key_env and would send the "local" placeholder to a gateway that needs a real credential. Routed it through resolved_api_key(), so all six call sites that send the key now honour the environment variable. Also updated the onboard-benchmark skill, which said codex was only the judge: the harness-selection table now has a Responses-API row for it, and both that step and the judge step note that running --agent codex puts one tool on both sides of the run, so the judge model must differ from the model under test.
This was referenced Oct 11, 2026
Merged
aarora79
added a commit
that referenced
this pull request
Oct 13, 2026
build_vended_models.py stamped models.json with the commit that last touched the omp frontier JSON and that commit's date. A squash merge replaces the branch commit, so the stamp made on a PR no longer matched main and test_committed_models_json_matches_the_generator failed with every score still correct. It happened after #184, #210 and #219. Both fields now come from committed data, which a merge does not rewrite: - source_commit becomes source_sha256: the first 12 hex digits of the frontier JSON's SHA-256, the value `sha256sum <file> | cut -c1-12` prints. It changes only when the data does. - measured_on is the latest run_date among the run summaries of the models in the frontier, rather than a commit date in the merger's timezone. For the current data that is 2026-10-11, Sonnet 5.5's run. schema_version goes to 1.1 for the renamed field. route.py and the skill check only the major version, so both keep working. The generator no longer runs git, which removes its two subprocess calls. Tests: the git-based provenance test is replaced by checks that the hash matches the file bytes, changes with the data and ignores where the file lives, and that measured_on takes the latest frontier model's run date and is null when no summary records one. A simulated squash merge onto main, with a different commit, date and timezone, passes --check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #183.
The harness already implemented a codex agent that no orchestrator would run, and two accounting paths reported wrong token counts for it on the self-hosted path. This makes
--agent codexwork end to end and fixes the counting.Routing: an endpoint run reached Amazon Bedrock instead of the endpoint
_build_codex_envexportedOPENAI_BASE_URL, and-c model_provider=...was passed only forprovider=bedrock. codex 0.153.4 ignoresOPENAI_BASE_URLand takes its base URL from the provider its config selects, so an endpoint run went wherever~/.codex/config.tomlpointed. On a machine configured for the judge that is Amazon Bedrock, which answered404 The model 'minicpm5-2b' does not exist. Pointing codex atmodel_provider=openaiinstead reacheswss://api.openai.comand 401s._build_codex_cmdnow declares its own provider block forprovider=endpoint:wire_api = "responses"is not a choice: codex 0.153.4 removed the chat-completions wire and rejectswire_api = "chat". The context-window override silences theModel metadata not found. Defaulting to fallback metadatawarning, which otherwise leaves codex sizing its context from a guess._codex_base_urlvalidates the endpoint before it reaches argv: it requires http or https and refuses a URL that embeds credentials. The provider block is visible inpsand printed by--dry-run, so ahttps://user:pass@hostendpoint would leak that credential to any local user. The API key stays inOPENAI_API_KEY, and only the variable's name travels on the command line.Token totals: the cache read could vanish
_codex_result_from_eventssubtracts codex's cached and cache-written portions out ofinput_tokens, because codex reports the total prompt with those as subsets. The fields it hands on are therefore disjoint.compute_total_tokens_processedthen re-tested them for a partition, and at a cache hit rate near 50% fresh input equals the cache sum, which is exactly its partition signature: it dropped the whole cache read, halved the token count, and halved the cost per task derived from it, while logging that it had prevented a double count.Callers now declare the shape through
cache_partition_for_agent(agent)rather than letting it be guessed. Every recompute site that knows the agent passes it:run-swe-headless.py,summarize_run.py,gen_agent_report.py,plot_cost_quality.py,plot_cost_accuracy_bubble.pyandeval_swe_router.py. Detection stays the default for every other agent, so no existing run changes.The normalizer also sums across
turn.completedevents instead of overwriting. One exec emits one turn today, verified across seven probes, but a build that splits an exec into several turns (resume, compaction) would otherwise keep only the last.Retried requests are now visible
codex counts the attempt it accepted and ignores the ones it abandoned, so a flaky endpoint bills the GPU for prefill that never reaches the metrics. Measured on a retrying path: vLLM counted 26,022 prompt tokens across three requests while codex reported 8,700.
metrics.jsonnow carriesprompt_token_reconciliationfor an endpoint run at concurrency 1 -- the agent's prompt tokens against thevllm:prompt_tokens_totalwindow delta, with any shortfall named and logged.vLLM parser constraint, documented
codex sends flat Responses-shaped tools (
{"type": "function", "name": "exec_command", ...}) withstream: trueon every request. Seven of vLLM 0.29.0's 51 tool parsers read the nested chat-completions shape (tool.function.name) and abort tool extraction:minicpm5xml,dots,hy_v3,hy_v4,rust,step3,step3p5. Non-streaming returns HTTP 500; streaming returns 200, drops the tool call, and ends with noresponse.completed, which codex reads asstream disconnected before completionand retries five times before failing the turn.qwen3_coderandhermesnever touch those objects and work. The benchmark skill and the FP8 model guide now name both sets.The model is not the problem: MiniCPM5-2B returns a clean tool call on
/v1/chat/completions(finish_reason: tool_calls,{"command": ["ls"]}).Verification
A real run on
qwen3.6-35b-fp8(vLLM 0.29.0, 1x L40S, 262,144-token window,--tool-call-parser qwen3_coder):481,081 equals the server's own 468,647 prompt plus 12,434 generation tokens, and
prompt_token_reconciliationreported 0 unaccounted.total_cost_usdstays null on the self-hosted path, as it should.Also run:
uv run python -m unittest discover -s tests(468 tests; the one failure,test_committed_models_json_matches_the_generator, predates this branch -- the committedvend/swe-router/models.jsonpins commitc559a44whilemainis at5f32859),ruff check,ruff format,bandit -r scripts/(clean), andmypy scripts/run-swe-headless.py(14 errors, the same 14 the file has onmain).New tests cover the 50%-hit-rate case that used to drop the cache, the declared-versus-detected verdict, multi-turn usage accumulation, the endpoint provider block, and the endpoint validator (credentials, scheme, missing value), plus the reconciliation helper's healthy, retrying, concurrent and counter-missing paths.