e2e: fix the codex model exclusions, and stop truncating away the cause - #272
Merged
Conversation
The codex launch sweep has been red on main for at least five runs, and the
exclusion list had drifted in both directions.
Dropped `gpt-5-6-luna`/`sol`/`terra`. Their comment says the backing OpenAI
endpoint returns ENDPOINT_NOT_FOUND in the CI region; that is no longer true —
all three answer a `/ai-gateway/codex/v1/responses` request there now. So the
list was excluding the models ucode actually defaults to (`gpt-5.6-sol` is the
launcher's pick) while still testing ones that cannot run.
Added `gpt-5-3-codex`, which is what fails today. It is discoverable, in the
workspace catalog, and configured correctly by ucode — the gateway's upstream
OpenAI project just can't serve that snapshot from the CI region ("The requested
model snapshot is not available for your project's geography"). The gateway
relays it as a bare INTERNAL_ERROR, so the launch dies after codex-cli exhausts
its five reconnects. No ucode change can fix it, which is what makes it an
exclusion rather than a bug.
Net effect is more coverage, not less: 13 models tested before, 15 now.
Also switched the failure report from the first 200 characters of stdout/stderr
to the last 1500. Direction matters more than length here: codex-cli logs a
non-fatal model-listing error *first* and the real cause *last*, so a prefix
reliably reported the wrong problem. The geography failure above presented as a
`/v1/models` routing error until the tail was visible, which cost two rounds of
misdiagnosis.
Verified against a workspace in the CI region: each of the three re-enabled
models returns a response, `gpt-5-3-codex` returns INTERNAL_ERROR, and
`tests/test_e2e.py::TestCodexLaunch::test_launch_codex_per_model` passes across
all 15 remaining models.
Note the exclusion list is keyed to whatever workspace `UCODE_TEST_WORKSPACE`
points at, and regional availability has now shifted twice. Treating a
gateway-side availability error as a skip rather than a failure would remove this
maintenance, but that is a larger change.
Co-authored-by: Isaac
AarushiShah-db
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
TestCodexLaunch::test_launch_codex_per_modelhas been red onmainfor at least five consecutive runs. The exclusion list had drifted in both directions.Dropped
gpt-5-6-luna/sol/terra. Their comment says the backing OpenAI endpoint returnsENDPOINT_NOT_FOUNDin the CI region. That is no longer true — all three answer a/ai-gateway/codex/v1/responsesrequest there now:So the list was excluding the model ucode actually defaults to (
gpt-5.6-solis the launcher's pick) while still testing one that cannot run.Added
gpt-5-3-codex, which is what fails today. It is discoverable, in the workspace catalog, and configured correctly by ucode — the gateway's upstream OpenAI project just can't serve that snapshot from the CI region:The gateway relays that as a bare
INTERNAL_ERROR("invalid response from an upstream server"), so the launch dies after codex-cli exhausts its five reconnects. No ucode change can fix it, which is what makes it an exclusion rather than a bug.Net effect is more coverage, not less: 13 models tested before, 15 now.
Stopped truncating away the cause. The failure report used the first 200 characters of stdout/stderr; it now uses the last 1500. Direction matters more than length: codex-cli logs a non-fatal model-listing error first and the real cause last, so a prefix reliably reported the wrong problem. The geography failure above presented as a
/v1/modelsrouting error — the message died mid-word atRequest path '/ai-gat— and cost two rounds of misdiagnosis before the tail was visible.Testing
Verified against a workspace in the CI region:
gpt-5-3-codexreturnsINTERNAL_ERRORtests/test_e2e.py::TestCodexLaunch::test_launch_codex_per_modelpasses across all 15 remaining models (73s)Note for reviewers
The exclusion list is keyed to whatever workspace
UCODE_TEST_WORKSPACEpoints at, and regional availability has now shifted twice. I verified against a workspace in the CI region, but that secret is masked, so I can't confirm it is the same one — if it isn't, the5-6removals could re-introduce failures.Treating a gateway-side availability error as a skip rather than a failure would remove this maintenance burden entirely, since a model ucode configured correctly that the gateway can't serve isn't a ucode regression. That's a larger change than this fix, so it's left as a follow-up.
This pull request and its description were written by Isaac.