Skip to content

e2e: fix the codex model exclusions, and stop truncating away the cause - #272

Merged
tt-le merged 1 commit into
mainfrom
tien/codex-e2e-model-exclusions
Aug 5, 2026
Merged

e2e: fix the codex model exclusions, and stop truncating away the cause#272
tt-le merged 1 commit into
mainfrom
tien/codex-e2e-model-exclusions

Conversation

@tt-le

@tt-le tt-le commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Changes

TestCodexLaunch::test_launch_codex_per_model has been red on main for at least five consecutive runs. The exclusion list had drifted in both directions.

Dropped gpt-5-6-luna / sol / terra. Their comment says the backing OpenAI endpoint returns ENDPOINT_NOT_FOUND in the CI region. That is no longer true — all three answer a /ai-gateway/codex/v1/responses request there now:

databricks-gpt-5-6-luna    OK -> gpt-5.6-luna
databricks-gpt-5-6-sol     OK -> gpt-5.6-sol
databricks-gpt-5-6-terra   OK -> gpt-5.6-terra
databricks-gpt-5-3-codex   INTERNAL_ERROR

So the list was excluding the model ucode actually defaults to (gpt-5.6-sol is the launcher's pick) while still testing one that cannot run.

Added gpt-5-3-codex, which is what fails today. It is discoverable, in the workspace catalog, and configured correctly by ucode — the gateway's upstream OpenAI project just can't serve that snapshot from the CI region:

ERROR: stream disconnected before completion: The requested model snapshot is not
available for your project's geography.

The gateway relays that as a bare INTERNAL_ERROR ("invalid response from an upstream server"), so the launch dies after codex-cli exhausts its five reconnects. No ucode change can fix it, which is what makes it an exclusion rather than a bug.

Net effect is more coverage, not less: 13 models tested before, 15 now.

Stopped truncating away the cause. The failure report used the first 200 characters of stdout/stderr; it now uses the last 1500. Direction matters more than length: codex-cli logs a non-fatal model-listing error first and the real cause last, so a prefix reliably reported the wrong problem. The geography failure above presented as a /v1/models routing error — the message died mid-word at Request path '/ai-gat — and cost two rounds of misdiagnosis before the tail was visible.

Testing

Verified against a workspace in the CI region:

  • each of the three re-enabled models returns a response
  • gpt-5-3-codex returns INTERNAL_ERROR
  • tests/test_e2e.py::TestCodexLaunch::test_launch_codex_per_model passes across all 15 remaining models (73s)

Note for reviewers

The exclusion list is keyed to whatever workspace UCODE_TEST_WORKSPACE points at, and regional availability has now shifted twice. I verified against a workspace in the CI region, but that secret is masked, so I can't confirm it is the same one — if it isn't, the 5-6 removals could re-introduce failures.

Treating a gateway-side availability error as a skip rather than a failure would remove this maintenance burden entirely, since a model ucode configured correctly that the gateway can't serve isn't a ucode regression. That's a larger change than this fix, so it's left as a follow-up.

This pull request and its description were written by Isaac.

The codex launch sweep has been red on main for at least five runs, and the
exclusion list had drifted in both directions.

Dropped `gpt-5-6-luna`/`sol`/`terra`. Their comment says the backing OpenAI
endpoint returns ENDPOINT_NOT_FOUND in the CI region; that is no longer true —
all three answer a `/ai-gateway/codex/v1/responses` request there now. So the
list was excluding the models ucode actually defaults to (`gpt-5.6-sol` is the
launcher's pick) while still testing ones that cannot run.

Added `gpt-5-3-codex`, which is what fails today. It is discoverable, in the
workspace catalog, and configured correctly by ucode — the gateway's upstream
OpenAI project just can't serve that snapshot from the CI region ("The requested
model snapshot is not available for your project's geography"). The gateway
relays it as a bare INTERNAL_ERROR, so the launch dies after codex-cli exhausts
its five reconnects. No ucode change can fix it, which is what makes it an
exclusion rather than a bug.

Net effect is more coverage, not less: 13 models tested before, 15 now.

Also switched the failure report from the first 200 characters of stdout/stderr
to the last 1500. Direction matters more than length here: codex-cli logs a
non-fatal model-listing error *first* and the real cause *last*, so a prefix
reliably reported the wrong problem. The geography failure above presented as a
`/v1/models` routing error until the tail was visible, which cost two rounds of
misdiagnosis.

Verified against a workspace in the CI region: each of the three re-enabled
models returns a response, `gpt-5-3-codex` returns INTERNAL_ERROR, and
`tests/test_e2e.py::TestCodexLaunch::test_launch_codex_per_model` passes across
all 15 remaining models.

Note the exclusion list is keyed to whatever workspace `UCODE_TEST_WORKSPACE`
points at, and regional availability has now shifted twice. Treating a
gateway-side availability error as a skip rather than a failure would remove this
maintenance, but that is a larger change.

Co-authored-by: Isaac
@tt-le
tt-le enabled auto-merge (squash) August 5, 2026 17:28
@tt-le
tt-le merged commit a77ab5d into main Aug 5, 2026
3 of 4 checks passed
@tt-le
tt-le deleted the tien/codex-e2e-model-exclusions branch August 5, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants