fix(proxy): fail over dead account proxy routes before upstream dispatch

[Soju06]

Summary

Fix account-bound proxy connection failures that occur before upstream dispatch so movable Responses requests fail over transparently instead of repeatedly selecting the dead route. Ambiguous POST failures and hard account ownership stay fail-closed.

Supersedes #1322 — original implementation by @luawl, taken over per the maintainer notice on that PR. The original commit is preserved with @luawl's authorship, rebuilt onto current main: since #1322 was opened, main independently adopted the sanitized dispatch-provenance layer (retryable_same_contract + failure_phase == "connect" via is_pre_dispatch_connection_failure), so this rework reuses that single source of truth instead of the PR's original dispatch_state enum, and reimplements the failover semantics against the current selection/bridge architecture (#1395/#1476/#1492/#1499-era).

Fixes #1314

Changes

• Add is_confirmed_pre_dispatch_transport_error as the single predicate authorizing cross-account replay: proven pre-dispatch connect provenance only, explicitly excluding host-wide network loss (proxy_network_unavailable, which keeps its account-neutral process recovery path) and TLS verification failures.
• Allow a confirmed pre-dispatch connect failure to try the next endpoint in the same resolved proxy pool even for a non-idempotent POST (no request bytes reached upstream).
• Fail over movable requests to another eligible account across raw HTTP/SSE streaming, native Responses WebSocket connects, and HTTP bridge session startup (_HTTPBridgePreDispatchFailover), bounded by the existing attempt limits, request deadlines, and a monotonic exclusion set.
• Release the failed account's stream lease before recording health, then jump straight to the existing bounded transient backoff floor (record_error_backoff, shared ERROR_BACKOFF_THRESHOLD: 30s floor, 300s cap) so independent requests stop rediscovering the dead route one generic error at a time. No pause/deactivation/rate-limit/quota penalty.
• Preserve the original sanitized 502 when no replacement exists (instead of a generated no_accounts), and fail closed on it for previous-response, turn-state, uploaded-file, and required-account contracts. The request-scoped API-key reservation stays singular across internal failover and settles exactly once.
• Carry pre-dispatch provenance across the routed WebSocket sanitizing conversion in connect_responses_websocket (previously collapsed to the process-network case only).
• Ambiguous dispatch outcomes and idle disconnects never authorize replay or the backoff floor.

Diff composition

• production: 283 additions / 26 deletions across 10 files
• regression tests: 1,063 additions across 9 files
• OpenSpec: openspec/changes/retry-account-proxy-connect-failures/ (174 additions), strict validation passes

Test plan

pytest tests/unit/test_proxy_utils.py tests/unit/test_proxy_http_bridge.py -q                       → 1301 passed
pytest tests/unit/test_codex_client.py tests/unit/test_codex_upstream_paths.py \
  tests/unit/test_load_balancer_concurrency.py tests/unit/test_proxy_websocket_client.py -q         → 189 passed
pytest tests/integration/test_proxy_responses.py tests/integration/test_proxy_websocket_responses.py \
  tests/integration/test_http_responses_bridge.py -q                                                → 250 passed
pytest tests/integration/test_token_refresh_claims.py -q                                            → 42 passed
ruff check / ruff format --check / ty check / scripts/check_proxy_architecture.py                   → pass
openspec validate retry-account-proxy-connect-failures --strict                                     → valid

The new streaming (3) and bridge (3) regressions were verified to fail against unpatched main (dead route surfaced terminally / bridge startup raised to the client) and pass with this change.

🤖 Generated with Claude Code

[Soju06]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 44bc1cc738

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mereyabdenbekuly-ctrl]

I picked up the remaining review fixes. Follow-up PR #1546 targets this PR's feature branch and addresses both unresolved P1s:

• settlement/release is confirmed before keyed dead-route health writes across raw SSE, native WebSocket, and HTTP bridge
• post-refresh confirmed proxy failures now respect hard ownership boundaries and preserve the replacement attempt's terminal outcome
• HTTP-bridge cleanup is reservation-lifecycle-owned, including submit ownership transfer and cross-generation/concurrent-drain races

Commit: 5615948e
Validation: 5096 unit tests passed; 1510 affected unit/integration tests passed; Ruff, ty, and all 48 OpenSpec specs passed.

Please merge #1546 into fix/dead-account-proxy-route-failover; after that these review threads should be ready to resolve.

Update: GitHub CI is now fully green (CI Required and every applicable job passed). #1546 is MERGEABLE/CLEAN with no review findings or unresolved threads, so it is ready to merge into this feature branch.

[Komzpa]

@codex review

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[mereyabdenbekuly-ctrl]

?

[mereyabdenbekuly-ctrl]

I rebased the full three-commit PR series onto current upstream/main (c539a200) and resolved the single additive conflict by preserving the new capability-routing helpers while retaining the websocket proxy failover regression test.

Because my account does not have write permission to Soju06/codex-lb, GitHub rejected a direct force-with-lease update of this PR head with HTTP 403. The tested rebased series is available at:

• fork branch: mereyabdenbekuly-ctrl:codex/pr-1542-review-fixes
• HEAD: c067e603
• commits: 0e52433f, a1226d0c, c067e603

Maintainer handoff:

git fetch https://github.com/mereyabdenbekuly-ctrl/codex-lb.git codex/pr-1542-review-fixes
git switch fix/dead-account-proxy-route-failover
git reset --hard FETCH_HEAD
git push --force-with-lease origin fix/dead-account-proxy-route-failover

Validation on the rebased HEAD: 1,795 affected unit/integration tests passed; Ruff check/format, proxy architecture checks, ty, all 49 OpenSpec specs, and strict validation of retry-account-proxy-connect-failures passed. PR #1546 has been closed as superseded because its original fix was already cherry-picked here as 5a410f5f.

[mereyabdenbekuly-ctrl]

Both remaining P1 review threads are already marked outdated and are addressed by the settlement/ownership changes in the rebased c067e603 series. I attempted to resolve them through the GitHub GraphQL API, but GitHub returned FORBIDDEN because this contributor account lacks ResolveReviewThread permission. A maintainer will need to resolve those two threads after updating the PR head.

[Soju06]

Adopted @mereyabdenbekuly-ctrl's fork rebase (c067e603, branch codex/pr-1542-review-fixes) — thank you for carrying the series over the #1562 conflict and posting the handoff commands. Authorship of all three commits is preserved exactly (luawl / Soju06 / SSY), now rebased and force-pushed as 542905c6.

Topped up over today's merges on main:

• fix(http-bridge): stabilize silent and clean-close recovery #1394: union of the http_bridge/streaming.py import hunk; confirmed dead-route exclusion and lease release happen before any circuit-driven reselection.
• fix(proxy): sequence websocket health after settlement #1558: the series' settlement-before-health-write ordering now composes with the merged settlement machinery — took main's refined _settle_stream_api_key_usage/tracking (release_on_failure gating, connection-safety reconnect flags ahead of settlement) and kept the series' deferred-backoff lifecycle on top (drains only after confirmed settlement; no double-release).
• fix(proxy): preserve websocket connect errors #1517: ws connect errors now arrive unmasked and flow into the series' pre-dispatch provenance (proxy_connect_pre_dispatch) — clean interplay.
• docs(openspec): canonical-code signal for Codex WebSocket stale-anchor recovery (#1529) #1550: openspec-only (no runtime change); the failover paths do not touch stale-anchor surfacing.
• fix(compact): elide observed inline images at wire cap #1457: reconciled the renamed compaction integration test alongside the series' new failover test.

The two outdated review threads on streaming/retry.py (deferred health writes until reservation settlement; ownership-gated post-refresh failover) are addressed by the adopted series and re-verified after the rebase.

Tests: test_proxy_http_bridge, test_proxy_utils, test_codex_client, test_proxy_websocket_client, test_codex_upstream_paths, test_load_balancer_concurrency (1602 passed) plus integration test_http_responses_bridge, test_proxy_websocket_responses, test_proxy_responses (282 passed); ruff check/format clean. PR is MERGEABLE again.

[Komzpa]

@codex review

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[jaekwonhong]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4750bd7190

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Allow verified fresh replays past dead owner routes

When the first HTTP/SSE stream attempt for a locally verified full-resend continuation hits a confirmed pre-dispatch proxy connect failure, this check runs while require_preferred_account is still true, so the branch raises and returns the dead owner's sanitized 502 instead of using _move_verified_fresh_replay_from_owner to strip the owner anchor and select a healthy account. The same verified replay is already allowed for refresh/post-refresh failures, and without it a safe full-resend previous_response_id request remains pinned to an account whose proxy never dispatched any bytes; run the verified-owner replay step (with the same file/turn-state/single-account guards used below) before deciding this is hard ownership.

AGENTS.md reference: AGENTS.md:L105-L108

Useful? React with 👍 / 👎.

[Soju06]

Confirmed — the confirmed pre-dispatch branch in app/modules/proxy/_service/streaming/retry.py raises under require_preferred_account without attempting _move_verified_fresh_replay_from_owner, while the post-visible failover path at ~L2023 does. Since a confirmed pre-dispatch connect failure guarantees zero upstream bytes, a locally verified full-resend replay is safe to move off the dead owner. Will run the verified-owner replay step (same file/turn-state/single-account guards) ahead of the hard-ownership raise before merging.