fix(proxy): fail over dead account proxy routes

[luawl]

Summary

Fix account-bound proxy connection failures that occurred before upstream dispatch so movable Responses requests can fail over transparently instead of repeatedly selecting the dead route. The change keeps ambiguous POST failures and hard account ownership fail-closed.

Type of change

• fix: - bug fix (no behavior change beyond the bug)
• feat: - new user-facing feature or capability
• refactor: - internal refactor (no behavior change, no API change)
• docs: - documentation only
• chore: / ci: / build: - tooling, CI, packaging
• test: - test-only change
• Breaking change

Linked issue: Fixes #1314

OpenSpec

• This PR includes / updates an OpenSpec change
• Not applicable - bug fix that matches the existing spec
• Not applicable - docs / CI / chore only
• This PR touches a codex-faithful path and preserves upstream-equivalent behavior

Change directory: openspec/changes/retry-account-proxy-connect-failures/

Changes

• Add sanitized unknown / not_dispatched provenance to routed transport failures. Only explicit HTTP or SOCKS proxy-connect failures authorize a non-idempotent same-pool fallback.
• Retry a movable request on another eligible account across raw HTTP/SSE, native Responses WebSocket, and HTTP bridge startup after every same-pool endpoint fails pre-dispatch.
• Release response-create and stream leases before applying the existing bounded transient backoff floor (30 seconds, capped at 300 seconds), while retaining one request-scoped API-key reservation.
• Preserve the original sanitized 502 when no replacement exists. Previous-response, turn-state, uploaded-file, required-account, and single-account contracts remain fail-closed.
• Surface ambiguous POST dispatch outcomes without replaying them.
• Keep the HTTP bridge architecture ratchet green by extracting the small failover state helper instead of increasing the 2,400-line mixin limit.

Diff composition

The headline diff is test/spec heavy:

• production: 314 additions / 45 deletions across 11 files (269 net lines)
• regression tests: 1,055 additions / 4 deletions across 9 files
• OpenSpec: 146 additions

Test plan

pytest tests/unit/test_proxy_utils.py tests/unit/test_codex_client.py tests/unit/test_codex_upstream_paths.py tests/unit/test_load_balancer_concurrency.py tests/unit/test_proxy_websocket_client.py ...
778 passed, 4 deselected

pytest tests/unit/test_proxy_http_bridge.py -q
269 passed

pytest tests/integration/test_proxy_responses.py tests/integration/test_proxy_websocket_responses.py tests/integration/test_http_responses_bridge.py -q
213 passed

focused proxy-connect / dispatch-provenance matrix
18 passed

ruff check + ruff format --check
passed

ty check
passed

python scripts/check_proxy_architecture.py
passed

openspec validate retry-account-proxy-connect-failures --strict
passed

The four deselected tests are unrelated Windows environment-proxy precedence cases; the new core/service regressions and all affected endpoint suites ran.

Checklist

• Title is in Conventional Commits format (<type>(<scope>)?: <subject>).
• Linked the related issue / discussion above.
• Added or updated tests covering the change.
• Ran the relevant local CI subsets.
• Strict OpenSpec validation passes and the task checklist is complete.
• CHANGELOG is not edited by hand.

[Soju06]

Triage note: this PR is conflicting with main (proxy failover code moved with #1296/#1324 landing). Please rebase onto current main — happy to re-review/trigger Codex once it's green. If you'd rather not carry it forward, let us know and we'll close it for now.

[mereyabdenbekuly-ctrl]

Independent production corroboration from a codex-lb 1.21.0 deployment:

• We observed account-bound proxy connection/transport failures before any upstream response headers or Responses event were received.
• Without account exclusion/failover, retries could select the same account-bound dead route again, surfacing as repeated reconnects / stream disconnected before completion to clients.
• Our production mitigation only retries when the failure is known pre-dispatch, releases the failed account's leases before the next selection, and keeps previous-response/file/required-account continuity fail-closed.
• That behavior stopped the repeated dead-route loop in the affected traffic.

This matches the safety boundary and intent of this PR. I am not opening a competing failover PR; sharing the deployment evidence here for the rebase/review. No account identifiers, proxy addresses, or credentials are included.

[Soju06]

We still want this fix for #1314 — excluding dead account-bound proxy routes from retry selection is the right behavior, and the production corroboration above supports it. The branch is CONFLICTING against current main (head f740562, the failover path moved with #1296/#1324). Please rebase onto current main; merge gates (CI rollup, review threads, mergeable state) will run right after.

[Soju06]

There's been no activity here since the PR was opened on 07-14, and both rebase requests (07-16 and 07-24) have gone unanswered, while the underlying bug (#1314) remains one we want fixed — the independent production corroboration above reinforces that. The conflict surface has also grown: #1492 (Live Voice sideband) substantially reworked app/core/clients/proxy_websocket.py, proxy.py, and codex.py, exactly where this PR plumbs the connect-failure provenance, so a mechanical rebase is no longer realistic.

Plan: we'll adopt the design from this PR — sanitized unknown / not_dispatched provenance, proxy-connect-only authorization for non-idempotent same-pool fallback, and lease release before the bounded backoff floor — into a maintainer-owned branch rebuilt on current main, keeping your commits and attribution wherever they survive the rework. If you'd rather carry it forward yourself, reply within the next couple of days and it's yours.

[Soju06]

Maintainer takeover is now live as #1542 per the earlier notice — your commit is preserved there with original authorship, rebuilt against current main's dispatch-provenance layer (which absorbed part of this PR's design since it was opened). This PR will be closed when #1542 merges; if you're back and want to pick it up, comment there.