bug: remote sessions stay trapped behind 502 until /fork on v1.22.0

[labanmne]

Existing-report check

I searched the existing reports. The underlying failure appears related to the confirmed stale hard-affinity bug in #1505 (Bug 2) and the still-unmerged fix in #1417. This report documents the currently shipped multi-machine behavior and missing practical recovery path on the latest stable release.

codex-lb version

1.22.0, confirmed from the live server's x-app-version response header.

Deployment method

Central codex-lb instance exposed to multiple machines on the same LAN at http://192.168.1.223:2455.

Client used against codex-lb

Codex CLI from multiple machines.

ChatGPT account plan(s) involved

Multi-account pool.

Model(s) involved

Codex Responses traffic through /backend-api/codex/responses.

What happened?

An existing Codex conversation on another machine becomes permanently unusable and repeatedly returns:

unexpected status 502 Bad Gateway: No available accounts,
url: http://192.168.1.223:2455/backend-api/codex/responses,
request id: 01efef95-804e-4d03-ae4a-a38120c3b6d5

Retries do not recover the conversation. The codex-lb process itself remains reachable and healthy:

GET /health -> 200 {"status":"ok"}
x-app-version: 1.22.0

Running /fork is the only practical client-side workaround. The fork succeeds because it creates a fresh session, while the original conversation remains trapped behind its stale/unavailable hard-affinity owner. That also demonstrates that "No available accounts" is misleading: capacity exists for a fresh session, but the existing session cannot use it.

This is especially bad with one central codex-lb instance serving several machines. A user on a remote client cannot reasonably be expected to inspect or purge server-side sticky-session state every time an owner account becomes unavailable.

What did you expect to happen?

Before downstream-visible output, codex-lb should recover safely when a hard-affinity owner is durably unavailable:

• re-resolve the session onto another eligible account when continuity can safely be abandoned; or
• expose an explicit, session-scoped recovery operation that the client can invoke without requiring a new conversation; and
• return a precise continuity/affinity error instead of the false generic claim that there are no available accounts.

I am not asking codex-lb to cross accounts where upstream-owned state makes that unsafe. I am asking for a supported recovery path that is not "throw away or fork the conversation."

Steps to reproduce

1. Run a central codex-lb 1.22.0 instance with a multi-account pool.
2. Point Codex CLI installations on more than one machine at that LAN endpoint.
3. Continue an existing conversation whose hard-affinity owner has become unavailable, exhausted, paused, deleted, or otherwise excluded from selection.
4. Observe persistent 502 Bad Gateway: No available accounts across retries.
5. Run /fork from the failing conversation.
6. Observe that the fork works while the original conversation remains broken.

Relevant client error

unexpected status 502 Bad Gateway: No available accounts, url: http://192.168.1.223:2455/backend-api/codex/responses, request id: 01efef95-804e-4d03-ae4a-a38120c3b6d5

Server logs spanning this request can be supplied if needed.

Additional context

Bluntly: this project has become regression whack-a-mole. Every new release seems to break some new core workflow. A load balancer that strands existing conversations on other machines while a /fork immediately works is failing at its basic job. /fork is a conversation-management command, not an availability or repair mechanism.

#1417 is still open, currently conflicting, and its fix has not shipped in either stable 1.22.0 or 1.23.0-beta.2. Please prioritize shipping a stable recovery path rather than continuing to treat manual session purges or new conversations as acceptable operational workarounds.

[Soju06]

Triage notes, checked against v1.22.0 and current main:

• The misleading No available accounts is reproducible from the hard-affinity branch itself. When the owner row exists but is ineligible, the owner-scoped select fails with the core selector's generic message and app/modules/proxy/_load_balancer/sticky_selection.py:437-440 keeps it (result.error_message or "Hard affinity owner account is unavailable"); the generic string originates at app/core/balancer/logic.py:562. The specific owner-unavailable message only surfaces when the owner is missing from selection states entirely (sticky_selection.py:407-408). So the precise continuity error you're asking for is a small fix separable from any recovery-path decision.
• This is not the bug: dead account-bound proxy does not fail over before visible output #1314 family: bug: dead account-bound proxy does not fail over before visible output #1314 / fix(proxy): fail over dead account proxy routes before upstream dispatch #1542 cover pre-dispatch transport failures on account-bound proxy routes, and fix(proxy): fail over dead account proxy routes before upstream dispatch #1542 deliberately stays fail-closed for hard ownership / previous-response contracts — it will not unstick these sessions. feat(proxy): add durable capability lineage for reconnect #1562 (reconnect capability lineage) doesn't touch owner availability either.
• The durable-owner trap itself is what fix(proxy): purge stale hard codex_session mappings pinned to a durably unavailable owner #1417 bounds (leader-elected background purge of stale hard codex_session mappings whose owner stayed non-active past a 6h threshold, deletion-only, never rebinding). It currently reports mergeable again.
• One v1.22.0-specific way an owner looks durably unavailable while the rest of the pool has capacity — leaked bridge stream leases (HTTP-bridge stream lease held for whole session lifetime exhausts per-account stream cap (header-less stalls) #1354) — was root-fixed in fix(proxy): release idle bridge sessions' account stream leases #1476, merged 07-30 but not in any shipped release yet. Worth re-testing once a release containing it lands.

[Soju06]

Triage update: v1.23.0 (released today) contains two fixes that should change the behavior you saw on 1.22.0 — the affinity purge path (#1417) and the lease fix (#1476). Could you retest a pinned conversation against an unavailable owner on v1.23.0?

If recovery now works, we'll re-scope this issue to the one remaining piece: the misleading 502 No available accounts message on the hard-affinity continuity path (sticky_selection.py), which should say the session is pinned to an unavailable account and suggest /fork — a small separable fix.