HTTP-bridge reattach stalls with no response.created and needs a client restart to recover

[lkraider]

Version: ghcr.io/soju06/codex-lb:1.22.0
Client: pi coding agent 0.82.1, transport @earendil-works/pi-ai 0.82.1, model gpt-5.6-sol, provider codex-lb, WebSocket endpoint /backend-api/codex/responses.

Summary

When a session is resumed on the HTTP-bridge path, codex-lb reattaches by injecting a stored anchor and full-resending the conversation. If upstream never emits response.created, codex-lb waits its client-safe deadline (_HTTP_BRIDGE_EVENTLESS_RESPONSE_CREATED_MAX_SECONDS = 240), then ends the turn with "Upstream did not acknowledge response.create before the client-safe deadline". Retrying within the same client session rebuilds the identical reattach to the same account and stalls the same way, so the client cannot recover by retrying. Only starting a fresh client session recovers.

Evidence (one session, two consecutive attempts)

event=fresh_reattach_anchor_injected  bridge_key=sha256:7dfa… account_id=…7cb39f74
event=durable_full_resend_anchor_injected  response_id=resp_0563… stored_items=425
event=create  request_stage=follow_up  durable_session_id=40b06ad6…
(~240s later) WARNING event=missing_response_created_timeout  pending=1
event=close ; event=retire_stale_pending

• Both attempts used the identical bridge_key, account …7cb39f74, durable_session_id, anchor resp_0563…, and 425 resent items. Each stalled ~241s. reallocate_sticky=False.
• The client recorded each as a terminal error and did not auto-recover.
• A fresh client session started under the same conditions came in as kind=websocket request_stage=first_turn with no injected anchor and proceeded normally. So a recoverable path exists; the running session could not reach it.
• This deadline/stall path is HTTP-bridge only; the WebSocket-upstream path has no equivalent deadline.

Steps to reproduce

1. Run a long multi-turn session (codex session affinity on).
2. Let the upstream bridge go idle long enough to be retired (about 4 minutes).
3. Send the next turn. codex-lb does a fresh reattach with a full resend.
4. Upstream does not emit response.created; ~240s later the turn fails with the deadline error.
5. Retry in the same session: it reproduces identically. Restart the client (new session): it recovers.

Expected behavior

Repeated missing_response_created_timeout for a session should be recoverable without a full client restart. Under the same conditions a fresh session with no injected anchor succeeds, so the recoverable path already exists; a session that hits this should be able to reach it on its own.

Notes / scope

• The trigger (upstream not emitting response.created) appears to be backend-side and I can't confirm its cause, so I'm not attributing that to codex-lb. Bounding the wait at 240s is good; this report is only about the behavior after the timeout, where the identical reattach is replayed on retry.
• Related to WebSocket path renames upstream previous_response_not_found, and unlike the HTTP path does not keep the error client-recoverable #1529: same underlying loss of store:false connection continuity, different transport (HTTP-bridge vs WebSocket) and different symptom (stall vs immediate error). Both point at the same need: let the client recover without manual intervention.

[Soju06]

Confirmed against main (b91d9bb): the 240s client-safe deadline is at _HTTP_BRIDGE_EVENTLESS_RESPONSE_CREATED_MAX_SECONDS (http_bridge/helpers.py:191), and nothing marks the reattach state as poisoned after the timeout fires — the next turn rebuilds the identical anchor injection to the same account, so the session can never reach the fresh path your logs show working. Two tracks here:

1. One producer of the forced anchored resend (prefix-scanner false negative on a historical developer message) is being fixed in fix(proxy): preserve terminal delivery across detach #1527/fix(proxy): preserve developer-interleaved fresh resends #1537. If your stored prefix contains a custom_tool_call -> developer message -> custom_tool_call_output shape, those PRs likely remove your trigger.
2. Independently of any specific producer, we agree the post-timeout behavior is a defect: after missing_response_created_timeout the durable anchor for that bridge_key should be invalidated so the retry takes the fresh no-anchor path (optionally with sticky reallocation), instead of replaying a reattach that has already proven dead for 240s. That is a bounded, fail-closed recovery consistent with the existing fresh-session path.

If you can share whether your stored 425 items contain interleaved developer messages inside tool-call loops, that would tell us whether #1537 covers your trigger or whether this is a distinct producer.

[lkraider]

Thanks, Track 2 (invalidate the durable anchor after missing_response_created_timeout so the retry takes the fresh no-anchor path) is exactly the fix that matches what we saw.

On your question about the stored items: our context contains neither half of the custom_tool_call -> developer message -> custom_tool_call_output shape, so #1527/#1537 do not cover this trigger.

Verified from the client sessions (pi 0.82.1, gpt-5.6-sol):

• No custom_tool_call items at all. Every tool call serializes as function_call (fc_ item ids). In the failing session that's 442/442; across all recent sessions it is ~5470 fc_ and zero ctc_. Tools were bash, read, replace, write, and the project's own extension tools, none using grammar input.
• No developer messages interleaved in tool-call loops. The only developer-role item on the wire is the leading system prompt (reasoning models send instructions as developer), at input position 0, before any tool calls.

Anonymized, representative view of the flow. The stored prefix items are all well-formed; the failure is at the reattach step, not at any item:

# --- session running fine on a warm upstream bridge (delta turns) ---
developer                       # instructions, position 0 only
message role=user
reasoning
function_call         call_id=1
function_call         call_id=2
function_call_output  call_id=1
function_call_output  call_id=2
reasoning
function_call         call_id=3
function_call_output  call_id=3
... (turns continue, ~425 items accumulate) ...

# --- ~4 min idle (long tool call / think time) ---
# codex-lb idle-retires the upstream bridge:
#   event=close ; event=retire_stale_pending

# --- next turn: codex-lb reattaches ---
#   event=fresh_reattach_anchor_injected       (previous_response_id = resp_…)
#   event=durable_full_resend_anchor_injected   (re-sends all ~425 items above)
#   event=create  ->  send response.create
#   >>> STALL: upstream never emits response.created
#   >>> +240s: missing_response_created_timeout -> terminal error to client
#   >>> retry in same session rebuilds the identical reattach -> same stall
#   >>> only a fresh client session (no injected anchor) recovers

For contrast, the shape #1537 targets, which does not appear anywhere in our data (we emit no custom_tool_call, and no developer message mid-loop):

custom_tool_call         call_id=X
developer                        # interleaved between a call and its output
custom_tool_call_output  call_id=X

So this is a distinct producer: the forced durable_full_resend_anchor_injected was triggered by the idle-retire + reattach, not by a prefix-scanner false-negative on a developer message. Which points back at Track 2 as the fix that helps here.