bug: client-fault upstream 400 penalizes the serving account and 502s unrelated hard-sticky sessions (plus an unclassified 'tool search call' missing-tool-output variant)

[yeongjun-cigro]

Summary

A client-fault upstream 400 is charged against the upstream account that happened to serve it. Because that penalty feeds error_count -> exponential error backoff -> in_error_backoff, and because a hard Codex-session sticky request is narrowed to exactly one candidate account where the backoff rescue can never fire, one tenant's poisoned conversation degrades shared account health and 502s unrelated tenants.

The trigger we observed is the missing-tool-output 400, and it comes in a fourth wording that _is_missing_tool_output_error does not know about:

{"type":"error","status":400,
 "error":{"type":"invalid_request_error",
          "message":"No tool output found for tool search call call_XXXX",
          "param":"input"}}

Two separable defects, described below. Verified against origin/main @ b91d9bb8.

---

Defect A — the tool search call wording is unclassified

_MISSING_TOOL_OUTPUT_MESSAGE_PREFIXES (app/modules/proxy/service.py:2105) enumerates exactly three wordings:

_MISSING_TOOL_OUTPUT_MESSAGE_PREFIXES = (
    "no tool output found for function call call_",
    "no tool output found for custom tool call call_",
    "no tool output found for apply patch call call_",
)

so _is_missing_tool_output_error (service.py:2112) returns False for the tool-search wording. Consequences at each consumer of that classifier:

• app/modules/proxy/_service/http_bridge/upstream_events.py:670 and app/modules/proxy/_service/websocket/mixin.py:605 use is_missing_tool_output_event as prefer_previous_response_not_found when matching an anonymous upstream error event (no response.id) to a pending request. On a multiplexed bridge/WS session the unclassified 400 falls through to the generic matcher (_match_websocket_request_state_for_anonymous_event, websocket/helpers.py:1414), so it can be attributed to a pending request that did not cause it.
• app/modules/proxy/_service/websocket/helpers.py:1041 and :1303, and app/modules/proxy/_service/streaming/helpers.py:511 never rewrite it, so the raw upstream message and the raw call_XXXX leak to the client instead of the stream_incomplete continuity failure the other three wordings get, and _record_continuity_fail_closed(reason="missing_tool_output") is never recorded — the condition is invisible in continuity observability.

This is the same enumeration gap as #1168, which fixed the custom_tool_call / apply_patch_call wordings. #1168 named three function_call-only sites; on current main sites 1 and 2 have been generalised through the _PENDING_TOOL_CALL_OUTPUT_ITEM_TYPE_BY_CALL_TYPE table (app/modules/proxy/_service/support.py:60), so only site 3, the classifier, is still hard-coded to a fixed wording list.

The item type is real, but the synthesiser is not a safe place to extend

tool_search_call and tool_search_output are already modelled in this repo — app/modules/proxy/replay_safety.py:20-22 lists them in _ACCOUNT_NEUTRAL_REPLAY_OMITTED_ITEM_TYPES, and tests/unit/test_replay_safety.py:187-203 fixes their shape:

{"type": "tool_search_call",  "id": ..., "call_id": ..., "arguments": {...}, "execution": "client", "status": "completed"}
{"type": "tool_search_output",           "call_id": ..., "tools": [],       "execution": "client", "status": "completed"}

Note that the output item is tool_search_output (not tool_search_call_output) and that it carries a tools list, not the {"output": "<text>"} field that _synthetic_interrupted_function_call_output (app/modules/proxy/_service/response_create.py:363) emits for every entry in _PENDING_TOOL_CALL_OUTPUT_ITEM_TYPE_BY_CALL_TYPE. So adding "tool_search_call": "tool_search_output" to that table would synthesise an item whose shape we cannot verify against upstream; a wrong shape converts a maskable 400 into a different, unmaskable one. Adding it would also silently pull tool_search_call into _TTFT_OUTPUT_ITEM_TYPES (support.py:67).

Open question for maintainers: is the right long-term remedy for an orphaned tool_search_call (a) synthesising {"type": "tool_search_output", "call_id": ..., "tools": []}, or (b) dropping the orphaned tool_search_call — which replay_safety.py already proves upstream accepts, since the account-neutral replay projection omits both items entirely? Either way it needs upstream-shape evidence this repo does not currently contain, which is why the PR below is classifier-only.

Suggested fix scope (A)

Add the no tool output found for tool search call call_ prefix; keep the hosted web search call wording unclassified (a web_search_call is executed upstream and has no client-addressable output — tests/unit/test_proxy_utils.py already asserts it must stay False). Consider a follow-up that derives the prefixes from the pending-call-type table so the next tool-call variant is one edit, but note that the wordings ("apply patch call" vs apply_patch_call, "tool search call" vs tool_search_call) are upstream prose and do not mechanically follow from item-type names.

---

Defect B — client-fault 400 mutates the serving account's health (higher impact)

Mechanism, step by step

1. app/modules/proxy/_service/streaming/helpers.py:785-816 — _handle_stream_error. This is the only call site of LoadBalancer.record_error in the whole app (rg 'record_error\(' app returns one non-definition hit, helpers.py:809). It classifies, then returns early only when _facade()._is_account_neutral_error_code(code); otherwise a non-rate-limit / non-quota / non-permanent code lands in:

   await proxy._load_balancer.record_error(account)
   _facade().logger.info("Recorded transient account error account_id=%s request_id=%s code=%s", ...)

   Every surface funnels here: _handle_proxy_error (service.py:2038), the streaming retry loop (streaming/retry.py), the websocket mixin, and the compact path (_service/compact.py:1419, :1463).

2. app/modules/proxy/_service/support.py:1307 — _is_account_neutral_error_code.

   def _is_account_neutral_error_code(code: str | None) -> bool:
       return is_local_overload_error_code(code) or code in {
           PROCESS_NETWORK_UNAVAILABLE_CODE,
           "proxy_unavailable",
           "responses_compact_input_too_large",
       }

   invalid_request_error is not in that set, and classify_upstream_failure (app/modules/proxy/helpers.py:66) classifies it non_retryable — which failover_decision (app/core/balancer/logic.py:1190) turns into "surface". So there is no failover amplification; the only effect of the client's malformed payload is that it increments error_count on an innocent account, once per retry.

3. app/core/balancer/logic.py:484-497. error_count >= 3 puts the account into min(300, 30 * 2 ** (error_count - 3)) seconds of backoff and appends it to in_error_backoff instead of available.

4. app/core/balancer/logic.py:519. The rescue that pulls an account back out of backoff fires only when

   allow_backoff_fallback and (len(in_error_backoff) > 1 or (in_error_backoff and hard_blocked_exists))

   and hard_blocked_exists (:507) explicitly excludes accounts already in in_error_backoff.

5. app/modules/proxy/_load_balancer/sticky_selection.py:356-361. For a hard Codex-session sticky request with a resolved owner mapping, the candidate set is narrowed to that single account:

   if hard_sticky:
       selection_states = [state for state in states if state.account_id == sticky_existing_account_id]

   With one candidate, in_error_backoff can only ever have length 1 and hard_blocked_exists is False by construction, so step 4's rescue is unreachable. _select_account_preferring_budget_safe returns no account, sticky_selection.py:436-441 sets selection_error_code = "hard_affinity_saturated", and load_balancer.py:709/721 turns that into an immediate 502 — no failover, no wait, for a session that had nothing to do with the offending payload.

Field evidence (aggregate, 24h, codex-lb 1.21, shared multi-tenant deployment)

Signal	Count
HTTP 400, all attributable to one repeated tool-call id from one client	25,747
Recorded transient account error code=invalid_request_error	25,747 (1:1 with the 400s)
HTTP 502	2,192
... of which hard_affinity_saturated	1,583 (72%)
... of those, belonging to a single uninvolved tenant	~95%
... of those 1,583 present in request_logs	11

One five-minute window served zero successful requests. The 502s are near-invisible on the dashboard because selection fails before an account is assigned, so almost none of them reach request_logs — the failure is only legible in container logs.

The offending client's conversation was permanently poisoned: every retry re-sent the identical orphaned tool call, so this is a sustained, not bursty, load on account health.

Why not simply neutralise invalid_request_error

Some 400s carrying invalid_request_error are genuinely account-scoped and must keep backing their account off:

• the model-entitlement rejection from bug: codex-lb lists gpt-5.3-codex as available even though ChatGPT-backed Codex accounts cannot use it #876, The '<model>' model is not supported when using Codex with a ChatGPT account. — already matched narrowly by _is_account_model_unsupported_error (app/modules/proxy/helpers.py:45);
• account-scoped hosted state referenced from input (file_id, item_reference, and the rest of _ACCOUNT_SCOPED_HOSTED_INPUT_TYPES, replay_safety.py:114), which is why gating on param == "input" alone does not prove account independence either.

The invariant worth writing into the spec is narrower and provable from the payload alone:

An error that would reproduce identically on every account must never mutate one account's health.

Suggested fix scope (B)

In _handle_stream_error, skip all account-health mutation for an upstream rejection that satisfies that invariant, decided by the classified message (never by the code alone) plus http_status == 400 when a status is known. Log the skip so the decision stays observable. Leave classify_upstream_failure, failover_decision, and the client-visible status/body untouched.

Candidates for that set, in increasing order of debatability:

1. missing tool output (all wordings) — unambiguous: the input array contradicts itself, no account state is involved;
2. previous_response_not_found (_is_previous_response_not_found_error, app/core/errors.py:88) — arguably a proxy/client anchor-bookkeeping failure rather than account fault, but a stale anchor is account-scoped state, so this one is a real judgement call and is left out of the PR;
3. the whole param == "input" class — rejected above.

Secondary, clearly separate proposal

Independently of B, consider letting hard-sticky selection rescue a single-element in_error_backoff. A hard-pinned session has no alternative account by construction, so len(in_error_backoff) > 1 or hard_blocked_exists is structurally unreachable for it; a request that can only ever go to one account arguably should be allowed to try that account rather than 502 while it waits out an advisory, replica-local backoff window (openspec/specs/account-routing/spec.md, "Transient balancer health signals are replica-local"). Not bundled with B — it changes routing policy, whereas B only stops a bogus health signal from being produced in the first place.

---

PRs

• Defect A: classifier-only, fix(proxy): classify tool-search missing-tool-output rejections
• Defect B: fix(proxy): keep upstream request-shape rejections account neutral

Both branch from origin/main @ b91d9bb8 and merge cleanly with each other.

Refs

• bug: interrupted custom_tool_call outputs are not synthesized, causing upstream 400 'No tool output found for custom tool call' #1168 — the same classifier-enumeration gap for the custom_tool_call / apply_patch_call wordings; its sites 1 and 2 are now table-driven, site 3 is not.
• bug: durable session state diverges from disconnected clients (Request blocked; gate starvation; dangling custom tool call 400s) #1167 — dangling custom-tool-call 400s from durable session state diverging from a disconnected client; a different producer of the same class of orphaned tool call.
• ■ unexpected status 502 Bad Gateway: No available accounts, url: http://127.0.0.1:2455/backend-api/codex/responses #1505 — the same error_count -> cooldown -> backoff -> No available accounts cascade, reached from a stale plan_type instead of from a client payload. Same downstream mechanism, different upstream cause.
• bug: codex-lb lists gpt-5.3-codex as available even though ChatGPT-backed Codex accounts cannot use it #876 — the account-scoped invalid_request_error that B must not neutralise.

[yeongjun-cigro]

PRs, both branched from main @ b91d9bb8 and mergeable with each other independently:

• Defect A (classifier-only): fix(proxy): classify tool-search missing-tool-output rejections #1531
• Defect B (account-health neutrality): fix(proxy): keep upstream request-shape rejections account neutral #1532

B does not depend on A: its regression tests use the already-classified custom tool call wording, so it lands and is testable on its own. A is what extends B's coverage to the tool search call wording actually observed in the field.

Open questions left for maintainers, restated:

1. the correct remedy for an orphaned tool_search_call — synthesise {"type": "tool_search_output", "call_id": ..., "tools": []}, or drop the orphaned call (which replay_safety.py's account-neutral projection already proves upstream accepts). Needs upstream-shape evidence, so neither PR guesses;
2. whether previous_response_not_found belongs in B's account-neutral set;
3. whether hard-sticky selection should be allowed to rescue a single-element in_error_backoff.

[Soju06]

Triage: both defects verified against main @ b91d9bb — the prefix tuple at service.py:2105 and the neutral-code set at support.py:1307 are exactly as described, and the hard-sticky rescue unreachability argument checks out (single-candidate selection_states can never satisfy len(in_error_backoff) > 1 or hard_blocked_exists). Accepting the invariant as stated: an error that reproduces identically on every account must never mutate one account's health. Review will proceed on #1531 and #1532 as soak-exempt bug fixes.

On the three open questions: (1) orphaned tool_search_call remedy — agreed that neither synthesis shape nor drop can be chosen without upstream-shape evidence; classifier-only scope for A is right, please keep the follow-up out of #1531. (2) previous_response_not_found stays OUT of the account-neutral set — a stale anchor is account-scoped state, your own analysis is correct. (3) rescuing a single-element in_error_backoff for hard-sticky is a routing-policy change that interacts with #1417's continuity_abandoned_at purge design; we will decide it there, not bundle it into #1532.