feat(proxy): congestion-aware per-API-key fair-share stream admission (one bursty key can starve all others)

[Soju06]

Summary

Please add a congestion-aware per-API-key fair-share admission gate for stream capacity, so a single key's concurrency burst cannot exhaust the pool-wide per-account stream slots and starve every other key.

The desired policy, in one sentence: when the pool is loose, any key may use as much concurrency as it wants; when the pool gets tight, the keys consuming the most are throttled first. A static per-key concurrency cap does not satisfy this — it wastes idle capacity in the common case.

Production observation (sanitized)

From a codex-lb 1.23.0-beta.2 deployment (14 accounts, default proxy_account_stream_limit=8, single replica → 104 effective stream slots after the recovery reserve):

• One API key ran an agentic fan-out workload holding ~30-36 concurrent streams (measured 91 live upstream connections pool-wide at peak, ~87% of capacity).
• One account additionally hit a genuine upstream 429 and went rate_limited, shrinking the pool to 13 accounts.
• From then on, Account cap exhausted during sticky selection lease_kind=stream reason=account_stream_cap candidates=13 logged continuously — all remaining accounts at cap — and selection degraded to no_accounts.
• Result: sustained 503 no_accounts bursts (20-60% of requests per minute) affecting every key, including interactive keys holding 0-2 streams. Client retries amplified the load.

The per-account stream cap did its job (protecting accounts from upstream 429s), but there is no fairness layer above it: slot allocation across API keys is first-come-first-served, so one bursty key starves the rest.

Requested behavior

Max-min fair share on stream-lease admission, active only under congestion:

• Track in-flight stream leases per API key (in-memory, same replica-statistical scope as the existing per-account caps).
• When pool utilization (in-flight / pool stream capacity over the selection's candidate accounts) is below a configurable threshold: admit everyone unconditionally (work-conserving, zero behavior change).
• At or above the threshold: admit a key only if key_inflight + 1 <= max(min_guarantee, pool_capacity // active_keys). Keys under their fair share always admit (a small min-guarantee makes light/interactive keys starvation-proof); keys over their share cannot take newly freed slots until they drop back under.
• Denial should reuse the existing local-capacity semantics (waiting_for_account_capacity park-and-retry, then 429 rate_limit_error with Retry-After on budget exhaustion) rather than 503 — related: bug(proxy): return 429 usage_limit_reached when the eligible account pool is exhausted #1246.
• Default off (threshold unset/0), one dashboard/env setting.

Related

• feat(admission): isolate media-bearing Responses traffic from text stream capacity #1414 (media admission bulkhead — sibling admission-control concern: slot occupancy, not request rate)
• bug(proxy): return 429 usage_limit_reached when the eligible account pool is exhausted #1246 (429 semantics when the eligible pool is exhausted)
• HTTP-bridge stream lease held for whole session lifetime exhausts per-account stream cap (header-less stalls) #1354 (bridge stream leases held for whole session lifetime — an amplifier of the same exhaustion)
• The archived 2026-06-02-stabilize-responses-concurrency-streaming change explicitly deferred cross-key fairness ("universal multi-replica fairness without affinity requires a later shared-pressure design").

I have an implementation ready to submit as a PR (OpenSpec change + gate in both selection paths + settings + dashboard field + tests).