fix(proxy): never inject a cross-account previous_response_id anchor - #1274
fix(proxy): never inject a cross-account previous_response_id anchor#1274aididhaiqal wants to merge 3 commits into
Conversation
|
@codex review |
1 similar comment
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Codex review at the current head is clean, but after syncing with
|
5f2bee5 to
5f85df3
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Re-triaged against current main: still very much wanted. Main has no equivalent guard — |
|
To use Codex here, create an environment for this repo. |
|
Ping @aididhaiqal — the 07-27 rebase ask still stands, and the conflict remains confined to |
The HTTP-bridge compact-anchor optimization injects `previous_response_id = session.last_completed_response_id` and trims the stored history prefix. A `previous_response_id` is account-scoped upstream: only the account that created a response can resume it. The injection had no ownership check, so when a Codex session failed over to a different account (durable owner unavailable, or a durable record restored onto a session bound to another account) the anchor pointed at a response the serving account never created. Upstream accepted the `response.create` but never emitted `response.created`, and with the history trimmed away there was no fallback — the per-bridge `response_create_gate` (`Semaphore(1)`) wedged. The holder's client saw "idle timeout waiting for SSE"; queued requests saw "temporarily overloaded". Track the owning account of `last_completed_response_id` (`last_completed_response_account_id`), set in lockstep at both setters — the real `response.completed` path records the session's current account, the durable restore records the durable owner account — and gate the session-level anchor injection on owner == serving account. On mismatch, fall through to a full-history resend (correct output, more tokens), never a cross-account anchor. Regression tests: anchor injected when same-account; anchor skipped + full history resent after cross-account failover. OpenSpec: fix-cross-account-previous-response-anchor (sticky-session-operations).
5f85df3 to
9b93a15
Compare
|
Hey, sorry wasnt around. Should be clear now @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9b93a15b62
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @pytest.mark.asyncio | ||
| async def test_stream_via_http_bridge_skips_session_anchor_after_cross_account_failover( |
There was a problem hiding this comment.
Exercise the failover and response-create gate in the regression test
This regression test does not reproduce the production failure it claims to cover: _run_session_anchor_owner_stream returns a session that is already bound to acc-2, mocks _get_or_create_http_bridge_session and _submit_http_bridge_request, and preloads the event queue with None. Consequently, it never performs an account failover, sends a frame upstream, waits for response.created, or acquires/releases the response-create gate, so it would still pass if the real bridge path remained wedged. Add bridge-level coverage that completes an anchor on one account, reconnects to another, and verifies the full-history request completes without blocking the gate.
AGENTS.md reference: AGENTS.md:L125-L128
Useful? React with 👍 / 👎.
|
Rebase landed clean — CI fully green on 9b93a15. Two items left:
Fix-class, so still soak-exempt; with those two done this is ready to land. |
|
Rebased onto current main and finished the two open review items in #1638, with your three commits carried over verbatim (authorship intact) — the only conflict was mechanical, in mixin.py against main's newer try/abort structure. While rebasing I found and closed a second unguarded post-binding injection site (the owner-forward recovery rebind), and replaced the stubbed test with a real-bridge integration test that reproduces the wedge when the guard is removed. Your diagnosis was exactly right — this is the root cause of the wedge family we spent today mitigating from the consumer side. Happy to fold #1638 back into this branch instead if you prefer landing it under your own PR. |
Summary
The HTTP-bridge "compact anchor" continuity optimization injects
previous_response_id = session.last_completed_response_idand trims the already-stored history prefix so a follow-up turn only carries the new items. Aprevious_response_idis account-scoped upstream — only the account that created a response can resume it — but the injection had no ownership check.When a Codex session fails over to a different account (the durable owner account became unavailable, or a durable record is restored onto a session bound to another account), the anchor points at a response the serving account never created. Upstream accepts the WebSocket
response.createbut never emitsresponse.created, and because the history was trimmed away there is no fallback. The per-bridgeresponse_create_gate(Semaphore(1)) stays held:stream disconnected before completion: idle timeout waiting for SSE;codex-lb is temporarily overloaded during http_bridge_response_create_gate.Observed live: sessions fanned across several accounts repeatedly wedged on the same anchor even though the accounts had quota — freeing the gate doesn't help because the session re-injects the same cross-account anchor and re-wedges.
Fix: track the account that owns
last_completed_response_id(last_completed_response_account_id), set in lockstep at both setters — the realresponse.completedpath records the session's current account, the durable-restore path records the durable owner account — and gate the session-level anchor injection on owner == serving account. On mismatch, fall through to a full-history resend (correct output, slightly more tokens), never a cross-accountprevious_response_id.This is codex-faithful: the real Codex CLI never carries a
previous_response_idacross a connection/account boundary, so declining to inject a foreign anchor matches upstream behavior more closely, not less.OpenSpec
openspec/changes/fix-cross-account-previous-response-anchor/—sticky-session-operationsdelta: compactprevious_response_idanchors are account-scoped; codex-lb MUST NOT inject an anchor whose owning account differs from the serving account, and MUST resend the full history instead.openspec validate --specsgreen.Tests
test_stream_via_http_bridge_injects_session_anchor_when_account_owns_it— anchor still injected when the serving account owns it (no regression to the optimization).test_stream_via_http_bridge_skips_session_anchor_after_cross_account_failover— anchor skipped and full history resent on cross-account failover; no wedge.tests/unit/test_proxy_http_bridge.py(267) +tests/integration/test_http_responses_bridge.py(89) green.