Pre-flight checklist
Problem / motivation
codex-lb’s Responses proxy is a distributed asynchronous state machine spread across account selection, affinity, retry/failover, HTTP/SSE, WebSocket and HTTP-bridge transports, account leases, API-key usage reservations, request logging, and cancellation cleanup.
Correctness often depends on a particular ordering of events: whether output has become visible, whether an account is hard-pinned or excluded, whether an upstream EOF is ambiguous, whether a reservation has settled, whether a stream or response-create lease is still held, and whether cancellation races with background cleanup.
The existing handwritten tests cover many individual cases, but each test fixes one schedule and usually one transport. Recent fixes show recurring failures in this same state space: leaked bridge or WebSocket leases (#1476, #1282), reservation-settlement ordering (#1332), terminal no-replay boundaries (#1389), shared-session concurrency (#1432), and silent/clean-close bridge recovery (#1394). The cross-product is too large to enumerate by hand, so fixing one schedule or transport can leave an equivalent path untested.
Dan Luu’s testing discussion recommends continuously generating new randomized tests, retaining every discovered failure as a permanent regression, and requiring an executable reproducer and oracle rather than relying on an LLM’s claim that a bug exists.
codex-lb is unusually suitable for this approach because the relevant safety properties are strong and testable even though the event sequences that exercise them are combinatorial.
Proposed change
Add a developer- and CI-only “Responses lifecycle trace fuzzer”, initially scoped to streamed /v1/responses behavior across the HTTP/SSE, WebSocket, and HTTP-to-WebSocket bridge execution paths.
The harness should have four parts:
-
A deliberately small reference model that records request attempts, selected account ownership, downstream visibility, API-key reservation state, account leases, bridge/gate ownership, and terminal outcome. It should model safety properties, not duplicate the production routing algorithm.
-
A semantic trace generator using Hypothesis. Generated actions should include starting one or more requests, selecting hard or soft affinity, delivering upstream protocol events, rate-limit/server/network errors, silent or clean EOF, advancing a fake clock, changing account availability, cancelling a downstream request, and injecting failures at persistence/settlement boundaries. Interleavings must be controlled with explicit event gates and a fake clock rather than real sleeps.
-
Adapters that execute the generated trace through the real ProxyService, LoadBalancer, and API-key reservation code for each applicable transport. Network, clock, and persistence boundaries may be scripted, but the decision and cleanup logic under test must remain production code.
-
A normalized transcript and invariant checker. At minimum, it should assert:
- no account switch or replay after visible output or hard ownership;
- an excluded account is not selected again and an upstream penalty is attributed only to the selected account;
- every API-key reservation reaches one terminal state and every account lease is eventually released, without negative counters or cap overflow;
- required settlement happens before account-health mutation;
- cancellation, EOF, timeout, and retry leave no orphan task, gate waiter, pending request, or reusable session with a leaked lease;
- one request cannot receive another request’s anonymous upstream event;
- the client receives at most one normalized terminal outcome;
- transport paths produce equivalent normalized ownership, settlement, and cleanup outcomes wherever their public contracts are equivalent.
Hypothesis should shrink failures to a minimal action sequence. CI must print a reproduction blob/seed and upload the minimized JSON trace and normalized transcript. A confirmed failure must be committed as a permanent @example or corpus fixture. When a bug exposed a missing generator dimension or invariant, the generator/property catalog must be extended as part of the fix instead of adding only another isolated handwritten test.
generated semantic trace
│
├── HTTP/SSE adapter ───────┐
├── WebSocket adapter ──────┼── normalized transcript
└── HTTP-bridge adapter ────┘ │
▼
invariants + differential checks
Add these developer targets:
make test-proxy-fuzz: run the permanent corpus plus a bounded generated campaign. Default PR-CI budget: approximately 60 seconds.make fuzz-proxy: run the same properties continuously with an increased Hypothesis budget. HypoFuzz may be evaluated as an optional scheduled runner. Default scheduled campaign: approximately 30 minutes, with failures retained as workflow artifacts.
The initial fast lane should use the existing in-process test application and SQLite fixtures. A scheduled PostgreSQL lane can exercise reservation and persistence behavior. Before making the generated lane a required gate, validate that the harness detects at least three historical defects when their fixes are locally removed or mutation-tested.
This introduces no product API, CLI, dashboard behavior, CODEX_LB_* setting, or runtime dependency. The only defaults are test-run budgets; fuzzing is not enabled in deployed codex-lb processes.
Alternatives considered
Only open-source tools that can run locally and in ordinary project CI without a commercial service are in scope.
The smallest implementation would use Hypothesis alone. Hypothesis already provides semantic data generation, rule-based state machines, shrinking, reproducible failure examples, and a persistent example database, so it is sufficient for the initial beach-head.
This is a viable simplification and should be the first implementation step. Plain Hypothesis does not adapt a long CPU budget across many properties using coverage feedback, which is the additional role proposed for HypoFuzz. HypoFuzz should therefore be an optional scheduled-runner experiment, not a prerequisite for landing the harness or running its required regression lane.
Jepsen is designed for black-box verification of distributed-system histories under faults. A future Jepsen suite could run concurrent clients against several codex-lb replicas while a nemesis partitions replicas from PostgreSQL or the upstream, kills instances, and disturbs clocks. It would be particularly appropriate for durable OAuth-flow ownership, sticky-session fencing, leader election, cache-invalidation convergence, and atomic API-key reservation settlement.
It was not selected for the initial target because many current proxy failures are intra-process lifecycle violations—leaked asyncio tasks or leases, incorrect replay boundaries, event misdelivery, and cancellation cleanup. Those states are difficult to observe through a Jepsen client history. Jepsen also does not provide Python coverage guidance or shrink a semantic proxy trace to the same degree as Hypothesis.
Toxiproxy is a lightweight and CI-friendly way to inject connection resets, timeouts, latency, bandwidth limits, packet loss, and partial reads between codex-lb, PostgreSQL, and a mock upstream. It is likely the best fault-injection component for a later integration or nightly lane.
Toxiproxy is not a complete fuzzing solution by itself. It supplies faults but not generated application workloads, a request-lifecycle model, invariant checking, or failure shrinking. It would therefore complement the proposed trace fuzzer rather than replace it.
Chaos Mesh provides Kubernetes-native pod, network, I/O, clock, and resource fault injection. It could extend the existing kind/Helm testing into multi-replica chaos experiments and verify deployment-level recovery.
It was not selected initially because it requires a privileged Kubernetes environment and produces relatively slow, coarse-grained experiments. Like Toxiproxy, it does not provide the codex-lb-specific workload generator or correctness oracle needed to determine whether a retry, replay, settlement, or cleanup outcome is safe.
Atheris is a native coverage-guided fuzzer for Python and supports the project’s Python version range. It is well suited to high-throughput isolated targets such as SSE framing, Retry-After parsing, Responses replay-safety payloads, WebSocket event decoding, and request decompression.
It was not selected for the main lifecycle harness because byte mutation is a poor representation for long sequences of valid protocol events, concurrent requests, persistence transitions, and cancellations. Reaching deep proxy states would require a substantial structure-aware mutator that would effectively reimplement a semantic trace generator. Atheris remains a good candidate for several smaller follow-up fuzz targets.
Schemathesis can derive property-based and stateful API tests from codex-lb’s OpenAPI schema. It would be valuable for finding validation bypasses, malformed error envelopes, undocumented responses, and CRUD workflow failures.
It was not selected for this beach-head because the highest-value failures occur after a valid /v1/responses request has been accepted and depend on upstream streaming events, transport ownership, failover, and cancellation. Those interactions are not represented by the OpenAPI document.
Rust-based concurrency or fuzzing frameworks
Rust tools such as Shuttle and Loom provide powerful deterministic or randomized schedule exploration, but only for Rust code using their instrumented synchronization primitives. An external Rust driver could generate requests quickly, but it could neither control Python asyncio scheduling nor receive meaningful coverage guidance from the production proxy implementation.
Using one would become reasonable if a substantial proxy state machine were implemented in Rust. Introducing a second-language model solely for testing would instead risk testing the model more thoroughly than the Python behavior it is intended to verify.
Decision
Use Hypothesis as the required open-source foundation for semantic trace generation, shrinking, and permanent regression examples. Start with plain Hypothesis; evaluate HypoFuzz separately as an optional accelerator for longer scheduled campaigns. Keep the trace format and property checks independent enough that they can later drive Toxiproxy integration tests or a Jepsen system-level suite.
Area
No response
Additional context
This proposal uses “fuzzing” in the broad sense used by the linked article. The specific technique is semantic, stateful, model-based and differential testing, rather than sending arbitrary bytes to an endpoint.
Relevant prior art:
Representative regressions in the proposed initial state space include:
The harness should be judged by bug-finding ability and reproducibility, not only line coverage. Before it becomes a required gate, it should be backtested against at least three historical defects by temporarily removing their fixes or applying equivalent mutations. A successful run should demonstrate that the fuzzer finds the defect and reduces it to a short deterministic trace.
Every confirmed discovery should feed back into the system in three possible ways:
- retain the minimized trace permanently;
- add a missing generated action or interaction when the fuzzer could not previously express the failure;
- add or clarify an OpenSpec invariant when the expected behavior was not already normative.
Possible follow-up work, intentionally outside the initial issue, includes Atheris targets for pure parsers, Schemathesis for the wider HTTP API, and a three-replica PostgreSQL deployment exercised through Toxiproxy, Jepsen, or Chaos Mesh.
Pre-flight checklist
Problem / motivation
codex-lb’s Responses proxy is a distributed asynchronous state machine spread across account selection, affinity, retry/failover, HTTP/SSE, WebSocket and HTTP-bridge transports, account leases, API-key usage reservations, request logging, and cancellation cleanup.
Correctness often depends on a particular ordering of events: whether output has become visible, whether an account is hard-pinned or excluded, whether an upstream EOF is ambiguous, whether a reservation has settled, whether a stream or response-create lease is still held, and whether cancellation races with background cleanup.
The existing handwritten tests cover many individual cases, but each test fixes one schedule and usually one transport. Recent fixes show recurring failures in this same state space: leaked bridge or WebSocket leases (#1476, #1282), reservation-settlement ordering (#1332), terminal no-replay boundaries (#1389), shared-session concurrency (#1432), and silent/clean-close bridge recovery (#1394). The cross-product is too large to enumerate by hand, so fixing one schedule or transport can leave an equivalent path untested.
Dan Luu’s testing discussion recommends continuously generating new randomized tests, retaining every discovered failure as a permanent regression, and requiring an executable reproducer and oracle rather than relying on an LLM’s claim that a bug exists.
codex-lb is unusually suitable for this approach because the relevant safety properties are strong and testable even though the event sequences that exercise them are combinatorial.
Proposed change
Add a developer- and CI-only “Responses lifecycle trace fuzzer”, initially scoped to streamed
/v1/responsesbehavior across the HTTP/SSE, WebSocket, and HTTP-to-WebSocket bridge execution paths.The harness should have four parts:
A deliberately small reference model that records request attempts, selected account ownership, downstream visibility, API-key reservation state, account leases, bridge/gate ownership, and terminal outcome. It should model safety properties, not duplicate the production routing algorithm.
A semantic trace generator using Hypothesis. Generated actions should include starting one or more requests, selecting hard or soft affinity, delivering upstream protocol events, rate-limit/server/network errors, silent or clean EOF, advancing a fake clock, changing account availability, cancelling a downstream request, and injecting failures at persistence/settlement boundaries. Interleavings must be controlled with explicit event gates and a fake clock rather than real sleeps.
Adapters that execute the generated trace through the real
ProxyService,LoadBalancer, and API-key reservation code for each applicable transport. Network, clock, and persistence boundaries may be scripted, but the decision and cleanup logic under test must remain production code.A normalized transcript and invariant checker. At minimum, it should assert:
Hypothesis should shrink failures to a minimal action sequence. CI must print a reproduction blob/seed and upload the minimized JSON trace and normalized transcript. A confirmed failure must be committed as a permanent
@exampleor corpus fixture. When a bug exposed a missing generator dimension or invariant, the generator/property catalog must be extended as part of the fix instead of adding only another isolated handwritten test.Add these developer targets:
make test-proxy-fuzz: run the permanent corpus plus a bounded generated campaign. Default PR-CI budget: approximately 60 seconds.make fuzz-proxy: run the same properties continuously with an increased Hypothesis budget. HypoFuzz may be evaluated as an optional scheduled runner. Default scheduled campaign: approximately 30 minutes, with failures retained as workflow artifacts.The initial fast lane should use the existing in-process test application and SQLite fixtures. A scheduled PostgreSQL lane can exercise reservation and persistence behavior. Before making the generated lane a required gate, validate that the harness detects at least three historical defects when their fixes are locally removed or mutation-tested.
This introduces no product API, CLI, dashboard behavior,
CODEX_LB_*setting, or runtime dependency. The only defaults are test-run budgets; fuzzing is not enabled in deployed codex-lb processes.Alternatives considered
Only open-source tools that can run locally and in ordinary project CI without a commercial service are in scope.
Hypothesis without HypoFuzz
The smallest implementation would use Hypothesis alone. Hypothesis already provides semantic data generation, rule-based state machines, shrinking, reproducible failure examples, and a persistent example database, so it is sufficient for the initial beach-head.
This is a viable simplification and should be the first implementation step. Plain Hypothesis does not adapt a long CPU budget across many properties using coverage feedback, which is the additional role proposed for HypoFuzz. HypoFuzz should therefore be an optional scheduled-runner experiment, not a prerequisite for landing the harness or running its required regression lane.
Jepsen
Jepsen is designed for black-box verification of distributed-system histories under faults. A future Jepsen suite could run concurrent clients against several codex-lb replicas while a nemesis partitions replicas from PostgreSQL or the upstream, kills instances, and disturbs clocks. It would be particularly appropriate for durable OAuth-flow ownership, sticky-session fencing, leader election, cache-invalidation convergence, and atomic API-key reservation settlement.
It was not selected for the initial target because many current proxy failures are intra-process lifecycle violations—leaked asyncio tasks or leases, incorrect replay boundaries, event misdelivery, and cancellation cleanup. Those states are difficult to observe through a Jepsen client history. Jepsen also does not provide Python coverage guidance or shrink a semantic proxy trace to the same degree as Hypothesis.
Toxiproxy
Toxiproxy is a lightweight and CI-friendly way to inject connection resets, timeouts, latency, bandwidth limits, packet loss, and partial reads between codex-lb, PostgreSQL, and a mock upstream. It is likely the best fault-injection component for a later integration or nightly lane.
Toxiproxy is not a complete fuzzing solution by itself. It supplies faults but not generated application workloads, a request-lifecycle model, invariant checking, or failure shrinking. It would therefore complement the proposed trace fuzzer rather than replace it.
Chaos Mesh
Chaos Mesh provides Kubernetes-native pod, network, I/O, clock, and resource fault injection. It could extend the existing kind/Helm testing into multi-replica chaos experiments and verify deployment-level recovery.
It was not selected initially because it requires a privileged Kubernetes environment and produces relatively slow, coarse-grained experiments. Like Toxiproxy, it does not provide the codex-lb-specific workload generator or correctness oracle needed to determine whether a retry, replay, settlement, or cleanup outcome is safe.
Atheris/libFuzzer
Atheris is a native coverage-guided fuzzer for Python and supports the project’s Python version range. It is well suited to high-throughput isolated targets such as SSE framing, Retry-After parsing, Responses replay-safety payloads, WebSocket event decoding, and request decompression.
It was not selected for the main lifecycle harness because byte mutation is a poor representation for long sequences of valid protocol events, concurrent requests, persistence transitions, and cancellations. Reaching deep proxy states would require a substantial structure-aware mutator that would effectively reimplement a semantic trace generator. Atheris remains a good candidate for several smaller follow-up fuzz targets.
Schemathesis
Schemathesis can derive property-based and stateful API tests from codex-lb’s OpenAPI schema. It would be valuable for finding validation bypasses, malformed error envelopes, undocumented responses, and CRUD workflow failures.
It was not selected for this beach-head because the highest-value failures occur after a valid
/v1/responsesrequest has been accepted and depend on upstream streaming events, transport ownership, failover, and cancellation. Those interactions are not represented by the OpenAPI document.Rust-based concurrency or fuzzing frameworks
Rust tools such as Shuttle and Loom provide powerful deterministic or randomized schedule exploration, but only for Rust code using their instrumented synchronization primitives. An external Rust driver could generate requests quickly, but it could neither control Python
asyncioscheduling nor receive meaningful coverage guidance from the production proxy implementation.Using one would become reasonable if a substantial proxy state machine were implemented in Rust. Introducing a second-language model solely for testing would instead risk testing the model more thoroughly than the Python behavior it is intended to verify.
Decision
Use Hypothesis as the required open-source foundation for semantic trace generation, shrinking, and permanent regression examples. Start with plain Hypothesis; evaluate HypoFuzz separately as an optional accelerator for longer scheduled campaigns. Keep the trace format and property checks independent enough that they can later drive Toxiproxy integration tests or a Jepsen system-level suite.
Area
No response
Additional context
This proposal uses “fuzzing” in the broad sense used by the linked article. The specific technique is semantic, stateful, model-based and differential testing, rather than sending arbitrary bytes to an endpoint.
Relevant prior art:
Representative regressions in the proposed initial state space include:
The harness should be judged by bug-finding ability and reproducibility, not only line coverage. Before it becomes a required gate, it should be backtested against at least three historical defects by temporarily removing their fixes or applying equivalent mutations. A successful run should demonstrate that the fuzzer finds the defect and reduces it to a short deterministic trace.
Every confirmed discovery should feed back into the system in three possible ways:
Possible follow-up work, intentionally outside the initial issue, includes Atheris targets for pure parsers, Schemathesis for the wider HTTP API, and a three-replica PostgreSQL deployment exercised through Toxiproxy, Jepsen, or Chaos Mesh.