sub2api-recorder

A small Go reverse proxy that sits in front of sub2api. It transparently forwards every request to the upstream sub2api container and records LLM conversation traffic (headers + body, including SSE streams) into a Postgres table traffic_log for later research.

Storage is session-aware and incremental: requests carrying a session id (Codex headers or Claude Code body metadata) are grouped into (session_id, segment_seq) chains where only the new input/messages items of each request are stored (deltas), instead of the full monotonically growing history. This turns per-session storage from O(n²) into O(n).

The recorder does not depend on any sub2api internal package or version. You can docker-compose pull sub2api to get any upstream release; the recorder container stays unchanged.

Architecture

client → [recorder:8080] → [sub2api:8080] → [LLM API]
              │
              └→ Postgres (sub2api_recorder.traffic_log)

Environment variables

Var	Default	Notes
RECORDER_LISTEN_ADDR	:8080	bind
RECORDER_UPSTREAM_URL	http://sub2api:8080	sub2api inside docker network
RECORDER_DB_DSN	(required unless bypassing)	e.g. postgres://user:pass@postgres:5432/sub2api_recorder?sslmode=disable
RECORDER_BYPASS	false	when true, proxy traffic without recording or connecting to Postgres
RECORDER_MAX_BODY_BYTES	16777216	per direction; bytes beyond are dropped from the recording (the upstream still sees the full body)
RECORDER_RESPONSE_HEADER_TIMEOUT_SECONDS	300	upstream response-header timeout for the recorder reverse proxy; keep above slow /responses/compact latency
RECORDER_WORKER_COUNT	8	DB writer goroutines
RECORDER_QUEUE_SIZE	1024	async insert queue capacity
RECORDER_OVERFLOW_POLICY	drop	drop or block
RECORDER_INCLUDE_HEADERS	true	record request/response headers
RECORDER_REDACT_HEADERS	Authorization,Cookie,X-Api-Key,X-Goog-Api-Key	redacted header names (case-insensitive, comma-separated)
RECORDER_SYNC_SAVE_TIMEOUT	5s	Go duration; timeout for the synchronous per-session DB write
RECORDER_LOG_LEVEL	info	zap level
RECORDER_SHUTDOWN_TIMEOUT	60s	shared deadline for the whole graceful-shutdown sequence
RECORDER_ASYNC_SESSION_ENABLED	false	async session dispatcher master switch; keep false until the rollout runbook below opens it
RECORDER_ASYNC_SHARD_COUNT	8	dispatcher shards (1..256); one FIFO consumer per shard
RECORDER_ASYNC_MAX_QUEUED_RECORDS	4096	global record budget across enqueued + in-flight
RECORDER_ASYNC_MAX_QUEUED_BYTES	268435456	global byte budget for owned record payloads; capture buffers and the copy transient are headroom on top
RECORDER_ASYNC_SHARD_SOFT_RECORDS	1024	per-shard soft record cap (compact exempt)
RECORDER_ASYNC_SHARD_SOFT_BYTES	67108864	per-shard soft byte cap (compact exempt)
RECORDER_ASYNC_COMPACT_RESERVE_BYTES	67108864	byte slice only compact records may use — the loss-reduction reserve
RECORDER_ASYNC_SAVE_MAX_ATTEMPTS	3	bounded retries per record
RECORDER_ASYNC_SAVE_RETRY_BUDGET	15s	total retry time budget per record
RECORDER_ASYNC_TAINT_TTL	24h	tainted admission state survives idle eviction this long
RECORDER_ASYNC_TOKEN_ALLOWLIST	(empty)	CSV of bearer_token_hash; non-empty = only these tokens use the async path (canary stage)
RECORDER_ASYNC_PERCENT	100	when the allowlist is empty: fnv32a(session_key) % 100 < percent routes async; stable per session

What gets recorded

• Only conversation endpoints: POST to /responses, /responses/compact, /messages, /chat/completions (with or without the /v1 prefix). All other paths (health, admin UI, model listings, scanner probes) are proxied but never stored.
• Only 2xx responses. Failed requests are proxied to the client but not stored.
• Requests with a session id are written synchronously by default (the response completes only after the row is durable). With RECORDER_ASYNC_SESSION_ENABLED=true they go through the sharded session dispatcher instead: same state machine, same per-session FIFO order (finalize/enqueue completion order), but the HTTP handler no longer waits for the DB. Sessionless requests always use the async worker queue as full snapshots.
• Session ids: Codex from X-Client-Request-Id / X-Codex-Window-Id / X-Codex-Turn-Metadata headers; Claude Code from metadata.user_id.session_id in the request body. Chains are additionally namespaced by bearer_token_hash, so an unrelated client reusing a session id can never splice into another user's chain.
• Only the exact /<endpoint> and /v1/<endpoint> path shapes are recorded; alias prefixes some gateways accept (e.g. /backend-api/codex/responses) are proxied but not stored. The real deployment's clients only use the standard shapes.
• Known corner: if a compact request's own body is unusable (truncated or non-array input) AND the recorder restarts before the next request, the post-compact segment opens with reason prefix_mismatch instead of compact. Data is intact; only the reason label degrades.
• Segments roll over (segment_seq += 1) on /responses/compact, on input prefix mismatch (rollback / client rewrite), or on context change (model / instructions / tools). Every segment starts with a full snapshot (is_snapshot = true); subsequent rows store only the new items, verified by SHA-256 hashes over the raw request bytes (prefix_hash, full_input_hash, context_hash).
• Truncated or unparsable bodies are stored as standalone snapshots with request_seq NULL and do not disturb the chain. Ignore segment_seq on these off-chain rows — it is best-effort (0 when the recorder had no state for the session yet); group them by time instead.
• Responses are always stored in full; only request input is deduplicated.
• On delta rows request_body holds only the new items, while request_body_size keeps the original wire size of the full request (same convention as truncated rows, where the stored body is also smaller than the wire size).
• Transfer-integrity columns (migration 003, all nullable, NULL = unknown; rows written before the migration are never backfilled): upstream_complete (clean EOF from upstream, or headers-only by protocol), proxy_observed_complete (upstream_complete AND no downstream write error AND client did not cancel — it does not prove the client's TCP stack got every byte), body_expected (false for HEAD/204/304/Content-Length: 0), client_canceled (request context canceled; never inferred from write errors), upstream_read_error / downstream_write_error / request_read_error (first error text per leg), and request_read_complete (request body read to clean EOF). When the client cancels or the downstream write fails, the proxy stops reading upstream, so upstream_complete stays NULL — filter upstream-health metrics on client_canceled = false AND downstream_write_error IS NULL. Capture-limit truncation is a separate concept and stays in request_body_truncated / response_body_truncated.

Endpoints

• /healthz — 200 ok, used by docker healthcheck
• /recorder/stats — text counters: async queue (enqueued / saved / dropped / failed), session path (sync_saved / sync_failed / snapshots / deltas — these count rows saved through the session state machine regardless of mode, so they keep growing when the async dispatcher is enabled), identity_uncertain, and — when the async dispatcher is enabled — the async_* family (accepted/rejected/saved/tainted/queued bytes/per-shard depths); returns bypass true when RECORDER_BYPASS=true
• everything else — proxied to upstream

Bypass mode

Set RECORDER_BYPASS=true to keep the recorder reverse proxy online while disabling all request/response capture and database writes. In bypass mode the process does not connect to Postgres, does not create capture contexts, and does not tee request or response bodies.

For the compose deployment:

# in deploy/.env
RECORDER_BYPASS=true

docker compose -f docker-compose.local.yml up -d --build recorder

Set RECORDER_BYPASS=false and restart the recorder to resume recording.

Database

The recorder runs versioned migrations on startup (tracked in a schema_migrations table; existing deployments are baselined automatically), so the only prerequisite is that the database itself exists. For new compose stacks, deploy/postgres-init/02-recorder.sql creates it on first boot of the postgres volume. For existing deployments, run once:

-- as the postgres superuser:
CREATE DATABASE sub2api_recorder;

Upgrade flow

• upgrade sub2api: docker-compose pull sub2api && docker-compose up -d sub2api — recorder is not affected
• upgrade recorder: docker-compose build recorder && docker-compose up -d recorder

Sample queries

-- last 20 requests with model & latency
SELECT request_started_at,
       bearer_token_last4 AS user_last4,
       inbound_format, model, response_status,
       (extract(epoch from (request_finished_at - request_started_at)) * 1000)::int AS ms
FROM traffic_log
ORDER BY id DESC LIMIT 20;

-- per-user / per-model usage by day
SELECT date_trunc('day', request_started_at) AS day,
       bearer_token_last4 AS user, model, count(*) AS calls
FROM traffic_log
GROUP BY 1, 2, 3
ORDER BY 1 DESC, 4 DESC;

-- show one user's prompts
SELECT request_started_at, model,
       convert_from(request_body, 'UTF8') AS prompt_json
FROM traffic_log
WHERE bearer_token_hash = '<sha256_hex>'
ORDER BY id DESC LIMIT 50;

Local build & test

cd recorder
go mod tidy
go test ./...
go build ./...

Async rollout runbook

The async session path ships dark: RECORDER_ASYNC_SESSION_ENABLED=false. Opening it requires PR1–PR5 merged plus staging soak. Routing is a pure function of process-lifetime config, so one session never runs on both paths.

Stages (plan §10):

1. Canary — RECORDER_ASYNC_SESSION_ENABLED=true, RECORDER_ASYNC_TOKEN_ALLOWLIST=<one bearer_token_hash>. Watch at least one full compact cycle for that user.
2. Percentage — clear the allowlist, set RECORDER_ASYNC_PERCENT to 25, then 50, then 100. The stable session-key hash keeps each session pinned.
3. Watch /recorder/stats: async_rejected_*, async_tainted_sessions, async_compact_rejected (must stay 0 in healthy flow), per-shard depths, and producers_active during deploys.

Canary health SQL — upstream-health metrics must exclude client-side aborts:

SELECT count(*) FILTER (WHERE upstream_complete IS FALSE) AS upstream_incomplete,
       count(*) AS denominator
FROM traffic_log
WHERE client_canceled = false AND downstream_write_error IS NULL
  AND request_started_at > now() - interval '1 hour';

Rollback (restart-based, the signed-off design): lower RECORDER_ASYNC_PERCENT (or flip the flag off) in the environment, then send SIGTERM. The shutdown sequence freezes intake, drains the dispatcher backlog to Postgres under RECORDER_SHUTDOWN_TIMEOUT, and the restarted process routes by the new config. In-flight long streams at the deadline are force-closed — the same cost as any deploy.