fix(indexer): make stream circuit breaker error-type-aware

[olathedev]

Closes #1179

Problem

The live-stream loop in services/indexer/src/stream.ts tracked a single consecutiveFailures counter that incremented on any error and permanently breaks the stream once it hit a hardcoded threshold of 10. That meant:

• A Soroban RPC rolling restart (kubectl rollout restart), which produces ~10s of consecutive ECONNREFUSED, halted the indexer for good.
• A Postgres pool checkout timeout during a DB spike had the same effect.
• A short burst of rapid-fire timeouts (e.g. during a GC pause) could do it too.
• Recovery required an operator to manually restart the process — the stream never came back on its own.

Fix

Introduces an error-type-aware StreamCircuitBreaker (services/indexer/src/stream-circuit.ts):

1. Classification — isRetriableStreamError() treats transport blips as retriable and lets them retry indefinitely without counting toward the trip threshold:

   • ECONNREFUSED, ECONNRESET, ETIMEDOUT, EPIPE, EHOSTUNREACH, ENETUNREACH, ENETDOWN, EAI_AGAIN
   • undici/fetch timeout codes (UND_ERR_CONNECT_TIMEOUT, UND_ERR_HEADERS_TIMEOUT, UND_ERR_BODY_TIMEOUT, UND_ERR_SOCKET)
   • 429 / 5xx RPC responses (a 4xx other than 429 is treated as persistent — a malformed request won't fix itself)
   • pg pool exhaustion, matched by message since it carries no error code (e.g. "timeout exceeded when trying to connect", "too many clients already")
   • Node's fetch wraps transport errors in a TypeError, so the cause chain is unwrapped when classifying.
   • ENOTFOUND is deliberately not retriable (a permanently unresolvable host is a misconfiguration), unlike EAI_AGAIN (a temporary resolver failure).

2. Separate counters — retriable failures are tracked only for observability; only unclassified/persistent failures increment the counter that can trip the breaker.

3. Configurable threshold — STREAM_CIRCUIT_BREAKER_THRESHOLD env var (default 10, unchanged from before).

4. Half-open recovery — tripping the breaker no longer ends the stream. It opens, waits STREAM_CIRCUIT_BREAKER_PROBE_INTERVAL_MS (default 30000), then allows a single probe attempt through. Success closes the breaker and streaming resumes; failure reopens it for another wait. No more manual restarts.

5. Structured metrics — stream_circuit_open, stream_circuit_half_open, stream_circuit_closed, and stream_retriable_error are emitted as JSON log lines on every state transition, ready for alerting.

Changes

• services/indexer/src/stream-circuit.ts (new) — the breaker and error classifier.
• services/indexer/src/stream.ts — replaced the inline counter/break with the new breaker; retriable errors retry-and-continue, persistent errors drive open → half-open → closed.
• services/indexer/src/config.ts — added streamCircuitBreakerThreshold / streamCircuitBreakerProbeIntervalMs, sourced from STREAM_CIRCUIT_BREAKER_THRESHOLD / STREAM_CIRCUIT_BREAKER_PROBE_INTERVAL_MS.
• services/indexer/src/index.ts — wires the new config through to streamEvents.
• services/indexer/.env.example — documents the two new env vars.
• services/indexer/src/__tests__/stream-circuit.test.ts (new) — unit tests for the classifier and breaker state machine.
• services/indexer/src/__tests__/stream.test.ts — integration tests against streamEvents itself, including the exact scenario from the issue.

Acceptance criteria

• Retriable errors reset the failure counter on success and do NOT trip the breaker alone.
• Only persistent unclassified errors count toward the trip threshold.
• Circuit breaker threshold is configurable via env var.
• Half-open probe logic implemented with configurable probe interval.
• Structured metric events emitted on state transitions.
• Unit test: 15 consecutive ECONNREFUSED errors followed by a successful fetch → breaker never trips, stream continues.

Test plan

• npx jest in services/indexer — all 356 tests pass (2 pre-existing skips unrelated to this change), including:
  • stream-circuit.test.ts — classifier coverage for every retriable/persistent code, cause-chain unwrapping, and the breaker's closed → open → half-open → closed transitions.
  • stream.test.ts — new #1179 describe block: 15x ECONNREFUSED followed by success never opens the breaker; a run of unclassified errors opens it at the configured threshold; the breaker recovers through a half-open probe instead of terminating the stream.
• npx tsc --noEmit — no new type errors (the two pre-existing errors in search.ts/utils.ts are unrelated and present on main before this change).
• npx eslint on all changed files — clean.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
linkora-social-web	Skipped		Aug 29, 2026 5:08am

[olathedev]

CI is failing on 4 checks here, but all 4 are pre-existing on `main` at the commit this branch is based on (`eb2405b`) and unrelated to the files touched in this PR:

• JS/TS typecheck / Lint TypeScript Packages — `packages/sdk/src/submit.ts` has a real type error (`RpcServer` not assignable to `RpcClient`), and `services/indexer/src/utils.ts` is a corrupted/binary file that fails ESLint parsing, plus pre-existing `require()`-style import lint errors in `ratelimit.test.ts`. None of these files are touched by this PR.
• Secret Scanning (gitleaks) — fails with "missing gitleaks license" — the org's `GITLEAKS_LICENSE` secret isn't configured. Pure repo-config issue, unrelated to any code.
• Unit Tests — fails on `cargo fmt --check` against unformatted Rust in `packages/contracts`, which this PR doesn't touch.

Confirmed by running the CI workflow against `main`'s own HEAD (`eb2405b`) — same 4 failures there. Happy to open separate PRs for these if useful, but keeping them out of scope here to keep this PR focused on #1179.

Locally, `services/indexer`'s own `tsc --noEmit`, `eslint`, and `jest` (356 passed) are all clean for the files this PR changes.