Skip to content

feat(ethereum): build correctness-preserving RPC provider failover state machine - #1029

Open
Meet-hybrid wants to merge 2 commits into
StellaBridge:mainfrom
Meet-hybrid:feat/rpc-provider-failover-state-machine
Open

feat(ethereum): build correctness-preserving RPC provider failover state machine#1029
Meet-hybrid wants to merge 2 commits into
StellaBridge:mainfrom
Meet-hybrid:feat/rpc-provider-failover-state-machine

Conversation

@Meet-hybrid

Copy link
Copy Markdown

Closes #1027

Summary

Provider rotation and retries were not a state machine: concurrent callers could rotate inconsistently, timeout timers could outlive requests, and a response from a lagging provider could update shared block state after a newer response. This PR replaces the ad-hoc failover()/getProvider() rotation in EthereumRpcClient with per-chain provider circuits that make failover deterministic and correctness-preserving.

What changed

New module backend/src/services/ethereum/failover/:

  • errors.ts — error taxonomy that separates transport, timeout, rate_limit, provider_lag, invalid_data, and application failures (incl. ethers.js code mapping). Application errors are per-query outcomes and never rotate providers.
  • timeout.ts — cancellation-safe deadline: the timer is always cleared on settlement (no leaks) and the underlying task receives an AbortSignal so work is cancelled where the transport supports it; late-settling tasks cannot produce unhandled rejections.
  • circuit.ts — per-provider health state machine: healthy → degraded → cooling_down/unhealthy, consecutive-failure thresholds, cooldowns, idle health decay, and exported recovery criteria (failuresToRecover, recoveryEtaMs) with reason codes.
  • chainCircuit.ts — the per-chain circuit: generation tokens so late responses from superseded leases can never regress the accepted head/evidence; deterministic failover (lowest healthy index wins; sticky active while all providers are down) so concurrent decisions converge; header monotonicity (heights only advance; a lagging current provider is penalized); and request hedging for critical reads. All transitions are synchronous, so Node's single thread guarantees a consistent snapshot across concurrent callers.

Reworked EthereumRpcClient to drive all calls through the circuit:

  • getBlockNumber, "latest" blocks, and getBridgeReserves are hedged and feed the monotonicity guard.
  • Contract calls are bound to the selected provider inside the request, so a mid-request failover can never bind a contract to a stale provider (previously getBridgeReserves/getTokenInfo captured the provider before the retry/failover window).
  • Circuit events are surfaced as structured logs plus rpc_failover_total / rpc_all_providers_down_total metrics.
  • New introspection: getProviderStates(chainId) (recovery criteria + reason codes), getActiveProviderIndex(chainId); getLastKnownBlock now reads the circuit's monotonic accepted head.

Also fixes two pre-existing ESLint errors (prefer-const, no-useless-escape) that were failing the CI lint step on main.

Acceptance criteria

  • Late responses cannot regress last-known block or accepted evidence — generation-gated commit path + monotonicity guard, tested.
  • Concurrent failover decisions converge deterministically — lowest-healthy-index rule + sticky active, tested under a 20-caller thundering herd.
  • Timeouts cancel underlying work where supported and do not leak timerswithTimeout clears its timer on every settlement path; timer-leak tests with fake timers.
  • Provider states expose recovery criteria and reason codesChainCircuitSnapshot.providers[] exposes state, reason, failuresToRecover, recoveryEtaMs, ready.
  • Fault injection — unit tests cover partitions, stale success/lag, flapping providers (bounded oscillation), and thundering herds.

Testing

  • New: 50 unit tests in backend/tests/services/ethereum/failover/ (all passing).
  • npm --workspace=backend run lint passes.
  • npm --workspace=backend run build adds zero new TypeScript errors (the 35 pre-existing errors on main are untouched).
  • Full backend unit suite: no regressions vs main (identical pre-existing failures, which are DB-environment dependent).

Notes

CI on main is currently failing before jobs start (GitHub "workflow file issue") and the backend build already fails on 35 pre-existing type errors unrelated to this change. Neither is introduced by this PR.

…e machine

Provider rotation and retries were not a state machine: concurrent callers could
rotate inconsistently, timeout timers could outlive requests, and a response
from a lagging provider could update shared block state after a newer response.

Introduce per-chain provider circuits that make failover deterministic and
correctness-preserving:

- Generation tokens: every request leases the active provider with a generation
  snapshot; late responses from superseded generations can never regress the
  accepted head or accepted evidence.
- Cancellation-safe timeouts: the timer is always cleared on settlement and the
  underlying task receives an AbortSignal, so timeouts neither leak timers nor
  orphan in-flight work.
- Health decay: per-provider state machines track consecutive failures,
  cooldowns, and idle decay, and expose recovery criteria (failuresToRecover,
  recoveryEtaMs) and reason codes (transport, timeout, rate_limit, provider_lag,
  invalid_data, application).
- Header monotonicity: accepted block heights only ever advance; a current
  provider reporting a lower height is flagged as provider lag and penalized.
- Request hedging for critical reads: getBlockNumber, latest-block, and bridge
  reserves race an alternate provider and take the first healthy result.
- Concurrency-safe transitions: all state changes are synchronous and converge
  deterministically (lowest healthy index wins; sticky active while all are down)
  so concurrent failover decisions always agree.
- Error separation: transport, timeout, rate-limit, provider lag, invalid data,
  and application errors are classified distinctly; application errors never
  rotate providers.

Fault-injection unit tests cover partitions, stale success, flapping providers,
and thundering herds, plus timer-leak and monotonicity guarantees.

Also fixes two pre-existing lint errors (prefer-const, no-useless-escape) that
were failing the CI lint step.

Closes StellaBridge#1027
Fixes the CI workflow file (job-level env referenced the runner context, which
GitHub rejects at parse time) and resolves 35 pre-existing TypeScript build
errors in the backend so the CI pipeline can run and pass.

- ci.yml: replace ${{ runner.temp }} with ${{ github.workspace }} in job-level
  env blocks (runner is only available at step level).
- config/index.ts: add missing env keys referenced by services/workers
  (JWT_*, MAINTENANCE_*, STATUS_PAGE_URL, WORMHOLE_WATCHED_ASSET_*,
  INGESTION_*, EXPORT_STREAMING_MAX_ROWS, BRIDGE_MISMATCH_THRESHOLD,
  HEALTH_SCORE_THRESHOLD).
- Fix broken named imports in route-groups (sessionRoutes -> sessionsRoutes,
  incidentTimeline -> incidentTimelineRoutes, jobsRoutes default import).
- mmrVerification.routes.ts: use nonnegative() (zod v3) instead of nonneg().
- mmrAccumulator.service.ts: widen hash helpers to Uint8Array to satisfy
  Buffer<ArrayBuffer> vs Buffer<ArrayBufferLike> typing.
- digestScheduler unit test: type the knex mock raw property.
- Remove unused worker threshold imports.
@Mosas2000

Copy link
Copy Markdown
Contributor

In errors.ts, mutating the existing error object via error.providerIndex = providerIndex inside toRpcError introduces a side-effect that could cause unexpected behavior if the same error instance is caught and reused elsewhere.

@Mosas2000

Copy link
Copy Markdown
Contributor

Consider returning a new cloned RpcCallError instance instead of modifying the existing one to maintain strict state immutability and ensure all checks are passing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Build Correctness-Preserving RPC Provider Failover State Machine

2 participants