feat(ethereum): build correctness-preserving RPC provider failover state machine - #1029
Open
Meet-hybrid wants to merge 2 commits into
Open
feat(ethereum): build correctness-preserving RPC provider failover state machine#1029Meet-hybrid wants to merge 2 commits into
Meet-hybrid wants to merge 2 commits into
Conversation
…e machine Provider rotation and retries were not a state machine: concurrent callers could rotate inconsistently, timeout timers could outlive requests, and a response from a lagging provider could update shared block state after a newer response. Introduce per-chain provider circuits that make failover deterministic and correctness-preserving: - Generation tokens: every request leases the active provider with a generation snapshot; late responses from superseded generations can never regress the accepted head or accepted evidence. - Cancellation-safe timeouts: the timer is always cleared on settlement and the underlying task receives an AbortSignal, so timeouts neither leak timers nor orphan in-flight work. - Health decay: per-provider state machines track consecutive failures, cooldowns, and idle decay, and expose recovery criteria (failuresToRecover, recoveryEtaMs) and reason codes (transport, timeout, rate_limit, provider_lag, invalid_data, application). - Header monotonicity: accepted block heights only ever advance; a current provider reporting a lower height is flagged as provider lag and penalized. - Request hedging for critical reads: getBlockNumber, latest-block, and bridge reserves race an alternate provider and take the first healthy result. - Concurrency-safe transitions: all state changes are synchronous and converge deterministically (lowest healthy index wins; sticky active while all are down) so concurrent failover decisions always agree. - Error separation: transport, timeout, rate-limit, provider lag, invalid data, and application errors are classified distinctly; application errors never rotate providers. Fault-injection unit tests cover partitions, stale success, flapping providers, and thundering herds, plus timer-leak and monotonicity guarantees. Also fixes two pre-existing lint errors (prefer-const, no-useless-escape) that were failing the CI lint step. Closes StellaBridge#1027
Fixes the CI workflow file (job-level env referenced the runner context, which
GitHub rejects at parse time) and resolves 35 pre-existing TypeScript build
errors in the backend so the CI pipeline can run and pass.
- ci.yml: replace ${{ runner.temp }} with ${{ github.workspace }} in job-level
env blocks (runner is only available at step level).
- config/index.ts: add missing env keys referenced by services/workers
(JWT_*, MAINTENANCE_*, STATUS_PAGE_URL, WORMHOLE_WATCHED_ASSET_*,
INGESTION_*, EXPORT_STREAMING_MAX_ROWS, BRIDGE_MISMATCH_THRESHOLD,
HEALTH_SCORE_THRESHOLD).
- Fix broken named imports in route-groups (sessionRoutes -> sessionsRoutes,
incidentTimeline -> incidentTimelineRoutes, jobsRoutes default import).
- mmrVerification.routes.ts: use nonnegative() (zod v3) instead of nonneg().
- mmrAccumulator.service.ts: widen hash helpers to Uint8Array to satisfy
Buffer<ArrayBuffer> vs Buffer<ArrayBufferLike> typing.
- digestScheduler unit test: type the knex mock raw property.
- Remove unused worker threshold imports.
Contributor
|
In errors.ts, mutating the existing error object via error.providerIndex = providerIndex inside toRpcError introduces a side-effect that could cause unexpected behavior if the same error instance is caught and reused elsewhere. |
Contributor
|
Consider returning a new cloned RpcCallError instance instead of modifying the existing one to maintain strict state immutability and ensure all checks are passing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1027
Summary
Provider rotation and retries were not a state machine: concurrent callers could rotate inconsistently, timeout timers could outlive requests, and a response from a lagging provider could update shared block state after a newer response. This PR replaces the ad-hoc
failover()/getProvider()rotation inEthereumRpcClientwith per-chain provider circuits that make failover deterministic and correctness-preserving.What changed
New module
backend/src/services/ethereum/failover/:errors.ts— error taxonomy that separatestransport,timeout,rate_limit,provider_lag,invalid_data, andapplicationfailures (incl. ethers.js code mapping). Application errors are per-query outcomes and never rotate providers.timeout.ts— cancellation-safe deadline: the timer is always cleared on settlement (no leaks) and the underlying task receives anAbortSignalso work is cancelled where the transport supports it; late-settling tasks cannot produce unhandled rejections.circuit.ts— per-provider health state machine:healthy → degraded → cooling_down/unhealthy, consecutive-failure thresholds, cooldowns, idle health decay, and exported recovery criteria (failuresToRecover,recoveryEtaMs) with reason codes.chainCircuit.ts— the per-chain circuit: generation tokens so late responses from superseded leases can never regress the accepted head/evidence; deterministic failover (lowest healthy index wins; sticky active while all providers are down) so concurrent decisions converge; header monotonicity (heights only advance; a lagging current provider is penalized); and request hedging for critical reads. All transitions are synchronous, so Node's single thread guarantees a consistent snapshot across concurrent callers.Reworked
EthereumRpcClientto drive all calls through the circuit:getBlockNumber,"latest"blocks, andgetBridgeReservesare hedged and feed the monotonicity guard.getBridgeReserves/getTokenInfocaptured the provider before the retry/failover window).rpc_failover_total/rpc_all_providers_down_totalmetrics.getProviderStates(chainId)(recovery criteria + reason codes),getActiveProviderIndex(chainId);getLastKnownBlocknow reads the circuit's monotonic accepted head.Also fixes two pre-existing ESLint errors (
prefer-const,no-useless-escape) that were failing the CI lint step onmain.Acceptance criteria
withTimeoutclears its timer on every settlement path; timer-leak tests with fake timers.ChainCircuitSnapshot.providers[]exposes state, reason,failuresToRecover,recoveryEtaMs,ready.Testing
backend/tests/services/ethereum/failover/(all passing).npm --workspace=backend run lintpasses.npm --workspace=backend run buildadds zero new TypeScript errors (the 35 pre-existing errors onmainare untouched).main(identical pre-existing failures, which are DB-environment dependent).Notes
CI on
mainis currently failing before jobs start (GitHub "workflow file issue") and the backend build already fails on 35 pre-existing type errors unrelated to this change. Neither is introduced by this PR.