Skip to content

Add endpoint health management: circuit breaker and background probing #49

Description

@N-thnI

Description

Extend RPC endpoint health handling with a proper breaker and proactive recovery.

Problem Statement

RpcClient currently quarantines an endpoint for a fixed duration after N transport failures — deliberately simple, but it neither backs off progressively for an endpoint failing repeatedly, nor probes carefully before restoring full traffic. Recovery is also only discovered when a user request happens to retry a quarantined endpoint, so the first request after an outage pays the latency cost. Both concerns touch the same health state, which is why they belong in one change.

Proposed Changes

  • src/resilience/circuit-breaker.ts with closed/open/half-open states; in half-open allow a single probe before fully restoring; expose state transitions for logging
  • src/resilience/health-scheduler.ts probing quarantined endpoints on a configurable interval, with start()/stop() where stop() clears every timer — no dangling handles keeping a process alive
  • Both opt-in: RpcClient must behave exactly as it does today when neither is supplied
  • Tests for both

Technical Implementation Scaffolding

  • Target Repository: vero-sdk
  • Target Path: src/resilience/
  • Branch Naming: feat/issue--endpoint-health
  • Authority Context: Reliability

Acceptance Criteria

  • A failing half-open probe reopens the breaker without restoring full traffic
  • A recovered endpoint returns to rotation without a user request triggering it
  • stop() leaves no active timers
  • Existing RpcClient behaviour is unchanged without opt-in
  • npm test, npm run typecheck, npm run lint, and npm run build all pass

Definition of Done

  • Reviewed by lead maintainer
  • Pull request merged via verified status check

This issue is self-contained. Everything it needs already exists on main; it does not wait on any other issue. Deliver the module and its tests in one PR.

Metadata

Metadata

Assignees

Labels

GrantFox OSSIssue tracked in GrantFox OSSMaybe RewardedIssue may be eligible for a GrantFox rewardThird CampaignCampaign: Third Campaign

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions