Skip to content

Make router health checks bounded and non-blocking#491

Open
Yiqing Wang (YQ-Wang) wants to merge 1 commit into
agent-substrate:mainfrom
YQ-Wang:feat/490-router-health-timeout
Open

Make router health checks bounded and non-blocking#491
Yiqing Wang (YQ-Wang) wants to merge 1 commit into
agent-substrate:mainfrom
YQ-Wang:feat/490-router-health-timeout

Conversation

@YQ-Wang

Copy link
Copy Markdown

Summary

  • make the Kubernetes /version health check observe the router context and a 500 ms dependency timeout
  • perform dependency network calls without holding the health report mutex, allowing /statusz to serve the most recently completed report
  • preserve health timestamps and counters with an atomic report update after each check cycle
  • add regression coverage for timeout, cancellation, report/status responsiveness, and health accounting

Root cause

The client-go Discovery().ServerVersion() helper uses context.TODO() internally. If the Kubernetes API accepted /version but withheld its response, canceling the router context could not terminate the request. The router also held the report mutex across dependency network calls, so the stalled request blocked /statusz and delayed shutdown.

Validation

  • make verify
  • go test ./cmd/atenet/internal/router -run 'Test(CheckK8s|HealthCheck|HealthStart)' -count=20
  • go test -race ./cmd/atenet/internal/router -count=1
  • Kind Kubernetes v1.36.1 fault injection: a proxy fetched the real cluster /version response and withheld it; /statusz remained responsive with HTTP 200, the request was canceled at the 500 ms deadline, and one Kubernetes health failure was recorded

Fixes #490

@YQ-Wang
Yiqing Wang (YQ-Wang) marked this pull request as ready for review July 22, 2026 09:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make router Kubernetes health checks bounded and non-blocking

1 participant