Expected Behavior
Router dependency health checks should be bounded by a timeout and stop when the router context is canceled.
The /statusz endpoint should remain responsive and return the most recently completed health report, even while a dependency check is slow or unavailable.
Actual Behavior
The Kubernetes health check calls Discovery().ServerVersion(), which uses context.TODO() internally and therefore does not observe the router context or its cancellation.
The router also holds the health report mutex while performing dependency network calls. If the Kubernetes API accepts the request but never returns a response:
- the health-check goroutine remains blocked;
/statusz blocks while waiting for the report mutex;- subsequent health checks cannot run;
- router shutdown can remain blocked waiting for the health goroutine.
This was reproduced with an actual client-go ServerVersion request against an HTTP server that accepted /version but withheld its response. Canceling the health context did not terminate the request, and Report() remained blocked until the server response was released.
Steps to Reproduce the Problem
- Start a router health check using a Kubernetes client whose
/version endpoint accepts the connection but does not return a response.
- Wait until the health check enters
Discovery().ServerVersion(), then cancel or expire the router health context.
- Call
Report() or request /statusz and observe that it remains blocked until the Kubernetes /version request eventually returns.
Specifications
- Version:
main at 45f20f6571f550e814f8df2e86990908921facef
- Platform: Kind Kubernetes v1.36.1 (
kindest/node:v1.36.1) on Colima/macOS arm64
- Fault injection: Proxy obtained the real Kind
/version response and withheld it from the router client
Expected Behavior
Router dependency health checks should be bounded by a timeout and stop when the router context is canceled.
The
/statuszendpoint should remain responsive and return the most recently completed health report, even while a dependency check is slow or unavailable.Actual Behavior
The Kubernetes health check calls
Discovery().ServerVersion(), which usescontext.TODO()internally and therefore does not observe the router context or its cancellation.The router also holds the health report mutex while performing dependency network calls. If the Kubernetes API accepts the request but never returns a response:
/statuszblocks while waiting for the report mutex;This was reproduced with an actual client-go
ServerVersionrequest against an HTTP server that accepted/versionbut withheld its response. Canceling the health context did not terminate the request, andReport()remained blocked until the server response was released.Steps to Reproduce the Problem
/versionendpoint accepts the connection but does not return a response.Discovery().ServerVersion(), then cancel or expire the router health context.Report()or request/statuszand observe that it remains blocked until the Kubernetes/versionrequest eventually returns.Specifications
mainat45f20f6571f550e814f8df2e86990908921facefkindest/node:v1.36.1) on Colima/macOS arm64/versionresponse and withheld it from the router client