Significant delay in endpoint IP address propagation from Envoy Gateway to Envoy Proxy during backend rollout

[coro]

Description:

What issue is being seen? Describe what should be happening instead of
the bug, for example: The expected value isn't returned, etc.

We have observed a significant delay in the propagation of backend Pod IP addresses from Envoy Gateway to Envoy Proxy during rolling restarts of said backend service. In some cases, we have seen it takes between 30-120s to propagate the IP addresses of new Pods to Envoy Proxy pods, meaning there is a delay before which Envoy itself starts routing traffic to new Pods.

The timeline looks as follows:

• A rollout restart is triggered of a backend Deployment
• Several new Pods come up (depending on rollout strategy of the backend Deployment) and become Ready
• (Immediately) The EndpointSlice for the Deployment is updated with the appropriate conditions for the new pods (e.g. ready/serving/terminating) and their IP addresses
  • This can be seen in the Kubernetes API
• (Immediately) The Envoy Gateway controller picks up the change via the Watch API and adjusts its own internal representation of the EndpointSlice to match
  • This can be seen by inspecting the resources.endpointSlices as returned by $ENVOY_GATEWAY_POD_IP:19000/api/config_dump?resource=all
• (After 30-120s) The endpoint config in the Envoy Proxy pod is updated with the new Pod IP addresses
  • This can be observed with "egctl config envoy-proxy endpoint

Given Envoy Gateway's own config dump near-immediately reflects the changes in endpoints, the delay lies somewhere in the xDS/translation layer between Envoy Gateway and the programmed Envoy Pods.

If a rollout restart happens too quickly, and all old Pods are replaced with new Pods before the updated IP addresses are propagated to Envoy Proxy Pods, then Envoy will be left unable to route any traffic as its only representation of Backend IP addresses is for Pods that no longer exist, even though both the Kubernetes API and Envoy Gateway almost immediately update with the new endpoint IPs. The only way to work around this race condition seems to be to significantly slow down rollouts with Deployment strategies and preStop hooks, however 30s seems far too long for such a race condition.

Repro steps:

Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Debug script to generate a visualisation of rollout of IP addresses is here: https://gist.github.com/coro/ab3638bd83c16adccb01d98625d5a395

While running this script, perform a rollout restart of a Deployment targeted by an Envoy Gateway HTTPRoute.

Note that we have tended to see this behaviour on specific Deployments with many (6+) replicas, and it is exacerbated by performing multiple restarts back to back (i.e. more Pod churn).

EDIT: As @duizabojul found in #7358, looks like this can actually also occur in single-replica Deployments, so perhaps not impacted by replica count.

Environment:

Include the environment like gateway version, envoy version and so on.

Seen across multiple versions, most recently:

EG v1.5.3
Envoy Proxy envoy:distroless-v1.35.3
EKS v1.32
Network topology: 2*Gateway, each with 6 Proxy Pods

Logs:

Include the access logs and the Envoy logs.

Output of the debug script is attached, showing the rollout of the IP addresses programmed in Envoy. Header meanings:

• ENVOY_HEALTH_STATUS - health of endpoint as measured in the Envoy Proxy Pod
• READY/SERVING/TERMINATING - endpoint condition as reported by the EndpointSlice
• EG_READY/EG_SERVING/EG_TERMINATING - endpoint condition as reported by Envoy Gateway controller config dump
• RED_DURATION - how long an endpoint had been in a 'red' state (which we define as Ready and Serving according to the EndpointSlice but not HEALTHY in Envoy Proxy

Output: out.txt

These logs show in this case delays of up to 30s for specific endpoints.

As discussed with @arkodg on Slack the Envoy Gateway controller logs are too large to attach here.

Related:
#4997
#7358

[arkodg]

thanks for the detailed debug o/p @coro

Since we've narrowed the delay/issue to somewhere from gateway-api runner subscribing to resources to xds server pushing config to envoy proxy, here are some next steps im proposing

1. better observability

   • Add support for watchable message tracing #7324 should help give us more accurate details about time taken in each layer
   • for now, we only have logs we can use, but there are no ids, so we need to rely on timestamp of the trigger (e.g. rollout)
     • K8s / Provider receives event: 2025-09-30T17:51:44.054Z INFO provider kubernetes/controller.go:272 reconciling gateways {"runner": "provider"}
     • Gateway API runner receives message: 2025-09-30T20:51:47.428Z INFO gateway-api runner/runner.go:131 received an update {"runner": "gateway-api"}
       xDS runner receives message: 2025-09-30T20:51:47.540Z INFO xds runner/runner.go:203 received an update {"runner": "xds"}

2. we've identified duplicate work that can be avoided coalesce updates to reduce intermediate updates #7328 and should be available to try out v1.6.0-rc.1 which should reduce the delay

[zhaohuabing]

I reproduced a similar issue with this setup. An unreachable OIDC issuer can delay the Gateway API translation, and the repeated updates from endpoint churn make it even worse.

Any other case that slows down translation could trigger the same behavior, and #7328 should help propagation delays by merging updates with the same keys.

[coro]

We have tested with v1.6.0-rc.1 and can confirm that this issue appears to be vastly better - IP propagation went from taking 30-120s to taking at most 4s, with the vast majority taking less than 1s. Thanks so much team! Can't wait to see how quick it is with the .well-known optimisation too!

[arkodg]

thanks for testing out the fix, and confirming that it drastically improves endpoint convergence, closing this issue for now

thanks @zhaohuabing for driving these enhancements !