GKE Autopilot deprecating insecure kubelet readonly port 10255--and RBAC node/proxy resource not supported

[tkoft]

GKE is deprecating the unsecured kubelet read-only port 10255 in its clusters in the near future, we got an email encouraging us to migrate any applications to the https 10250 port instead.

I disabled the insecure port on our autopilot cluster, and set DD_KUBERNETES_HTTPS_KUBELET_PORT=10250 in our helm chart.

After doing this, I'm still getting failures, e.g.:

2024-12-13 00:04:39 UTC | CORE | WARN | (comp/core/workloadmeta/impl/store.go:599 in func1) | error pulling from collector "kube_metadata": couldn't fetch "podlist": unexpected status code 403 on https://10.10.15.215:10250/pods: Forbidden (user=system:serviceaccount:ddagent:datadog-agent, verb=get, resource=nodes, subresource=proxy)

At first I thought this was an RBAC permissions issue with the agent's service account, but after looking through some docs it seems like the required node/proxy resource permission is disabled in GKE autopilot:

If your workload uses the /pods endpoint on the insecure kubelet read-only port, you need to grant the nodes/proxy RBAC permission to access the endpoint on the secure kubelet port. nodes/proxy is a powerful permission that you can't grant in GKE Autopilot clusters and that you shouldn't grant in GKE Standard clusters. Use the Kubernetes API with a fieldSelector for the node name instead.

(Incidentally, I think this is why the HTTPS port was bypassed for GKE autopilot in the first place in the datadog agent helm chart, a separate issue I posted about here)

Is there any workaround or planned fix for this?

[tbavelier]

Re-adding my reply from the Helm chart issue: DataDog/helm-charts#1635 (comment)

We are aware of this incoming depreciation and are working with Google and other Datadog engineering teams towards a solution as indeed, nodes/proxy cannot be used preventing our usage of the HTTPS kubelet port. Until such work is completed, to keep full Agent functionality, the insecure port should remain enabled for GKE Autopilot clusters.