[BUG] AKS 1.32 has coredns 1.12 which has a breaking change!

[Michael-Sinz]

Describe the bug
With AKS 1.32, the version of coredns is now at 1.12 which includes a breaking change: coredns/coredns#6898

What this does is not generate a DNS entry for stateless service pods (the pod specific entry).

Now, many services may not notice since they just use the standard kubernetes loadbalancer but our service has a much more complex load balancing requirement due to the vast non-uniformity of individual request (in the range of 4 orders of magnitude different amount of time/cost between requests and it is not knowable a priori)

Anyway, I think the coredns team accepts that this broke the prior contract. We reported it here: coredns/coredns#7177

To Reproduce
Create a stateless service with a deployment/replicaset. Try to a reverse DNS lookup by the pod's IP address. In AKS 1.32 this fails. In all prior AKS versions this worked. Make sure that each pod (of the replicaset) returns a unique value for their unique IP addresses.

Expected behavior
That service pods, once ready/healthy, have a DNS entry for that specific service pod.

Additional context
The workaround we have to continue to test in AKS 1.32 is definitely not seamless - it requires changes to the helm charts (deployment/pod specs) and our new custom admission controller that sets a unique hostname field for each pod instance of services that need this. None of which is a documented change in kubernetes or AKS but is a side-effect of the change that broke the behavior or coredns.

You can see that in the comment on the bug: coredns/coredns#7177 (comment)

[Michael-Sinz]

Note that switching to statefulsets is not a viable option - our services scale to thousands of pods per service/replicaset and we have hundreds of unique microservices (replicasets) that scale independently (not all scale to the same size at the same time but some number of them hit very large scale during peak usage.)

[sjwaight]

Duplicate of #4823.