[Spike] Weighted DNS response parity

[maleck13]

What

Investigate how we can have a weighted DNS response that gives us parity on what we offer now with Cloud DNS providers.

Ideally we would figure this out via some form of PoC to prove the ideas

Done

• Identify if it is possible
• Create a plan for the implementation and document

[mikenairn]

Weighted records handling was added to the POC here

It makes use of the CoreDNS loadbalanced plugin logic.

We can't use that plugin directly since it only deals with weighted A records and expects a weight value to be assigned to the ip address. We want deal with weights being assigned to any type of record (CNAME or A), and dealt with at each stage of the recursive resolve of CNAME records down to A.

The logic is not overly complicated so i don't think its a huge issue to have that dealt with in our own code base, It gives us more control over how it works and makes fixing issues issues easier if needs be. Other similar plugins to ours, that do similar things with weighting have also opted to implement much of it themselves.