Allow for developer/commandline control of name resolution

[toppk]

I don't think httpx must create any new functionality but I think it will eventually choose to make users/developers lives
easier regarding this capability. I would be happy to tailor my code around any such changes or simply allow others
to leverage my code as a starting point for them building a similar capability.

the below gist has this commentary and code (including a main that exercises the code using httpbin.org service ip addreses)

https://gist.github.com/toppk/a5838c9727d806c7845f7be0e2eacf2f

There are two crazy things here, what I am doing and why.

First the why:

There is a need to distinguish name resolution from a traditional http
client. There are three primary use cases:

1. HTTP ServerName or TLS SAN/cn doesn't match DNS. There are deployments
   that for one reason or another decided to use orthogonal naming for DNS
   and inside x509 certificates. This includes major VPN providers.

2. The second case is if you wish to control DNS queries outside of a
   traditional client, meaning that you don't want to do a query each time
   a new connection is needed (for example, if you make a call every 20
   seconds (outside most http keep-alive settings) a new dns lookup will
   occur. The goal may be to reduce call volume or latency, or
   network chatter. There is also an issue with cascading service failure
   during a DNS outage. If DNS isn't currently operating it may be
   desirable to remember DNS lookup results for an extended period of time
   (hours or even days), and only attempt a new lookup if http calls are
   not responding correctly.

There are a couple of notable DNS outages where there was significant
fallout due to these cascading failures which impeading resolving the DNS
issue (e.g. if authentication service fails when DNS is down, and you need
authentication to remedy the DNS outage)

3. There is another use case where you wish to target a specific host insider a
   "multiple a record" dns hostname, and you wish to specifiy an individual IP
   rather then the hostname that is in used by http/tls on that IP. This is a
   poor man's service discovery layer.

A well implement Service Discovery layer generally addresses a bunch of these
issues but I think it is desirable to expose name resolution to developers.
In most deployments, it's entirely unknown to the application when a dns call
is being made, in fact, due to using OS based name resolution where there is
an opportunity for caching, it is likely unknown to the library as well when
a dns query is actually made.

Here is an implementation to add this capability. This is the how:

Creating a custom Transport was the basis for the implementation, which operates
a basic mechanism.

1. it monkey patches the socket connection method (connect_tcp) inside httpcore backend.
2. inside the Transport.handle_request it populates necessary datastructure needed
   for the patched connection call
3. Additionally there are several additional facilities inside the Transport

• methods to retry based on connectionfailure if there are multiple target IP for a hostname
• methods to update hard coded hostname -> ip mappings
• a facility to provide a user lookup method, so DNS can be replaced with something else

My thoughts

use cases

I do think that a proper service discovery capability would be built as a separate
service or with dns srv records, and not with multiple A records. Although I do
think it is entirely reasonable for a http library to have some advanced logic here.
Even WUA have some handling of multiple A records (or they did at one point).

I think the caching/dns outages resiliency is an obvious need, and there are examples
in other libraries that allow for a user provided lookup function.

Finally having a separate hostname used in http/tls layer from hostname used for
socket creation is also an obvious need, especially by command line tools, as shown by
implementations in several popular command line http clients.

implementation

I think providing a developer defined lookup that could be plumbed into the appropriate
httpcore connect_tcp methods would be a good option. I happened to indepently create
a method with the identical name and signature as in okhttp, which makes sense if you
look at it:

 `lookup(hostname:str) -> List[ip_address]`

Another option would be to plumb a separate hostname inside httpcore's origin, which
if present would cause connect_tcp to be provided this alternative hostname.

Additionally since the connection will be maintained inside the pool, there is a
desire to manage the lifecycle of those connections if for example you want to
switch a connection to a different socket. Without going into the internals of
the pool, there is only the choice of connection.close(). This has an abrupt
effect, and it would be desirable to mark a connection as pending such that as
soon as any in-flight requests are wrapped up it is to be closed and removed,
and any new request should immediately start a new connection. for example, if
I have a client who starts getting 500's and wants to cycle to the next IP on the
list, you need to force the connection to go away. A little bit more subtle is
if the server returns a status code that indicates server overwhelmed. Again, you
would want to shift to a different IP, but not terminate the socket right away
under any other threads happen to be using the connection.

These pool lifecycle tricks are really for the poor mans service discovery, and
I think it's entirely reasonable to punt on them, but I do think that there are
absolutely mutiple ip address potentially associated with a hostname, and the
question really what would you do if inside the library if you were aware of that?

prior art

urllib3 suggests using a special connection pool, which is an interesting solution. I was thinking of a special proxy as an alternative implementation. For my requests/urllib3 needs, I chose the monkey patch route (urllib3.util.connection.create_connection)

https://urllib3.readthedocs.io/en/latest/advanced-usage.html#custom-sni-hostname

curl has two methods to change host that I connect to vs hostname that I use inside http/tls

curl --connect-to --resolve

okhttp user provided lookup implementation:

https://square.github.io/okhttp/4.x/okhttp/okhttp3/-dns/
https://developpaper.com/httpdns-okhttp-best-practices/

some random obligatory semi useless SE link:

https://serverfault.com/questions/327708/how-browsers-handle-multiple-ips

[tomchristie]

Great question!

As it happens httpx and httpcore do currently allow you to connect to a different IP than is used in the Host: header, but that's not a lot of use on it's own, because, eg...

# This will connect to 1.2.3.4 but using the `www.example.com` header,
# *but* the sni hostname will also be 1.2.3.4 which isn't what we wanted.
$ httpx https://1.2.3.4 --header Host www.example.com

I think we've got space for two different request extensions here...

r = httpx.get("https://www.example.com", extensions={"connect_to": ...})

r = httpx.get("https://www.example.com", extensions={"sni_hostname": ...})

We'd then probably want to expose these to the command line client in a way that maps obviously onto the programmatic API.

Perhaps this...

$ httpx https://www.example.com --x-connect-to ...

$ httpx https://www.example.com --x-sni-hostname ...

At some point I would also like to see httpcore get some low-level API for programmatically setting the network backends, controlling DNS lookups etc, but these two extensions probably cover enough of what you've asked about on their own.

[toppk]

okay, i'll take a stab at those extensions, let me know if you've already gotten started.

[tomchristie]

Fab! I'd suggest starting with just the sni_hostname extension, in httpcore.

(There's some complexities with connect_to that I'll discuss when we get there.)

🌟

[Allgot]

I've searched as much as I can, but I couldn't find any feasible answers except this one.

So, was there any advance?

If so, could you explain how to specify both of a hostname and an IP address?

(Added: Sorry for pulling the old thread, I wasn't aware of that..)