Add support for a multi-region LB for tiles_tlog

[cmurphy]

This change adds submodules to tiles_tlog:

1. tiles_tlog/shared creates common resources like health checks and
   security policies that only need to be created once, universally, no
   matter the kind of deployment.
2. tiles_tlog/global creates another set of load balancer resources that
   support global write endpoints that may route to shards living in any
   given region.

The current region-specific load balancer resources remain the same,
because read traffic will always need to use the regional endpoint and
because the write endpoint makes it easier to check region-specific
liveness.

Relates to https://github.com/sigstore/public-good-instance/issues/3603

Summary

Release Note

Documentation

[Hayden-IO]

the current region-specific load balancer resources remain the same ... because the write endpoint makes it easier to check region-specific liveness.

To confirm, this means that a client could be configured to write directly to one of the regional logs? I don't think that's a concern, only would be an issue if we were using the LB to round-robin between instances.

[Hayden-IO]

Don't we need some of these resources for each tiles_tlog instance still in a multi-region deployment?

A per-shard healthcheck? I assume we'd need a healthcheck for the global LB as well, but there's no guarantee that its healthcheck would verify the health of each of the regional shards.

And if the write path is publicly accessible, should each shard have its own security policy? We could have a global rate limit and a per-shard rate limit.

[cmurphy]

We do still need them in a multi-region deployment, but no they do not need to be per-shard. A health check is a super generic resource, it is just information saying what path and port to make the health check on and it can be reused by every shard and every region, there is no need to have more than one if they are using the same port and path.

We could let each shard have its own security policy if we want. Do you realistically foresee that the global and per-shard rate limits should be different?

[cmurphy]

To confirm, this means that a client could be configured to write directly to one of the regional logs?

Yes, as it stands right now each shard is still writeable by their own shard-specific addresses. I did this more to keep the code simpler than anything. It also adds a little bit of redundancy, so that if a control plane outage takes down the global address the regional address is still available. But none of the other services have that redundancy so it's not providing a huge benefit.