On-demand server-timing

[steve-chavez]

Problem

Setting server-timing-enabled to true has a ~6.5% RPS decrease. Using GETSimple on https://github.com/PostgREST/postgrest-benchmark:

no timings = 2176.4712194 rps
with timings = 2033.3584706 rps

I don't think there's a way around it since calculating the timings requires using clock time which needs a system call (clock_gettime).

Solution

Allow getting server timing headers through a new preference header.

Prefer: metrics=timings

metrics=timings would be the only value for now, but we could extend it later on.

This can later be enabled/disabled with #2987.

While at it, deprecate server-timing-enabled since it has a perf impact.

[develop7]

Setting server-timing-enabled to true has a ~6.5% RPS decrease.

Damn, I wonder how much slower will OTel be when enabled then.

[steve-chavez]

Yeah, this makes sense since server-timing is a trace and by definition:

Running traces takes too much time and consumes too many resources to trace every request an application receives.
https://www.techtarget.com/searchitoperations/tip/The-3-pillars-of-observability-Logs-metrics-and-traces

Damn, I wonder how much slower will OTel be when enabled then.

Not yet familiar with OTel (and the Push model) but doesn't it do some buffering before sending the events to a collector?

I've also read that they do sampling, so not all requests are traced. This would reduce resource consumption.

---

It would be interesting to have an internal sampling mechanism, so we could have some server-timing traces exposed as metrics. (just an idea for now, not sure if worth it)

[taimoorzaeem]

@steve-chavez Do we really need the Preference-Applied: ... for this? I guess not.

How about we solve this with a custom header instead of a preference? It would be easier to parse it in Auth.hs module because it would just a be simple header lookup. Maybe something like X-Metrics: timings? Later, we can extend it to other values too?

[steve-chavez]

Hm, we always frown upon using non-standard headers as it adds complexity (for us and the user) plus the Auth.hs module doesn't seem like the right place to do this. I think this should wait until the proposed refactor in #4064 (comment) is done.