Option to propagate context via http request body instead of headers

[scheler]

we have a situation where one of the services (it's a 3rd party) is unable to participate in a trace and pass on the trace context. We want to leverage the fact that this 3rd party service passes on the http request content as is to the next service, and insert the tracecontext into the request.

The API propagator spec states that the carrier of a propagator can be an outgoing message or HTTP request, without specifying HTTP headers explicitly. yet, all the http instrumentations (fetch, xhr, node http) call the propagator.inject only to insert the context into HTTP headers.

I have a prototype to demonstrate what I am looking for - please see scheler#5

I am wondering what the community's thoughts are on this. If this is something that's generally helpful, then one way to address this is to modify the instrumentations so that they pass the entire http request (headers+body) to the propagator and leave to it to decide where to inject the context.

[Flarna]

I think the approach itself is ok for special case.
e.g. based on content-type one might know the body is formatted in a way that W3C tracecontext can be safe added even if the remote is not OTel aware.

Passing the whole request body to propagator might be problematic/difficult in some cases:

• body is compressed - at least in node.js compression happens before data arrives on http layer
• body is large - at least node.js support streaming therefore no need to have all data in memory

Therefore I assume such a propagator is hard to add on default. It would be needed to decided based on e.g. request headers that inject into body should be done to decide on buffering the whole body to inject and finally send.

Update:
Another problem might be on the server side. HTTP request headers are present from beginning therefore starting a span with correct parent is easy. If W3C context is in (potentially compressed) body it would be needed to first receive the whole body before the span with correct parent can be started.

[scheler]

@Flarna thanks for your comments. In my special case, the client is browser and so let's say we assume the body is smaller and uncompressed. Also, let's say the downsides you listed for the server side are also acceptable. What do your thoughts on how we can make this configurable so that the xhr instrumentation is able to pass the entire request (headers+body) to the propagator?

[Flarna]

As I have no experience with browser instrumentations/APIs this would be best answered by an browser expert. XHR instrumentation was initially created by @obecny, maybe he has some good idea.

Anyhow, did a fast look into the XHR instrumentation and it seems send() is already patched an there plugin._addHeaders(this, spanUrl) is called which actually calls inject.

I think it should be not that hard to include the body in this call chain.

The harder part is to find out which propagator(s) are actually configured to provide the correct setter to inject to do the correct body modifications.
Propagators are configured global therefore instrumentations don't know if/which ones are present.
Main idea here is that OTel configuration and instrumentation implementation is decoupled.

In your special case an inject hook might be the best joice. if configured instrumentation calls your hook along with url/span/headers/body/... and your hook takes over the actual modification. Return value could indicates to instrumentation if global propagator should be called in addition.

Alternative is to do all this in an handcrafted layer on top of XHR in your code and disable XHR instrumentation. It would be by far easier to debug and easier to understand in my opinion.

[obecny]

what if you try to patch the xhr / fetch and simply do any extraction you need from the 3rd party service (from body etc.) and inject this into headers. So this way you dont need to modify antyhing from otel. You would simply write a new plugin for you specific case. this plugin can be added either on top of xhr or fetch, or simply disable the otel xhr and fetch and do the handling you need.

I also think that trying to modify the body would be a big NO, because there can be very specific data and the risk of leaking this data anywhere else is also worth to notice. So I would personally not touch the body at all. Otel should not cause any harm, and even if you think your code will not do it, there is always some potential in breaking something if you start manipulating the area which is not meant for that.

[scheler]

@obecny thanks for your comments. Patching xhr/fetch separately outside otel and injecting the header into the body sounds like a better idea. I am not sure if patching twice could cause any issues, but I guess we could try and see.

Disabling otel xhr instrumentation is not an option as that's a critical piece providing the browser side span in the trace.

Regarding the security concern you raised, what is the reason sqsExtractContextPropagationFromPayload exists?

[Flarna]

I am not sure if patching twice could cause any issues, but I guess we could try and see.

Monkeypatching is always a risk. Doing it twice at least doubles the risk. But as JS provides no other mechanism for monitoring/instrumentation there is not that much one can do.

Disabling otel xhr instrumentation is not an option as that's a critical piece providing the browser side span in the trace.

Well, it depends how you patch XHR. If you fork and tune the OTel XHR instrumentation for your needs it's likely better to disable the OTel variant otherwise you get two spans for each XHR. This avoids also the double patching topic.