Composite policy does not observe the policy_order in the config #36955
Labels
bug
Something isn't working
needs triage
New item requiring triage
processor/tailsampling
Tail sampling processor
Comments
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
|
If we're in agreement that this is indeed a bug, I'm happy to update the code to perform a sorting and add the tests. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component(s)
processor/tailsampling
What happened?
Description
Regardless of the order set in
composite.policy_order, the actual order of processing is the order in which the subpolicies are defined incomposite.composite_sub_policy.Steps to Reproduce
policy_orderdiffers from the order of subpolicies incomposite_sub_policyDEBUGmode to see logs about which policy is being usedExpected Result
The probabilistic subpolicy (PSP) receives the traces first, and any unsampled traces then move on to the always sample subpolicy (ASSP). If a ramping increase of spans/sec is used, we will at first see a 100% sampling rate for this Composite policy followed by an inflection point where the ASSP saturates at 100 spans/sec and 80 % of spans after this point are not sampled. Similarly, when we look at the exporter's spans/sec rate, we can see that there is an inflection at around this time, where the slope of the graph after the inflection is 20% of the slop before.
Logs for PSP seen before any logs for ASSP. For example:
Actual Result
The AASP always receives traces, PSP never receives anything. At first we see a 100% sampling rate in a ramping ingress, and then the AASP saturates, and we see the non sampling rate gradually increases. Similarly, we see the exporter's spans/sec rate flatline at 100 spans/sec (the saturation of the AASP).
Logs for ASSP always seen first, PP never seen at all:
Collector version
v0.116.0
Environment information
Environment
OS: Ubuntu 22.04
Compiler(if manually compiled): [email protected] go1.23.4
OpenTelemetry Collector configuration
Log output
Additional context
There's definitely a space for the policy order inside the config struct:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/config.go#L107
It's just that it's not being referenced anywhere:
https://github.com/search?q=repo%3Aopen-telemetry/opentelemetry-collector-contrib%20PolicyOrder&type=code
And the only test that's using it is just looking at this as an afterthought:
opentelemetry-collector-contrib/processor/tailsamplingprocessor/composite_helper_test.go
Line 21 in cb8b5d3
The text was updated successfully, but these errors were encountered: