Add pending traces to tracer flare #8053
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksStartupParameters
See matching parameters
SummaryFound 6 performance improvements and 16 performance regressions! Performance is the same for 33 metrics, 8 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~7504373a0c, baseline=1.45.0-SNAPSHOT~e8e9292d06
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1058304
Total [baseline] (8.666 s) : 0, 8665540
Agent [candidate] (1.094 s) : 0, 1094455
Total [candidate] (8.649 s) : 0, 8648938
section iast
Agent [baseline] (1.175 s) : 0, 1174972
Total [baseline] (9.186 s) : 0, 9185947
Agent [candidate] (1.224 s) : 0, 1224472
Total [candidate] (9.213 s) : 0, 9213074
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.174 s) : 0, 1174210
Total [baseline] (9.168 s) : 0, 9168336
Agent [candidate] (1.224 s) : 0, 1223800
Total [candidate] (9.195 s) : 0, 9194581
section iast_TELEMETRY_OFF
Agent [baseline] (1.17 s) : 0, 1170156
Total [baseline] (9.187 s) : 0, 9187326
Agent [candidate] (1.222 s) : 0, 1222460
Total [candidate] (9.213 s) : 0, 9213460
gantt
title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~7504373a0c, baseline=1.45.0-SNAPSHOT~e8e9292d06
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (716.226 ms) : 0, 716226
BytebuddyAgent [candidate] (695.937 ms) : 0, 695937
GlobalTracer [baseline] (256.565 ms) : 0, 256565
GlobalTracer [candidate] (319.017 ms) : 0, 319017
AppSec [baseline] (58.669 ms) : 0, 58669
AppSec [candidate] (54.507 ms) : 0, 54507
Remote Config [baseline] (691.391 µs) : 0, 691
Remote Config [candidate] (676.888 µs) : 0, 677
Telemetry [baseline] (11.113 ms) : 0, 11113
Telemetry [candidate] (10.582 ms) : 0, 10582
section iast
BytebuddyAgent [baseline] (826.704 ms) : 0, 826704
BytebuddyAgent [candidate] (815.147 ms) : 0, 815147
GlobalTracer [baseline] (245.885 ms) : 0, 245885
GlobalTracer [candidate] (308.663 ms) : 0, 308663
AppSec [baseline] (57.419 ms) : 0, 57419
AppSec [candidate] (56.987 ms) : 0, 56987
Remote Config [baseline] (646.28 µs) : 0, 646
Remote Config [candidate] (617.364 µs) : 0, 617
Telemetry [baseline] (8.424 ms) : 0, 8424
Telemetry [candidate] (7.472 ms) : 0, 7472
IAST [baseline] (20.991 ms) : 0, 20991
IAST [candidate] (21.771 ms) : 0, 21771
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (825.935 ms) : 0, 825935
BytebuddyAgent [candidate] (815.349 ms) : 0, 815349
GlobalTracer [baseline] (245.776 ms) : 0, 245776
GlobalTracer [candidate] (307.808 ms) : 0, 307808
AppSec [baseline] (57.475 ms) : 0, 57475
AppSec [candidate] (57.69 ms) : 0, 57690
Remote Config [baseline] (644.853 µs) : 0, 645
Remote Config [candidate] (612.527 µs) : 0, 613
Telemetry [baseline] (8.402 ms) : 0, 8402
Telemetry [candidate] (7.452 ms) : 0, 7452
IAST [baseline] (21.089 ms) : 0, 21089
IAST [candidate] (21.096 ms) : 0, 21096
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (822.213 ms) : 0, 822213
BytebuddyAgent [candidate] (813.223 ms) : 0, 813223
GlobalTracer [baseline] (245.736 ms) : 0, 245736
GlobalTracer [candidate] (307.986 ms) : 0, 307986
AppSec [baseline] (57.66 ms) : 0, 57660
AppSec [candidate] (57.482 ms) : 0, 57482
Remote Config [baseline] (649.15 µs) : 0, 649
Remote Config [candidate] (623.568 µs) : 0, 624
Telemetry [baseline] (8.364 ms) : 0, 8364
Telemetry [candidate] (7.493 ms) : 0, 7493
IAST [baseline] (20.695 ms) : 0, 20695
IAST [candidate] (21.854 ms) : 0, 21854
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~7504373a0c, baseline=1.45.0-SNAPSHOT~e8e9292d06
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1055676
Total [baseline] (10.487 s) : 0, 10486704
Agent [candidate] (1.093 s) : 0, 1093175
Total [candidate] (10.466 s) : 0, 10465667
section appsec
Agent [baseline] (1.188 s) : 0, 1187583
Total [baseline] (10.713 s) : 0, 10712895
Agent [candidate] (1.234 s) : 0, 1233738
Total [candidate] (10.671 s) : 0, 10671416
section iast
Agent [baseline] (1.174 s) : 0, 1174208
Total [baseline] (10.941 s) : 0, 10941392
Agent [candidate] (1.238 s) : 0, 1238401
Total [candidate] (11.024 s) : 0, 11024253
section profiling
Agent [baseline] (1.269 s) : 0, 1268615
Total [baseline] (10.788 s) : 0, 10787607
Agent [candidate] (1.318 s) : 0, 1317882
Total [candidate] (10.854 s) : 0, 10854419
gantt
title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~7504373a0c, baseline=1.45.0-SNAPSHOT~e8e9292d06
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (713.674 ms) : 0, 713674
BytebuddyAgent [candidate] (695.682 ms) : 0, 695682
GlobalTracer [baseline] (255.92 ms) : 0, 255920
GlobalTracer [candidate] (318.847 ms) : 0, 318847
AppSec [baseline] (57.906 ms) : 0, 57906
AppSec [candidate] (54.323 ms) : 0, 54323
Remote Config [baseline] (695.258 µs) : 0, 695
Remote Config [candidate] (675.988 µs) : 0, 676
Telemetry [baseline] (12.539 ms) : 0, 12539
Telemetry [candidate] (9.906 ms) : 0, 9906
section appsec
BytebuddyAgent [baseline] (730.054 ms) : 0, 730054
BytebuddyAgent [candidate] (716.115 ms) : 0, 716115
GlobalTracer [baseline] (253.29 ms) : 0, 253290
GlobalTracer [candidate] (318.032 ms) : 0, 318032
AppSec [baseline] (170.332 ms) : 0, 170332
AppSec [candidate] (167.01 ms) : 0, 167010
Remote Config [baseline] (649.731 µs) : 0, 650
Remote Config [candidate] (647.896 µs) : 0, 648
Telemetry [baseline] (8.634 ms) : 0, 8634
Telemetry [candidate] (8.182 ms) : 0, 8182
IAST [baseline] (19.196 ms) : 0, 19196
IAST [candidate] (19.96 ms) : 0, 19960
section iast
BytebuddyAgent [baseline] (826.213 ms) : 0, 826213
BytebuddyAgent [candidate] (824.864 ms) : 0, 824864
GlobalTracer [baseline] (245.389 ms) : 0, 245389
GlobalTracer [candidate] (311.29 ms) : 0, 311290
AppSec [baseline] (57.755 ms) : 0, 57755
AppSec [candidate] (58.572 ms) : 0, 58572
Remote Config [baseline] (643.02 µs) : 0, 643
Remote Config [candidate] (628.034 µs) : 0, 628
Telemetry [baseline] (8.399 ms) : 0, 8399
Telemetry [candidate] (7.605 ms) : 0, 7605
IAST [baseline] (20.949 ms) : 0, 20949
IAST [candidate] (21.38 ms) : 0, 21380
section profiling
BytebuddyAgent [baseline] (699.479 ms) : 0, 699479
BytebuddyAgent [candidate] (690.306 ms) : 0, 690306
GlobalTracer [baseline] (369.901 ms) : 0, 369901
GlobalTracer [candidate] (433.684 ms) : 0, 433684
AppSec [baseline] (53.941 ms) : 0, 53941
AppSec [candidate] (53.115 ms) : 0, 53115
Remote Config [baseline] (655.066 µs) : 0, 655
Remote Config [candidate] (657.854 µs) : 0, 658
Telemetry [baseline] (7.806 ms) : 0, 7806
Telemetry [candidate] (7.698 ms) : 0, 7698
ProfilingAgent [baseline] (95.29 ms) : 0, 95290
ProfilingAgent [candidate] (93.202 ms) : 0, 93202
Profiling [baseline] (95.315 ms) : 0, 95315
Profiling [candidate] (93.226 ms) : 0, 93226
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~7504373a0c, baseline=1.45.0-SNAPSHOT~e8e9292d06
dateFormat X
axisFormat %s
section baseline
no_agent (370.861 µs) : 351, 390
. : milestone, 371,
iast (494.079 µs) : 472, 516
. : milestone, 494,
iast_FULL (650.186 µs) : 629, 672
. : milestone, 650,
iast_GLOBAL (527.907 µs) : 505, 551
. : milestone, 528,
iast_HARDCODED_SECRET_DISABLED (491.176 µs) : 470, 513
. : milestone, 491,
iast_INACTIVE (455.291 µs) : 434, 477
. : milestone, 455,
iast_TELEMETRY_OFF (481.669 µs) : 460, 504
. : milestone, 482,
tracing (449.39 µs) : 428, 470
. : milestone, 449,
section candidate
no_agent (375.379 µs) : 355, 396
. : milestone, 375,
iast (495.84 µs) : 474, 517
. : milestone, 496,
iast_FULL (650.776 µs) : 629, 672
. : milestone, 651,
iast_GLOBAL (514.775 µs) : 492, 537
. : milestone, 515,
iast_HARDCODED_SECRET_DISABLED (492.949 µs) : 472, 514
. : milestone, 493,
iast_INACTIVE (453.1 µs) : 431, 475
. : milestone, 453,
iast_TELEMETRY_OFF (484.049 µs) : 462, 506
. : milestone, 484,
tracing (444.789 µs) : 424, 465
. : milestone, 445,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~7504373a0c, baseline=1.45.0-SNAPSHOT~e8e9292d06
dateFormat X
axisFormat %s
section baseline
no_agent (1.348 ms) : 1329, 1367
. : milestone, 1348,
appsec (1.749 ms) : 1726, 1773
. : milestone, 1749,
appsec_no_iast (1.737 ms) : 1711, 1762
. : milestone, 1737,
iast (1.502 ms) : 1478, 1526
. : milestone, 1502,
profiling (1.519 ms) : 1496, 1542
. : milestone, 1519,
tracing (1.487 ms) : 1461, 1513
. : milestone, 1487,
section candidate
no_agent (1.349 ms) : 1329, 1369
. : milestone, 1349,
appsec (1.762 ms) : 1739, 1786
. : milestone, 1762,
appsec_no_iast (1.757 ms) : 1730, 1784
. : milestone, 1757,
iast (1.499 ms) : 1476, 1521
. : milestone, 1499,
profiling (1.49 ms) : 1467, 1514
. : milestone, 1490,
tracing (1.496 ms) : 1471, 1522
. : milestone, 1496,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~7504373a0c, baseline=1.45.0-SNAPSHOT~e8e9292d06
dateFormat X
axisFormat %s
section baseline
no_agent (1.468 ms) : 1456, 1479
. : milestone, 1468,
appsec (2.344 ms) : 2302, 2386
. : milestone, 2344,
iast (2.08 ms) : 2027, 2132
. : milestone, 2080,
iast_GLOBAL (2.121 ms) : 2067, 2174
. : milestone, 2121,
profiling (1.975 ms) : 1931, 2019
. : milestone, 1975,
tracing (1.924 ms) : 1883, 1964
. : milestone, 1924,
section candidate
no_agent (1.467 ms) : 1456, 1479
. : milestone, 1467,
appsec (2.336 ms) : 2294, 2377
. : milestone, 2336,
iast (2.078 ms) : 2025, 2131
. : milestone, 2078,
iast_GLOBAL (2.118 ms) : 2065, 2171
. : milestone, 2118,
profiling (2.445 ms) : 2256, 2634
. : milestone, 2445,
tracing (1.924 ms) : 1883, 1964
. : milestone, 1924,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~7504373a0c, baseline=1.45.0-SNAPSHOT~e8e9292d06
dateFormat X
axisFormat %s
section baseline
no_agent (15.542 s) : 15542000, 15542000
. : milestone, 15542000,
appsec (14.975 s) : 14975000, 14975000
. : milestone, 14975000,
iast (18.995 s) : 18995000, 18995000
. : milestone, 18995000,
iast_GLOBAL (17.705 s) : 17705000, 17705000
. : milestone, 17705000,
profiling (15.458 s) : 15458000, 15458000
. : milestone, 15458000,
tracing (15.107 s) : 15107000, 15107000
. : milestone, 15107000,
section candidate
no_agent (14.845 s) : 14845000, 14845000
. : milestone, 14845000,
appsec (14.898 s) : 14898000, 14898000
. : milestone, 14898000,
iast (18.725 s) : 18725000, 18725000
. : milestone, 18725000,
iast_GLOBAL (17.99 s) : 17990000, 17990000
. : milestone, 17990000,
profiling (14.967 s) : 14967000, 14967000
. : milestone, 14967000,
tracing (15.221 s) : 15221000, 15221000
. : milestone, 15221000,
|
dd-trace-core/src/main/java/datadog/trace/core/flare/TracerFlareService.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
PerfectSlayer
changed the title
mcculls
reviewed
Dec 12, 2024
mcculls
reviewed
Dec 12, 2024
mcculls
reviewed
Dec 12, 2024
dd-trace-core/src/main/java/datadog/trace/core/PendingTrace.java
Outdated
Show resolved
Hide resolved
mcculls
reviewed
Dec 12, 2024
dd-trace-core/src/main/java/datadog/trace/core/util/TracerDump.java
Outdated
Show resolved
Hide resolved
|
Have you considered tackling this another way, by adding the reporting to |
mhlidd
force-pushed
the
mhlidd/pending_traces
branch
from
4be2999 to
3e4955a
Compare
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
| } | ||
|
|
||
| DumpSupplier supplier = new DumpSupplier(DumpDrain.data); | ||
| buffer.queue.fill(supplier, supplier.data.size()); |
There was a problem hiding this comment.
I wonder if filling the queue back with the original element can't be done when handling the dump element.
It would limit having new element inserted in the same time and messing with the queue order. WDYT?
There was a problem hiding this comment.
If so, does it make sense having the same object implement both Consummer and Supplier?
There was a problem hiding this comment.
Will add this in the new commit!
| if (e instanceof PendingTrace) { | ||
| PendingTrace trace = (PendingTrace) e; | ||
| for (DDSpan span : trace.getSpans()) { | ||
| dumpText.append(span.toString()).append("\n"); |
There was a problem hiding this comment.
'\n' for single char (instead of using a String object)
There was a problem hiding this comment.
Have drain and fill together feels way better! 🙌
Now, about using the collected elements:
- You can declare your comparator as a
private final Comparator<List<Span>> TRACE_BY_START_DATEfield so its meaning will be clear and it will only be allocated once - You will need to figure out how to filter, limit, and map your elements. I would recommend having a look at the Java
StreamAPI - Now you get most of the part in place, try building some tests / triggering your code. You should be able to find some NPE by yourself 😉
| @@ -182,11 +242,18 @@ public void run() { | |||
|
|
|||
| if (pendingTrace instanceof FlushElement) { | |||
| // Since this is an MPSC queue, the drain needs to be called on the consumer thread | |||
| queue.drain(WriteDrain.WRITE_DRAIN); | |||
| queue.drain(WriteDrain.WRITE_DRAIN, 50); | |||
There was a problem hiding this comment.
Hmm I was trying to use the MPSC drain method that takes in a limit to the number of elements that get drained. Do you think that I should avoid doing this and using the Java stream API instead?
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
dd-trace-core/src/main/java/datadog/trace/core/PendingTraceBuffer.java
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: APMAPI-906