-
Notifications
You must be signed in to change notification settings - Fork 293
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Revert "Add avoid double instrumenting lambda non-streaming handlers." #8247
Revert "Add avoid double instrumenting lambda non-streaming handlers." #8247
Conversation
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
Here’s my assessment of what’s going on. The original PR was a collaboration between myself and There are two types of lambda functions in java: streaming and non-streaming. In the non-streaming case, the handler signature will be something like public APIGatewayV2HTTPResponse handleRequest(APIGatewayV2HTTPEvent event, Context context) { While in the streaming case it’s always public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context) throws IOException { The problem we were seeing initially was that, due to using a single instrumentation class for both streaming and non-streaming, we were always getting null for the response payload in both cases. The thing that was causing this is how aws implements these two handler types under the hood. The lambda runtime defines a special non-streaming handler, which has the signature of a streaming handler, but calls the customer’s non-streaming handler under the hood. This way, the non-streaming handlers are executed by the runtime the same as streaming handlers. Our original version of the instrumentation for the non-streaming case wrapped both of these handlers, — the customer’s handler and the special runtime defined non-streaming converted to streaming handler — and unfortunately for us, the instrumentation on the streaming handler was firing first, thus preventing the instrumentation on the customer defined handler from executing. The change that we’re reverting forced the instrumentation to ignore the special lambda runtime defined non-streaming converted to streaming handler. |
FYI that reverting this change will put back into place a regression that has been present in the Datadog Lambda Java layer for some time. The instrumentation was not properly wrapping the handler and so was always receiving This meant that customers were unable to see their response payloads on their trace when |
78f10c6
to
13262f4
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 3 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.46.0-SNAPSHOT~356a1bd549, baseline=1.46.0-SNAPSHOT~0b239ae5cc
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.068 s) : 0, 1067724
Total [baseline] (10.612 s) : 0, 10611669
Agent [candidate] (1.054 s) : 0, 1053638
Total [candidate] (10.49 s) : 0, 10489868
section appsec
Agent [baseline] (1.19 s) : 0, 1189961
Total [baseline] (10.682 s) : 0, 10682182
Agent [candidate] (1.194 s) : 0, 1194340
Total [candidate] (10.733 s) : 0, 10732930
section iast
Agent [baseline] (1.182 s) : 0, 1182489
Total [baseline] (11.065 s) : 0, 11064934
Agent [candidate] (1.193 s) : 0, 1192925
Total [candidate] (11.01 s) : 0, 11010385
section profiling
Agent [baseline] (1.261 s) : 0, 1261134
Total [baseline] (10.876 s) : 0, 10875721
Agent [candidate] (1.252 s) : 0, 1252132
Total [candidate] (10.903 s) : 0, 10902785
gantt
title petclinic - break down per module: candidate=1.46.0-SNAPSHOT~356a1bd549, baseline=1.46.0-SNAPSHOT~0b239ae5cc
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (723.823 ms) : 0, 723823
BytebuddyAgent [candidate] (713.292 ms) : 0, 713292
GlobalTracer [baseline] (257.663 ms) : 0, 257663
GlobalTracer [candidate] (255.789 ms) : 0, 255789
AppSec [baseline] (56.618 ms) : 0, 56618
AppSec [candidate] (55.721 ms) : 0, 55721
Remote Config [baseline] (739.965 µs) : 0, 740
Remote Config [candidate] (751.763 µs) : 0, 752
Telemetry [baseline] (13.718 ms) : 0, 13718
Telemetry [candidate] (13.062 ms) : 0, 13062
section appsec
BytebuddyAgent [baseline] (732.371 ms) : 0, 732371
BytebuddyAgent [candidate] (734.826 ms) : 0, 734826
GlobalTracer [baseline] (253.251 ms) : 0, 253251
GlobalTracer [candidate] (253.885 ms) : 0, 253885
AppSec [baseline] (170.753 ms) : 0, 170753
AppSec [candidate] (171.852 ms) : 0, 171852
IAST [baseline] (19.423 ms) : 0, 19423
IAST [candidate] (19.699 ms) : 0, 19699
Remote Config [baseline] (666.921 µs) : 0, 667
Remote Config [candidate] (675.685 µs) : 0, 676
Telemetry [baseline] (8.168 ms) : 0, 8168
Telemetry [candidate] (8.221 ms) : 0, 8221
section iast
BytebuddyAgent [baseline] (831.862 ms) : 0, 831862
BytebuddyAgent [candidate] (839.461 ms) : 0, 839461
GlobalTracer [baseline] (246.712 ms) : 0, 246712
GlobalTracer [candidate] (248.175 ms) : 0, 248175
AppSec [baseline] (58.112 ms) : 0, 58112
AppSec [candidate] (58.591 ms) : 0, 58591
IAST [baseline] (21.537 ms) : 0, 21537
IAST [candidate] (21.972 ms) : 0, 21972
Remote Config [baseline] (670.756 µs) : 0, 671
Remote Config [candidate] (677.0 µs) : 0, 677
Telemetry [baseline] (8.71 ms) : 0, 8710
Telemetry [candidate] (8.942 ms) : 0, 8942
section profiling
BytebuddyAgent [baseline] (707.18 ms) : 0, 707180
BytebuddyAgent [candidate] (702.567 ms) : 0, 702567
GlobalTracer [baseline] (351.338 ms) : 0, 351338
GlobalTracer [candidate] (349.586 ms) : 0, 349586
AppSec [baseline] (54.875 ms) : 0, 54875
AppSec [candidate] (53.7 ms) : 0, 53700
Remote Config [baseline] (705.374 µs) : 0, 705
Remote Config [candidate] (695.857 µs) : 0, 696
Telemetry [baseline] (8.824 ms) : 0, 8824
Telemetry [candidate] (8.788 ms) : 0, 8788
ProfilingAgent [baseline] (96.031 ms) : 0, 96031
ProfilingAgent [candidate] (94.946 ms) : 0, 94946
Profiling [baseline] (96.055 ms) : 0, 96055
Profiling [candidate] (94.969 ms) : 0, 94969
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.46.0-SNAPSHOT~356a1bd549, baseline=1.46.0-SNAPSHOT~0b239ae5cc
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.059 s) : 0, 1059098
Total [baseline] (8.635 s) : 0, 8634610
Agent [candidate] (1.066 s) : 0, 1065620
Total [candidate] (8.639 s) : 0, 8639394
section iast
Agent [baseline] (1.185 s) : 0, 1185324
Total [baseline] (9.191 s) : 0, 9191080
Agent [candidate] (1.192 s) : 0, 1192496
Total [candidate] (9.23 s) : 0, 9229747
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.182 s) : 0, 1181562
Total [baseline] (9.173 s) : 0, 9172731
Agent [candidate] (1.182 s) : 0, 1182119
Total [candidate] (9.171 s) : 0, 9170524
section iast_TELEMETRY_OFF
Agent [baseline] (1.177 s) : 0, 1176774
Total [baseline] (9.187 s) : 0, 9186735
Agent [candidate] (1.18 s) : 0, 1180397
Total [candidate] (9.214 s) : 0, 9213675
gantt
title insecure-bank - break down per module: candidate=1.46.0-SNAPSHOT~356a1bd549, baseline=1.46.0-SNAPSHOT~0b239ae5cc
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (715.977 ms) : 0, 715977
BytebuddyAgent [candidate] (723.158 ms) : 0, 723158
GlobalTracer [baseline] (257.396 ms) : 0, 257396
GlobalTracer [candidate] (257.838 ms) : 0, 257838
AppSec [baseline] (55.928 ms) : 0, 55928
AppSec [candidate] (55.939 ms) : 0, 55939
Remote Config [baseline] (740.075 µs) : 0, 740
Remote Config [candidate] (742.57 µs) : 0, 743
Telemetry [baseline] (14.045 ms) : 0, 14045
Telemetry [candidate] (12.714 ms) : 0, 12714
section iast
BytebuddyAgent [baseline] (833.867 ms) : 0, 833867
BytebuddyAgent [candidate] (839.023 ms) : 0, 839023
GlobalTracer [baseline] (246.936 ms) : 0, 246936
GlobalTracer [candidate] (248.339 ms) : 0, 248339
AppSec [baseline] (58.35 ms) : 0, 58350
AppSec [candidate] (58.598 ms) : 0, 58598
IAST [baseline] (21.591 ms) : 0, 21591
IAST [candidate] (21.691 ms) : 0, 21691
Remote Config [baseline] (669.881 µs) : 0, 670
Remote Config [candidate] (675.704 µs) : 0, 676
Telemetry [baseline] (8.916 ms) : 0, 8916
Telemetry [candidate] (8.923 ms) : 0, 8923
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (831.06 ms) : 0, 831060
BytebuddyAgent [candidate] (831.312 ms) : 0, 831312
GlobalTracer [baseline] (246.569 ms) : 0, 246569
GlobalTracer [candidate] (246.482 ms) : 0, 246482
AppSec [baseline] (58.05 ms) : 0, 58050
AppSec [candidate] (58.14 ms) : 0, 58140
IAST [baseline] (21.432 ms) : 0, 21432
IAST [candidate] (21.682 ms) : 0, 21682
Remote Config [baseline] (672.362 µs) : 0, 672
Remote Config [candidate] (675.831 µs) : 0, 676
Telemetry [baseline] (8.813 ms) : 0, 8813
Telemetry [candidate] (8.853 ms) : 0, 8853
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (827.423 ms) : 0, 827423
BytebuddyAgent [candidate] (830.062 ms) : 0, 830062
GlobalTracer [baseline] (246.165 ms) : 0, 246165
GlobalTracer [candidate] (247.044 ms) : 0, 247044
AppSec [baseline] (57.947 ms) : 0, 57947
AppSec [candidate] (57.959 ms) : 0, 57959
IAST [baseline] (20.906 ms) : 0, 20906
IAST [candidate] (20.926 ms) : 0, 20926
Remote Config [baseline] (668.788 µs) : 0, 669
Remote Config [candidate] (675.193 µs) : 0, 675
Telemetry [baseline] (8.71 ms) : 0, 8710
Telemetry [candidate] (8.696 ms) : 0, 8696
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 1 performance regressions! Performance is the same for 11 metrics, 16 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~356a1bd549, baseline=1.46.0-SNAPSHOT~0b239ae5cc
dateFormat X
axisFormat %s
section baseline
no_agent (1.363 ms) : 1344, 1383
. : milestone, 1363,
appsec (1.732 ms) : 1708, 1756
. : milestone, 1732,
appsec_no_iast (1.783 ms) : 1759, 1808
. : milestone, 1783,
iast (1.509 ms) : 1484, 1534
. : milestone, 1509,
profiling (1.474 ms) : 1451, 1498
. : milestone, 1474,
tracing (1.5 ms) : 1476, 1525
. : milestone, 1500,
section candidate
no_agent (1.372 ms) : 1352, 1391
. : milestone, 1372,
appsec (1.757 ms) : 1733, 1781
. : milestone, 1757,
appsec_no_iast (1.761 ms) : 1738, 1784
. : milestone, 1761,
iast (1.519 ms) : 1494, 1543
. : milestone, 1519,
profiling (1.576 ms) : 1551, 1601
. : milestone, 1576,
tracing (1.494 ms) : 1470, 1518
. : milestone, 1494,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.46.0-SNAPSHOT~356a1bd549, baseline=1.46.0-SNAPSHOT~0b239ae5cc
dateFormat X
axisFormat %s
section baseline
no_agent (370.993 µs) : 351, 391
. : milestone, 371,
iast (509.341 µs) : 487, 531
. : milestone, 509,
iast_FULL (743.752 µs) : 722, 766
. : milestone, 744,
iast_GLOBAL (558.274 µs) : 536, 581
. : milestone, 558,
iast_HARDCODED_SECRET_DISABLED (508.514 µs) : 486, 531
. : milestone, 509,
iast_INACTIVE (462.835 µs) : 441, 485
. : milestone, 463,
iast_TELEMETRY_OFF (491.384 µs) : 469, 513
. : milestone, 491,
tracing (455.251 µs) : 434, 477
. : milestone, 455,
section candidate
no_agent (375.251 µs) : 355, 395
. : milestone, 375,
iast (509.396 µs) : 488, 531
. : milestone, 509,
iast_FULL (738.155 µs) : 716, 760
. : milestone, 738,
iast_GLOBAL (555.448 µs) : 533, 578
. : milestone, 555,
iast_HARDCODED_SECRET_DISABLED (500.811 µs) : 479, 523
. : milestone, 501,
iast_INACTIVE (461.536 µs) : 440, 483
. : milestone, 462,
iast_TELEMETRY_OFF (489.826 µs) : 468, 511
. : milestone, 490,
tracing (451.842 µs) : 431, 473
. : milestone, 452,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~356a1bd549, baseline=1.46.0-SNAPSHOT~0b239ae5cc
dateFormat X
axisFormat %s
section baseline
no_agent (1.471 ms) : 1460, 1483
. : milestone, 1471,
appsec (2.36 ms) : 2317, 2403
. : milestone, 2360,
iast (2.1 ms) : 2046, 2155
. : milestone, 2100,
iast_GLOBAL (2.153 ms) : 2098, 2208
. : milestone, 2153,
profiling (1.973 ms) : 1929, 2017
. : milestone, 1973,
tracing (1.962 ms) : 1919, 2004
. : milestone, 1962,
section candidate
no_agent (1.469 ms) : 1458, 1481
. : milestone, 1469,
appsec (2.361 ms) : 2318, 2404
. : milestone, 2361,
iast (2.096 ms) : 2042, 2150
. : milestone, 2096,
iast_GLOBAL (2.156 ms) : 2101, 2211
. : milestone, 2156,
profiling (2.0 ms) : 1955, 2044
. : milestone, 2000,
tracing (1.951 ms) : 1909, 1993
. : milestone, 1951,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.46.0-SNAPSHOT~356a1bd549, baseline=1.46.0-SNAPSHOT~0b239ae5cc
dateFormat X
axisFormat %s
section baseline
no_agent (15.416 s) : 15416000, 15416000
. : milestone, 15416000,
appsec (15.047 s) : 15047000, 15047000
. : milestone, 15047000,
iast (18.687 s) : 18687000, 18687000
. : milestone, 18687000,
iast_GLOBAL (18.174 s) : 18174000, 18174000
. : milestone, 18174000,
profiling (15.103 s) : 15103000, 15103000
. : milestone, 15103000,
tracing (14.995 s) : 14995000, 14995000
. : milestone, 14995000,
section candidate
no_agent (15.512 s) : 15512000, 15512000
. : milestone, 15512000,
appsec (15.031 s) : 15031000, 15031000
. : milestone, 15031000,
iast (18.852 s) : 18852000, 18852000
. : milestone, 18852000,
iast_GLOBAL (17.949 s) : 17949000, 17949000
. : milestone, 17949000,
profiling (14.959 s) : 14959000, 14959000
. : milestone, 14959000,
tracing (15.298 s) : 15298000, 15298000
. : milestone, 15298000,
|
e40cc3c
to
356a1bd
Compare
Reverts #8073
This caused a tracing regression on Lambda (serverless) where AWS inferred spans were no longer being created.
Built and test manually; inferred spans now work on this branch: