Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

POC of a simplified code origin probe #8081

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

POC of a simplified code origin probe #8081

wants to merge 3 commits into from

Conversation

evanchooly
Copy link
Contributor

Motivation

In order to minimize the impact of the code origin feature as much as possible, this is a POC to make the code origin probe only care about code location and not bother with snapshots. However, to preserve the UX of automatically getting snapshot information at the code origin sites, a dynamic snapshot probe can be optionally added at the same site to capture that snapshot during a debug session. This preserves the current functionality while minimizing the impact outside of a debug session.

@pr-commenter
Copy link

pr-commenter bot commented Dec 12, 2024 •
edited
Loading

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1734105697 1734106080
end_time 2024-12-13T16:02:53 2024-12-13T16:09:17
git_branch master evanchooly/codeorigin
git_commit_sha bdc9eae 199e8e8
start_time 2024-12-13T16:01:38 2024-12-13T16:08:01
See matching parameters
Baseline Candidate
ci_job_id 738636513 738636513
ci_pipeline_id 51090856 51090856
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date 1734105188 1734105188

Summary

Found 4 performance improvements and 0 performance regressions! Performance is the same for 5 metrics, 6 unstable metrics.

scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:loop better
[-424.742µs; -257.620µs] or [-4.023%; -2.440%]		 better
[-446.070µs; -266.849µs] or [-4.153%; -2.484%]		 better
[-465.783µs; -271.802µs] or [-4.297%; -2.507%]		 unsure
[-626.122µs; -19.917µs] or [-5.569%; -0.177%]		 better
[+1.719op/s; +3.572op/s] or [+1.856%; +3.858%]
See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-16.941µs; +27.080µs] or [-6.420%; +10.261%]		 unstable
[-24.769µs; +43.627µs] or [-8.213%; +14.467%]		 unstable
[-35.309µs; +58.387µs] or [-11.186%; +18.497%]		 unstable
[-66.551µs; +152.227µs] or [-11.435%; +26.157%]		 same
scenario:basic same unsure
[+2.157µs; +21.281µs] or [+0.715%; +7.054%]		 unsure
[+1.287µs; +23.695µs] or [+0.413%; +7.599%]		 unstable
[-58.627µs; +72.114µs] or [-10.773%; +13.252%]		 unstable
[-223.185op/s; +94.979op/s] or [-8.704%; +3.704%]
Request duration reports for reports 
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (301.569 µs) : 278, 325
.   : milestone, 302,
basic (301.698 µs) : 292, 311
.   : milestone, 302,
loop (10.741 ms) : 10628, 10854
.   : milestone, 10741,
section candidate
noprobe (310.998 µs) : 271, 351
.   : milestone, 311,
basic (313.418 µs) : 304, 323
.   : milestone, 313,
loop (10.385 ms) : 10352, 10418
.   : milestone, 10385,
Loading
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 301.569 µs [278.112 µs, 325.027 µs]
basic 301.698 µs [292.47 µs, 310.927 µs]
loop 10.741 ms [10.628 ms, 10.854 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 310.998 µs [271.308 µs, 350.688 µs]
basic 313.418 µs [303.952 µs, 322.883 µs]
loop 10.385 ms [10.352 ms, 10.418 ms]

@pr-commenter
Copy link

pr-commenter bot commented Dec 12, 2024 •
edited
Loading

Benchmarks

Startup

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-12-13T16:00:45 2024-12-13T16:07:46
git_branch master evanchooly/codeorigin
git_commit_date 1734102010 1734105188
git_commit_sha bdc9eae 199e8e8
release_version 1.45.0-SNAPSHOT~bdc9eae8ff 1.45.0-SNAPSHOT~199e8e8072
start_time 2024-12-13T16:00:32 2024-12-13T16:07:32
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1734106421 1734106421
ci_job_id 738636508 738636508
ci_pipeline_id 51090856 51090856
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~199e8e8072, baseline=1.45.0-SNAPSHOT~bdc9eae8ff
    dateFormat X
    axisFormat %s
section baseline
no_agent (375.066 µs) : 355, 395
.   : milestone, 375,
iast (497.96 µs) : 476, 519
.   : milestone, 498,
iast_FULL (653.448 µs) : 632, 675
.   : milestone, 653,
iast_GLOBAL (518.388 µs) : 497, 540
.   : milestone, 518,
iast_HARDCODED_SECRET_DISABLED (495.636 µs) : 474, 517
.   : milestone, 496,
iast_INACTIVE (453.329 µs) : 432, 474
.   : milestone, 453,
iast_TELEMETRY_OFF (488.161 µs) : 466, 510
.   : milestone, 488,
tracing (449.576 µs) : 429, 470
.   : milestone, 450,
section candidate
no_agent (371.993 µs) : 352, 392
.   : milestone, 372,
iast (495.62 µs) : 474, 518
.   : milestone, 496,
iast_FULL (658.531 µs) : 637, 680
.   : milestone, 659,
iast_GLOBAL (516.241 µs) : 495, 538
.   : milestone, 516,
iast_HARDCODED_SECRET_DISABLED (490.673 µs) : 469, 512
.   : milestone, 491,
iast_INACTIVE (456.071 µs) : 435, 477
.   : milestone, 456,
iast_TELEMETRY_OFF (484.694 µs) : 463, 506
.   : milestone, 485,
tracing (448.022 µs) : 428, 468
.   : milestone, 448,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 375.066 µs [355.489 µs, 394.642 µs] -
iast 497.96 µs [476.454 µs, 519.467 µs] 122.895 µs (32.8%)
iast_FULL 653.448 µs [631.728 µs, 675.167 µs] 278.382 µs (74.2%)
iast_GLOBAL 518.388 µs [496.869 µs, 539.907 µs] 143.322 µs (38.2%)
iast_HARDCODED_SECRET_DISABLED 495.636 µs [473.889 µs, 517.383 µs] 120.57 µs (32.1%)
iast_INACTIVE 453.329 µs [432.166 µs, 474.492 µs] 78.263 µs (20.9%)
iast_TELEMETRY_OFF 488.161 µs [466.429 µs, 509.892 µs] 113.095 µs (30.2%)
tracing 449.576 µs [429.172 µs, 469.979 µs] 74.51 µs (19.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 371.993 µs [352.305 µs, 391.681 µs] -
iast 495.62 µs [473.508 µs, 517.733 µs] 123.627 µs (33.2%)
iast_FULL 658.531 µs [636.982 µs, 680.08 µs] 286.538 µs (77.0%)
iast_GLOBAL 516.241 µs [494.886 µs, 537.595 µs] 144.247 µs (38.8%)
iast_HARDCODED_SECRET_DISABLED 490.673 µs [469.171 µs, 512.175 µs] 118.68 µs (31.9%)
iast_INACTIVE 456.071 µs [434.684 µs, 477.458 µs] 84.078 µs (22.6%)
iast_TELEMETRY_OFF 484.694 µs [463.306 µs, 506.083 µs] 112.701 µs (30.3%)
tracing 448.022 µs [427.63 µs, 468.414 µs] 76.029 µs (20.4%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~199e8e8072, baseline=1.45.0-SNAPSHOT~bdc9eae8ff
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.371 ms) : 1351, 1391
.   : milestone, 1371,
appsec (1.759 ms) : 1736, 1782
.   : milestone, 1759,
appsec_no_iast (1.737 ms) : 1711, 1762
.   : milestone, 1737,
iast (1.5 ms) : 1478, 1523
.   : milestone, 1500,
profiling (1.552 ms) : 1526, 1577
.   : milestone, 1552,
tracing (1.49 ms) : 1465, 1515
.   : milestone, 1490,
section candidate
no_agent (1.367 ms) : 1348, 1387
.   : milestone, 1367,
appsec (1.755 ms) : 1732, 1778
.   : milestone, 1755,
appsec_no_iast (1.76 ms) : 1735, 1785
.   : milestone, 1760,
iast (1.495 ms) : 1472, 1518
.   : milestone, 1495,
profiling (1.549 ms) : 1525, 1574
.   : milestone, 1549,
tracing (1.501 ms) : 1476, 1525
.   : milestone, 1501,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.371 ms [1.351 ms, 1.391 ms] -
appsec 1.759 ms [1.736 ms, 1.782 ms] 388.195 µs (28.3%)
appsec_no_iast 1.737 ms [1.711 ms, 1.762 ms] 366.007 µs (26.7%)
iast 1.5 ms [1.478 ms, 1.523 ms] 129.262 µs (9.4%)
profiling 1.552 ms [1.526 ms, 1.577 ms] 180.893 µs (13.2%)
tracing 1.49 ms [1.465 ms, 1.515 ms] 119.092 µs (8.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.367 ms [1.348 ms, 1.387 ms] -
appsec 1.755 ms [1.732 ms, 1.778 ms] 387.28 µs (28.3%)
appsec_no_iast 1.76 ms [1.735 ms, 1.785 ms] 392.417 µs (28.7%)
iast 1.495 ms [1.472 ms, 1.518 ms] 127.837 µs (9.3%)
profiling 1.549 ms [1.525 ms, 1.574 ms] 181.698 µs (13.3%)
tracing 1.501 ms [1.476 ms, 1.525 ms] 133.241 µs (9.7%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master evanchooly/codeorigin
git_commit_date 1734102010 1734105188
git_commit_sha bdc9eae 199e8e8
release_version 1.45.0-SNAPSHOT~bdc9eae8ff 1.45.0-SNAPSHOT~199e8e8072
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1734107066 1734107066
ci_job_id 738636509 738636509
ci_pipeline_id 51090856 51090856
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~199e8e8072, baseline=1.45.0-SNAPSHOT~bdc9eae8ff
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.472 ms) : 1460, 1483
.   : milestone, 1472,
appsec (2.336 ms) : 2294, 2378
.   : milestone, 2336,
iast (2.091 ms) : 2038, 2144
.   : milestone, 2091,
iast_GLOBAL (2.141 ms) : 2088, 2194
.   : milestone, 2141,
profiling (1.959 ms) : 1916, 2001
.   : milestone, 1959,
tracing (1.941 ms) : 1900, 1982
.   : milestone, 1941,
section candidate
no_agent (1.467 ms) : 1456, 1479
.   : milestone, 1467,
appsec (2.349 ms) : 2307, 2391
.   : milestone, 2349,
iast (2.092 ms) : 2038, 2145
.   : milestone, 2092,
iast_GLOBAL (2.138 ms) : 2085, 2192
.   : milestone, 2138,
profiling (1.95 ms) : 1908, 1992
.   : milestone, 1950,
tracing (1.926 ms) : 1886, 1967
.   : milestone, 1926,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.472 ms [1.46 ms, 1.483 ms] -
appsec 2.336 ms [2.294 ms, 2.378 ms] 864.0 µs (58.7%)
iast 2.091 ms [2.038 ms, 2.144 ms] 619.309 µs (42.1%)
iast_GLOBAL 2.141 ms [2.088 ms, 2.194 ms] 668.984 µs (45.5%)
profiling 1.959 ms [1.916 ms, 2.001 ms] 486.801 µs (33.1%)
tracing 1.941 ms [1.9 ms, 1.982 ms] 469.065 µs (31.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.467 ms [1.456 ms, 1.479 ms] -
appsec 2.349 ms [2.307 ms, 2.391 ms] 881.474 µs (60.1%)
iast 2.092 ms [2.038 ms, 2.145 ms] 624.098 µs (42.5%)
iast_GLOBAL 2.138 ms [2.085 ms, 2.192 ms] 670.855 µs (45.7%)
profiling 1.95 ms [1.908 ms, 1.992 ms] 482.852 µs (32.9%)
tracing 1.926 ms [1.886 ms, 1.967 ms] 458.907 µs (31.3%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~199e8e8072, baseline=1.45.0-SNAPSHOT~bdc9eae8ff
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.605 s) : 15605000, 15605000
.   : milestone, 15605000,
appsec (15.125 s) : 15125000, 15125000
.   : milestone, 15125000,
iast (18.889 s) : 18889000, 18889000
.   : milestone, 18889000,
iast_GLOBAL (17.856 s) : 17856000, 17856000
.   : milestone, 17856000,
profiling (14.899 s) : 14899000, 14899000
.   : milestone, 14899000,
tracing (15.161 s) : 15161000, 15161000
.   : milestone, 15161000,
section candidate
no_agent (14.97 s) : 14970000, 14970000
.   : milestone, 14970000,
appsec (14.96 s) : 14960000, 14960000
.   : milestone, 14960000,
iast (18.707 s) : 18707000, 18707000
.   : milestone, 18707000,
iast_GLOBAL (17.614 s) : 17614000, 17614000
.   : milestone, 17614000,
profiling (15.551 s) : 15551000, 15551000
.   : milestone, 15551000,
tracing (15.057 s) : 15057000, 15057000
.   : milestone, 15057000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.605 s [15.605 s, 15.605 s] -
appsec 15.125 s [15.125 s, 15.125 s] -480.0 ms (-3.1%)
iast 18.889 s [18.889 s, 18.889 s] 3.284 s (21.0%)
iast_GLOBAL 17.856 s [17.856 s, 17.856 s] 2.251 s (14.4%)
profiling 14.899 s [14.899 s, 14.899 s] -706.0 ms (-4.5%)
tracing 15.161 s [15.161 s, 15.161 s] -444.0 ms (-2.8%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.97 s [14.97 s, 14.97 s] -
appsec 14.96 s [14.96 s, 14.96 s] -10.0 ms (-0.1%)
iast 18.707 s [18.707 s, 18.707 s] 3.737 s (25.0%)
iast_GLOBAL 17.614 s [17.614 s, 17.614 s] 2.644 s (17.7%)
profiling 15.551 s [15.551 s, 15.551 s] 581.0 ms (3.9%)
tracing 15.057 s [15.057 s, 15.057 s] 87.0 ms (0.6%)

@evanchooly evanchooly force-pushed the evanchooly/codeorigin branch from 690026b to c0d6b75 Compare December 12, 2024 20:08
@evanchooly evanchooly force-pushed the evanchooly/codeorigin branch from c0d6b75 to 7ba729f Compare December 12, 2024 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@evanchooly