-
Notifications
You must be signed in to change notification settings - Fork 312
Configure Circuit breaker in OpenLineage instrumentation #9554
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Configure Circuit breaker in OpenLineage instrumentation #9554
Conversation
5e5d650
to
3face35
Compare
🎯 Code Coverage 🔗 Commit SHA: 3face35 | Docs | Was this helpful? Give us feedback! |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 8 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~3face35265, baseline=1.54.0-SNAPSHOT~de79e08969
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.067 s) : 0, 1066651
Total [baseline] (10.691 s) : 0, 10691058
Agent [candidate] (1.062 s) : 0, 1062438
Total [candidate] (10.64 s) : 0, 10639630
section appsec
Agent [baseline] (1.232 s) : 0, 1231696
Total [baseline] (10.992 s) : 0, 10992446
Agent [candidate] (1.243 s) : 0, 1243127
Total [candidate] (11.037 s) : 0, 11037370
section iast
Agent [baseline] (1.2 s) : 0, 1199956
Total [baseline] (11.088 s) : 0, 11088304
Agent [candidate] (1.19 s) : 0, 1189593
Total [candidate] (11.034 s) : 0, 11033845
section profiling
Agent [baseline] (1.209 s) : 0, 1209479
Total [baseline] (11.02 s) : 0, 11019697
Agent [candidate] (1.21 s) : 0, 1210487
Total [candidate] (11.014 s) : 0, 11013939
gantt
title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~3face35265, baseline=1.54.0-SNAPSHOT~de79e08969
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.452 ms) : 0, 1452
crashtracking [candidate] (1.448 ms) : 0, 1448
BytebuddyAgent [baseline] (736.139 ms) : 0, 736139
BytebuddyAgent [candidate] (733.512 ms) : 0, 733512
GlobalTracer [baseline] (253.639 ms) : 0, 253639
GlobalTracer [candidate] (252.655 ms) : 0, 252655
AppSec [baseline] (30.822 ms) : 0, 30822
AppSec [candidate] (30.568 ms) : 0, 30568
Debugger [baseline] (6.454 ms) : 0, 6454
Debugger [candidate] (6.374 ms) : 0, 6374
Remote Config [baseline] (690.998 µs) : 0, 691
Remote Config [candidate] (693.767 µs) : 0, 694
Telemetry [baseline] (16.342 ms) : 0, 16342
Telemetry [candidate] (16.094 ms) : 0, 16094
section appsec
crashtracking [baseline] (1.45 ms) : 0, 1450
crashtracking [candidate] (1.473 ms) : 0, 1473
BytebuddyAgent [baseline] (754.006 ms) : 0, 754006
BytebuddyAgent [candidate] (762.68 ms) : 0, 762680
GlobalTracer [baseline] (245.032 ms) : 0, 245032
GlobalTracer [candidate] (246.707 ms) : 0, 246707
AppSec [baseline] (171.418 ms) : 0, 171418
AppSec [candidate] (171.704 ms) : 0, 171704
Debugger [baseline] (6.006 ms) : 0, 6006
Debugger [candidate] (6.002 ms) : 0, 6002
Remote Config [baseline] (610.261 µs) : 0, 610
Remote Config [candidate] (624.845 µs) : 0, 625
Telemetry [baseline] (8.433 ms) : 0, 8433
Telemetry [candidate] (8.543 ms) : 0, 8543
IAST [baseline] (23.604 ms) : 0, 23604
IAST [candidate] (23.934 ms) : 0, 23934
section iast
crashtracking [baseline] (1.463 ms) : 0, 1463
crashtracking [candidate] (1.456 ms) : 0, 1456
BytebuddyAgent [baseline] (859.638 ms) : 0, 859638
BytebuddyAgent [candidate] (851.547 ms) : 0, 851547
GlobalTracer [baseline] (247.173 ms) : 0, 247173
GlobalTracer [candidate] (244.573 ms) : 0, 244573
AppSec [baseline] (26.419 ms) : 0, 26419
AppSec [candidate] (27.991 ms) : 0, 27991
Debugger [baseline] (6.068 ms) : 0, 6068
Debugger [candidate] (6.045 ms) : 0, 6045
Remote Config [baseline] (613.38 µs) : 0, 613
Remote Config [candidate] (605.395 µs) : 0, 605
Telemetry [baseline] (8.311 ms) : 0, 8311
Telemetry [candidate] (8.283 ms) : 0, 8283
IAST [baseline] (29.07 ms) : 0, 29070
IAST [candidate] (28.088 ms) : 0, 28088
section profiling
crashtracking [baseline] (1.437 ms) : 0, 1437
crashtracking [candidate] (1.446 ms) : 0, 1446
BytebuddyAgent [baseline] (761.45 ms) : 0, 761450
BytebuddyAgent [candidate] (762.104 ms) : 0, 762104
GlobalTracer [baseline] (233.296 ms) : 0, 233296
GlobalTracer [candidate] (233.296 ms) : 0, 233296
AppSec [baseline] (30.371 ms) : 0, 30371
AppSec [candidate] (31.134 ms) : 0, 31134
Debugger [baseline] (10.58 ms) : 0, 10580
Debugger [candidate] (13.014 ms) : 0, 13014
Remote Config [baseline] (1.545 ms) : 0, 1545
Remote Config [candidate] (716.831 µs) : 0, 717
Telemetry [baseline] (11.917 ms) : 0, 11917
Telemetry [candidate] (9.501 ms) : 0, 9501
ProfilingAgent [baseline] (107.85 ms) : 0, 107850
ProfilingAgent [candidate] (108.023 ms) : 0, 108023
Profiling [baseline] (108.501 ms) : 0, 108501
Profiling [candidate] (108.704 ms) : 0, 108704
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~3face35265, baseline=1.54.0-SNAPSHOT~de79e08969
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.067 s) : 0, 1067080
Total [baseline] (8.641 s) : 0, 8641127
Agent [candidate] (1.06 s) : 0, 1060210
Total [candidate] (8.651 s) : 0, 8650834
section iast
Agent [baseline] (1.193 s) : 0, 1193184
Total [baseline] (9.316 s) : 0, 9316217
Agent [candidate] (1.2 s) : 0, 1199901
Total [candidate] (9.311 s) : 0, 9310630
gantt
title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~3face35265, baseline=1.54.0-SNAPSHOT~de79e08969
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.457 ms) : 0, 1457
crashtracking [candidate] (1.446 ms) : 0, 1446
BytebuddyAgent [baseline] (736.885 ms) : 0, 736885
BytebuddyAgent [candidate] (731.458 ms) : 0, 731458
GlobalTracer [baseline] (253.37 ms) : 0, 253370
GlobalTracer [candidate] (252.141 ms) : 0, 252141
AppSec [baseline] (30.746 ms) : 0, 30746
AppSec [candidate] (30.661 ms) : 0, 30661
Debugger [baseline] (6.451 ms) : 0, 6451
Debugger [candidate] (6.419 ms) : 0, 6419
Remote Config [baseline] (695.636 µs) : 0, 696
Remote Config [candidate] (699.098 µs) : 0, 699
Telemetry [baseline] (16.409 ms) : 0, 16409
Telemetry [candidate] (16.423 ms) : 0, 16423
section iast
crashtracking [baseline] (1.458 ms) : 0, 1458
crashtracking [candidate] (1.466 ms) : 0, 1466
BytebuddyAgent [baseline] (854.291 ms) : 0, 854291
BytebuddyAgent [candidate] (859.716 ms) : 0, 859716
GlobalTracer [baseline] (245.413 ms) : 0, 245413
GlobalTracer [candidate] (247.51 ms) : 0, 247510
AppSec [baseline] (26.395 ms) : 0, 26395
AppSec [candidate] (25.62 ms) : 0, 25620
Debugger [baseline] (6.049 ms) : 0, 6049
Debugger [candidate] (6.075 ms) : 0, 6075
Remote Config [baseline] (589.536 µs) : 0, 590
Remote Config [candidate] (593.072 µs) : 0, 593
Telemetry [baseline] (8.194 ms) : 0, 8194
Telemetry [candidate] (8.196 ms) : 0, 8196
IAST [baseline] (29.753 ms) : 0, 29753
IAST [candidate] (29.632 ms) : 0, 29632
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 2 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~3face35265, baseline=1.54.0-SNAPSHOT~de79e08969
dateFormat X
axisFormat %s
section baseline
no_agent (37.477 ms) : 37173, 37781
. : milestone, 37477,
appsec (49.177 ms) : 48738, 49615
. : milestone, 49177,
code_origins (45.545 ms) : 45162, 45927
. : milestone, 45545,
iast (44.776 ms) : 44386, 45166
. : milestone, 44776,
profiling (48.985 ms) : 48521, 49449
. : milestone, 48985,
tracing (43.088 ms) : 42733, 43443
. : milestone, 43088,
section candidate
no_agent (38.237 ms) : 37930, 38544
. : milestone, 38237,
appsec (48.929 ms) : 48508, 49351
. : milestone, 48929,
code_origins (44.901 ms) : 44509, 45294
. : milestone, 44901,
iast (44.687 ms) : 44293, 45082
. : milestone, 44687,
profiling (50.072 ms) : 49577, 50568
. : milestone, 50072,
tracing (45.154 ms) : 44778, 45529
. : milestone, 45154,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~3face35265, baseline=1.54.0-SNAPSHOT~de79e08969
dateFormat X
axisFormat %s
section baseline
no_agent (4.409 ms) : 4357, 4460
. : milestone, 4409,
iast (9.234 ms) : 9084, 9383
. : milestone, 9234,
iast_FULL (13.861 ms) : 13586, 14135
. : milestone, 13861,
iast_GLOBAL (10.227 ms) : 10047, 10407
. : milestone, 10227,
profiling (9.583 ms) : 9434, 9732
. : milestone, 9583,
tracing (7.63 ms) : 7515, 7745
. : milestone, 7630,
section candidate
no_agent (4.352 ms) : 4300, 4405
. : milestone, 4352,
iast (9.906 ms) : 9738, 10073
. : milestone, 9906,
iast_FULL (14.079 ms) : 13801, 14358
. : milestone, 14079,
iast_GLOBAL (10.298 ms) : 10115, 10482
. : milestone, 10298,
profiling (8.857 ms) : 8720, 8995
. : milestone, 8857,
tracing (7.874 ms) : 7755, 7994
. : milestone, 7874,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~3face35265, baseline=1.54.0-SNAPSHOT~de79e08969
dateFormat X
axisFormat %s
section baseline
no_agent (1.474 ms) : 1463, 1486
. : milestone, 1474,
appsec (3.725 ms) : 3509, 3941
. : milestone, 3725,
iast (2.204 ms) : 2141, 2266
. : milestone, 2204,
iast_GLOBAL (2.242 ms) : 2179, 2305
. : milestone, 2242,
profiling (2.07 ms) : 2018, 2123
. : milestone, 2070,
tracing (2.029 ms) : 1980, 2078
. : milestone, 2029,
section candidate
no_agent (1.474 ms) : 1463, 1486
. : milestone, 1474,
appsec (3.704 ms) : 3486, 3923
. : milestone, 3704,
iast (2.204 ms) : 2141, 2267
. : milestone, 2204,
iast_GLOBAL (2.248 ms) : 2185, 2311
. : milestone, 2248,
profiling (2.457 ms) : 2239, 2675
. : milestone, 2457,
tracing (2.03 ms) : 1980, 2080
. : milestone, 2030,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~3face35265, baseline=1.54.0-SNAPSHOT~de79e08969
dateFormat X
axisFormat %s
section baseline
no_agent (15.053 s) : 15053000, 15053000
. : milestone, 15053000,
appsec (14.612 s) : 14612000, 14612000
. : milestone, 14612000,
iast (18.594 s) : 18594000, 18594000
. : milestone, 18594000,
iast_GLOBAL (17.941 s) : 17941000, 17941000
. : milestone, 17941000,
profiling (15.837 s) : 15837000, 15837000
. : milestone, 15837000,
tracing (14.938 s) : 14938000, 14938000
. : milestone, 14938000,
section candidate
no_agent (15.63 s) : 15630000, 15630000
. : milestone, 15630000,
appsec (14.988 s) : 14988000, 14988000
. : milestone, 14988000,
iast (18.762 s) : 18762000, 18762000
. : milestone, 18762000,
iast_GLOBAL (17.928 s) : 17928000, 17928000
. : milestone, 17928000,
profiling (15.965 s) : 15965000, 15965000
. : milestone, 15965000,
tracing (14.915 s) : 14915000, 14915000
. : milestone, 14915000,
|
What Does This Do
Motivation
Datadog tracer sets up OpenLineage listener once OpenLineage classes are detected on the classpath. Enabling
circuitBreaker
is one of the key OpenLineage flagues to be enabled on prod environment. This assuers OpenLineage code will execute at most configurable amount of time and gets killed if it exceeds time limit. This guarantees OpenLineage code will not affect production Spark jobs.As a drawback, circuit breaker results in lineage gaps and hides problems which makes them even more difficult to solve. That's why it makes sense to provide a flag in tracer to control the behavior. By default, circuit breaker should be enabled. However, the flag allows disabling it on lower environments like
staging
.Additional Notes
Contributor Checklist
type:
and (comp:
orinst:
) labels in addition to any usefull labelsclose
,fix
or any linking keywords when referencing an issue.Use
solves
instead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]