-
Notifications
You must be signed in to change notification settings - Fork 147
Add support for Microsoft.Azure.Functions.Worker.Core v2 #6472
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing the following branches/commits: Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6472) - mean (68ms) : 65, 72
. : milestone, 68,
master - mean (71ms) : 60, 82
. : milestone, 71,
section CallTarget+Inlining+NGEN
This PR (6472) - mean (976ms) : 951, 1001
. : milestone, 976,
master - mean (981ms) : 956, 1005
. : milestone, 981,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6472) - mean (107ms) : 104, 110
. : milestone, 107,
master - mean (108ms) : 105, 110
. : milestone, 108,
section CallTarget+Inlining+NGEN
This PR (6472) - mean (669ms) : 649, 690
. : milestone, 669,
master - mean (680ms) : 665, 695
. : milestone, 680,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6472) - mean (91ms) : 88, 93
. : milestone, 91,
master - mean (91ms) : 88, 95
. : milestone, 91,
section CallTarget+Inlining+NGEN
This PR (6472) - mean (632ms) : 616, 648
. : milestone, 632,
master - mean (633ms) : 616, 649
. : milestone, 633,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6472) - mean (194ms) : 189, 199
. : milestone, 194,
master - mean (193ms) : 189, 197
. : milestone, 193,
section CallTarget+Inlining+NGEN
This PR (6472) - mean (1,101ms) : 1063, 1138
. : milestone, 1101,
master - mean (1,101ms) : 1075, 1127
. : milestone, 1101,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6472) - mean (279ms) : 274, 284
. : milestone, 279,
master - mean (279ms) : 275, 283
. : milestone, 279,
section CallTarget+Inlining+NGEN
This PR (6472) - mean (868ms) : 838, 897
. : milestone, 868,
master - mean (874ms) : 844, 904
. : milestone, 874,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (6472) - mean (268ms) : 264, 271
. : milestone, 268,
master - mean (266ms) : 262, 271
. : milestone, 266,
section CallTarget+Inlining+NGEN
This PR (6472) - mean (853ms) : 816, 890
. : milestone, 853,
master - mean (856ms) : 821, 891
. : milestone, 856,
|
Datadog ReportBranch report: ✅ 0 Failed, 236983 Passed, 2923 Skipped, 31h 56m 54.47s Total Time |
Snapshots difference summaryThe following differences have been observed in committed snapshots. It is meant to help the reviewer. |
Benchmarks Report for tracer 🐌Benchmarks for #6472 compared to master:
The following thresholds were used for comparing the benchmark speeds:
Allocation changes below 0.5% are ignored. Benchmark detailsBenchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️Raw results
Benchmarks.Trace.SpanBenchmark - Slower
|
Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 | 1.135 | 481.21 | 546.32 |
Benchmark | base/diff | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472 | 1.154 | 961.33 | 832.93 | |
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 | 1.138 | 808.11 | 710.06 |
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | StartFinishSpan |
net6.0 | 399ns | 0.229ns | 0.887ns | 0.00804 | 0 | 0 | 576 B |
master | StartFinishSpan |
netcoreapp3.1 | 610ns | 0.455ns | 1.7ns | 0.00768 | 0 | 0 | 576 B |
master | StartFinishSpan |
net472 | 742ns | 0.301ns | 1.17ns | 0.0915 | 0 | 0 | 578 B |
master | StartFinishScope |
net6.0 | 481ns | 0.279ns | 1.08ns | 0.00991 | 0 | 0 | 696 B |
master | StartFinishScope |
netcoreapp3.1 | 808ns | 0.536ns | 2.08ns | 0.00929 | 0 | 0 | 696 B |
master | StartFinishScope |
net472 | 961ns | 0.532ns | 2.06ns | 0.104 | 0 | 0 | 658 B |
#6472 | StartFinishSpan |
net6.0 | 392ns | 0.184ns | 0.688ns | 0.0081 | 0 | 0 | 576 B |
#6472 | StartFinishSpan |
netcoreapp3.1 | 670ns | 0.521ns | 1.8ns | 0.00761 | 0 | 0 | 576 B |
#6472 | StartFinishSpan |
net472 | 713ns | 0.43ns | 1.67ns | 0.0918 | 0 | 0 | 578 B |
#6472 | StartFinishScope |
net6.0 | 546ns | 0.255ns | 0.953ns | 0.00973 | 0 | 0 | 696 B |
#6472 | StartFinishScope |
netcoreapp3.1 | 710ns | 0.294ns | 1.14ns | 0.00933 | 0 | 0 | 696 B |
#6472 | StartFinishScope |
net472 | 832ns | 0.842ns | 3.26ns | 0.104 | 0 | 0 | 658 B |
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️
Slower ⚠️ in #6472
Benchmark
diff/base
Base Median (ns)
Diff Median (ns)
Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0
1.187
588.32
698.05
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1
1.162
857.63
996.24
Benchmark | diff/base | Base Median (ns) | Diff Median (ns) | Modality |
---|---|---|---|---|
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 | 1.187 | 588.32 | 698.05 | |
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑netcoreapp3.1 | 1.162 | 857.63 | 996.24 |
Raw results
Branch | Method | Toolchain | Mean | StdError | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|---|---|
master | RunOnMethodBegin |
net6.0 | 588ns | 0.358ns | 1.38ns | 0.00986 | 0 | 0 | 696 B |
master | RunOnMethodBegin |
netcoreapp3.1 | 858ns | 0.299ns | 1.12ns | 0.00947 | 0 | 0 | 696 B |
master | RunOnMethodBegin |
net472 | 1.14μs | 0.509ns | 1.97ns | 0.104 | 0 | 0 | 658 B |
#6472 | RunOnMethodBegin |
net6.0 | 698ns | 0.721ns | 2.79ns | 0.00983 | 0 | 0 | 696 B |
#6472 | RunOnMethodBegin |
netcoreapp3.1 | 998ns | 1.16ns | 4.19ns | 0.0095 | 0 | 0 | 696 B |
#6472 | RunOnMethodBegin |
net472 | 1.12μs | 1.11ns | 4.31ns | 0.104 | 0 | 0 | 658 B |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for doing this, I was somewhat dreading it! 😅
One thing I'm not 100% on - are we actually running multi-APIs for Azure Functions apps... I don't think we are, so this basically means we're no longer testing the 1.6.x train...
That's not a reason not to make this PR, and having it in dependabot is still a win. We should just see if we can fix it to run all of them.
As for #6203, I suspect that will become a problem at some point, but it's a pain to fix. It will likely bite us one of these library updates, so we can always punt till then I think...
...s/azure-functions/Samples.AzureFunctions.V4Isolated/Samples.AzureFunctions.V4Isolated.csproj
Outdated
Show resolved
Hide resolved
@andrewlock So would it be easier/quicker to just have two samples for this? One for V1 and one for V2? |
I don't think splitting them provides much value. That wouldn't fix the fact that we want to test with the "latest" version of the package - we'd still need to remember to "manually" update to the latest. But that's essentially a separate requirement - I don't think it should block what you've done here, we will "just" need to update the build to make sure we use multi-build for this sample too, but that can/should be a separate PR IMO. |
a5f2a66
to
eb7b489
Compare
eb7b489
to
ce887d9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
FileStringBuilder.AppendLine(string.Format(EntryFormat, packageVersionEntry.SampleProjectName, packageVersion, properties, requiresDockerDependency)); | ||
// HACK: support Azure Functions as it is in a different location | ||
var testFolder = "integrations"; | ||
if (packageVersionEntry.SampleProjectName.Contains("Samples.AzureFunctions")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gah, good catch 💀
## Summary of changes Attempt to fix flakiness in Azure Functions multi-sample build ## Reason for change #6472 added multi-version building for Azure Functions samples. Unfortunately, this caused the build stage to be flaky, because Azure Functions apparently generates and builds a [completely separate _.csproj_](Azure/azure-functions-dotnet-worker#1252). Unfortunately, it also ignores all the artifact output stuff, which means when we restore/build multiple versions of the app (with different api versions, it stomps over itself. e.g. we can see the sample app build itself doing the right thing: ``` 08:59:05 [DBG] Samples.AzureFunctions.V4Isolated -> /project/artifacts/bin/Samples.AzureFunctions.V4Isolated/release_net6.0_1.6.0/Samples.AzureFunctions.V4Isolated.dll 08:59:06 [DBG] Samples.AzureFunctions.V4Isolated -> /project/artifacts/publish/Samples.AzureFunctions.V4Isolated/release_net6.0_1.6.0/ ``` But there's also the "WorkerExtensions" thing which is going completely off the rails, ignores all of the Directory.Build.props etc and just does its own thing ``` 08:59:07 [DBG] WorkerExtensions -> /tmp/oh4iajvh.kcj/publishout/ 08:59:07 [DBG] WorkerExtensions -> /tmp/zv30zzl2.cxz/publishout/Microsoft.Azure.Functions.Worker.Extensions.dll 08:59:08 [DBG] WorkerExtensions -> /tmp/zv30zzl2.cxz/publishout/ ``` Unfortunately, this shows it's trying to build two separate versions to the same location, and things break. ## Implementation details My initial attempt in #6521 failed. In this attempt I split the v1 SDK and v2 SDK into two separate projects. The SDK versions are tracked separately. I was thinking that as long as we only build a _single_ version per project we _should_ be ok. It was not OK. So in the end I tore out the Azure Functions version tracking. We could/should consider adding it back in some way, but right now this is causing too many issues An alternative is to just remove the Azure Functions samples from the "auto-updating multi version". I kept the split between v1 and v2 of the Azure Functions SDK to give us a _little_ more coverage; I think it's about the best we can do. ## Test coverage If this all finally passes, we should be good. ## Other details An important thing to note is that _currently_, even though we will now get dependabot notifactions about updates to the azure functions libs, these _aren't_ tested automatically. That's because currently our testing on Windows does _not_ run the "multi version" tests like we do on Linux. And we _only_ test Azure functions on Windows. This is something I'd like to address longer term with PRs like #6498, but for the meantime, we'll need to manually keep the samples up to date with the latest SDK version when there's a dependabot PR
Summary of changes
Adds support for
Microsoft.Azure.Functions.Worker.Core
V2.Reason for change
A new major version of it was released and instrumentations stopped at V1.
Implementation details
Test coverage
I relied on CI to run the tests 🙃
Noticed that the snapshots exception message in the span changed unsure why
Other details
I noticed this old PR #6203 I'm unsure if this impacts this?
Fixes DataDog/datadog-aas-extension#295
https://datadoghq.atlassian.net/browse/AIDM-518
https://datadoghq.atlassian.net/browse/AIDM-519