Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[apmbench] Add trace only benchmarks #15434

Merged
merged 7 commits into from
Feb 3, 2025
Merged

[apmbench] Add trace only benchmarks #15434

merged 7 commits into from
Feb 3, 2025

Conversation

carsonip
Copy link
Member

@carsonip carsonip commented Jan 28, 2025

Motivation/summary

Existing benchmarks are a mix of logs, metrics and traces (sampled and unsampled transactions). Using these benchmarks will introduce noise in benchmark results to assess TBS performance. It will create strange results where the slower the disk, the higher the throughput, because traces ingestion slows and ingestion of other events speeds up to take up the CPU time spared by slow disk, making benchmark results very hard to interpret.

Create a special benchmark with only sampled traces to accurately benchmark traces. Benchmark files are obtained by e.g. grep -E '^{"span|^{"transaction|^{"meta' apm-nodejs-3.29.0.ndjson > apm-nodejs-traces-3.29.0.ndjson

We may want to revert this in the future to add these files to upstream apm-perf and bump apm-server to use upstream apm-perf e.g. #15379 . It is currently using a very old apm-perf.

Checklist

For functional changes, consider:

  • Is it observable through the addition of either logging or metrics?
  • Is its use being published in telemetry to enable product improvement?
  • Have system tests been added to avoid regression?

How to test these changes

Related issues

Useful to benchmark #15235 and related TBS optimizations

Copy link
Contributor

mergify bot commented Jan 28, 2025

This pull request does not have a backport label. Could you fix it @carsonip? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-7.17 is the label to automatically backport to the 7.17 branch.
  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
  • backport-8.x is the label to automatically backport to the 8.x branch.

Copy link
Contributor

mergify bot commented Jan 28, 2025

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Jan 28, 2025
@carsonip carsonip mentioned this pull request Jan 28, 2025
12 tasks
@carsonip carsonip marked this pull request as ready for review February 3, 2025 14:19
@carsonip carsonip requested a review from a team as a code owner February 3, 2025 14:19
@carsonip carsonip added backport-skip Skip notification from the automated backport with mergify and removed backport-8.x Automated backport to the 8.x branch with mergify labels Feb 3, 2025
@kruskall kruskall merged commit 9ad3be4 into main Feb 3, 2025
16 checks passed
@kruskall kruskall deleted the apmbench-add-trace-only branch February 3, 2025 18:01
@carsonip
Copy link
Member Author

carsonip commented Feb 7, 2025

@Mergifyio backport 9.0 8.x 8.18

Copy link
Contributor

mergify bot commented Feb 7, 2025

backport 9.0 8.x 8.18

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Feb 7, 2025
* Add BenchmarkTracesAgentNodeJS

* Add more agents

* Update comment

* Exclude BenchmarkTraces* from scheduled runs to avoid skewing nightly results

* Use loadgen.NewEventHandler

(cherry picked from commit 9ad3be4)
mergify bot pushed a commit that referenced this pull request Feb 7, 2025
* Add BenchmarkTracesAgentNodeJS

* Add more agents

* Update comment

* Exclude BenchmarkTraces* from scheduled runs to avoid skewing nightly results

* Use loadgen.NewEventHandler

(cherry picked from commit 9ad3be4)
mergify bot pushed a commit that referenced this pull request Feb 7, 2025
* Add BenchmarkTracesAgentNodeJS

* Add more agents

* Update comment

* Exclude BenchmarkTraces* from scheduled runs to avoid skewing nightly results

* Use loadgen.NewEventHandler

(cherry picked from commit 9ad3be4)
mergify bot added a commit that referenced this pull request Feb 7, 2025
* Add BenchmarkTracesAgentNodeJS

* Add more agents

* Update comment

* Exclude BenchmarkTraces* from scheduled runs to avoid skewing nightly results

* Use loadgen.NewEventHandler

(cherry picked from commit 9ad3be4)

Co-authored-by: Carson Ip <[email protected]>
mergify bot added a commit that referenced this pull request Feb 7, 2025
* Add BenchmarkTracesAgentNodeJS

* Add more agents

* Update comment

* Exclude BenchmarkTraces* from scheduled runs to avoid skewing nightly results

* Use loadgen.NewEventHandler

(cherry picked from commit 9ad3be4)

Co-authored-by: Carson Ip <[email protected]>
mergify bot added a commit that referenced this pull request Feb 7, 2025
* Add BenchmarkTracesAgentNodeJS

* Add more agents

* Update comment

* Exclude BenchmarkTraces* from scheduled runs to avoid skewing nightly results

* Use loadgen.NewEventHandler

(cherry picked from commit 9ad3be4)

Co-authored-by: Carson Ip <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-skip Skip notification from the automated backport with mergify
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants