Use skiplist for auto date aggreagation #20057
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ing sub aggregation Signed-off-by: Ankit Jain <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
Signed-off-by: Asim Mahmood <[email protected]>
Signed-off-by: Asim Mahmood <[email protected]>
Signed-off-by: Asim Mahmood <[email protected]>
…ent idea oh how to seperately filter rewrite path vs regular path. Signed-off-by: Asim Mahmood <[email protected]>
Signed-off-by: Asim Mahmood <[email protected]>
Signed-off-by: Asim Mahmood <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
Signed-off-by: Ankit Jain <[email protected]>
…gRangeCollector can call it out. Signed-off-by: Asim Mahmood <[email protected]>
Signed-off-by: Asim Mahmood <[email protected]>
Signed-off-by: Asim Mahmood <[email protected]>
Signed-off-by: Asim Mahmood <[email protected]>
Contributor
|
❌ Gradle check result for 27f8248: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Contributor
Author
|
Functionally correct but not showing improvement QueryResult |
Contributor
Author
Contributor
|
@asimmahmood1 - Have you verified that the regression is only with |
…by filter rewrite * TODO: * add more test cases, currently does not assert that skiplist was used * add auto date benchmark to http_logs * verify existing benchmarks have no issue Signed-off-by: Asim Mahmood <[email protected]>
Contributor
Author
|
So I figured out why performance was not up to par. Short answer is when auto date moves onto large intervals, we need to track of not just preparedRounding but also bucketOrds. If preparedRounding has changed since last time, we need to restart the skiplist logic. Otherwise, we'll collect too many docs, and although the end result doesn't change (i.e. unit test passes), performance is too low. Auto date histo has two modes: FromSingle and FromMany. FromSingle is often used in top level aggregation, so It's similar to Date Histogram where parent is null. FromMany is used e.g in big5's See validation below. Note: will update this PR after #19573 is merged. |
Contributor
Author
range auto date: from 1335 to 139 (89%) |
Contributor
Author
range-auto-date-with-metrics (22% lower)this is similar to the date-with-metrics, since the time is bounded by tavg stat |
Signed-off-by: Asim Mahmood <[email protected]>
Contributor
|
❌ Gradle check result for 4d7f22d: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
github-project-automation
bot
moved this from Done
to In Progress
in Performance Roadmap
Contributor
Author
|
{"run-benchmark-test": "id_3"} |
Contributor
Author
|
{"run-benchmark-test": "id_11"} |
This was referenced Nov 24, 2025
Contributor
|
❌ Gradle check result for 4d7f22d: Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Contributor
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/5174/ . Final results will be published once the job is completed. |
Collaborator
Benchmark ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/5174/
|
Collaborator
Benchmark Baseline Comparison ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/210/
|
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This is a built on top #19573, once that is merged will update this PR.
Related Issues
Resolves #19827
Part of #18882
Also #19384
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.