chore(engine): introduce ScanSet node #19524
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ScanSet is a single node which represents a set of targets to scan in a single operation. As part of this change, a TopK can no longer be injected at the scan level. Instead, processing a SORT from the logical plan directly results in a TopK. buildNodeGroup and overlappingShardDescriptors are now unused and have been removed. The scheduler can use ScanSet to manually create smaller per-section tasks.
Set MergePrefetchCount on new engine to slightly speed up query times. Signed-off-by: Robert Fratto <[email protected]>
Now that the physical planner emits TopK and ScanSet, Merge and SortMerge are unused. The executor pipeline for SortMerge is also removed. The executor Merge pipeline remains, as the local implementation of executing ScanSet still uses it.
|
Since this change moves TopK to as close to the end of the pipeline as possible, the correctness tests once again pass as of this PR: https://github.com/grafana/loki/actions/runs/18567553549/job/52933101996 |
|
LGTM |
spiridonov
approved these changes
Oct 16, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
This PR may be easier to review commit-by-commit.
This PR introduces a new physical engine node, ScanSet, which represents a set of targets to scan. Currently, each target is a data object section. In the future, this may be expanded to support chunks as a target.
ScanSet is emitted as part of processing the MAKE_TABLE operation from the logical plan. This comes with some other changes:
The ScanSet node will be split into many DataObjScan tasks from the workflow (#19511), removing it from the resulting per-task physical plan. However, to be able to test physical plans without the workflow, a ScanSet can still be executed. Executing a ScanSet returns the data from each target in sequence, similar to Merge.