BlockSourceReader should apply source filtering #136438
Draft
+256
−115
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to avoid a performance bottleneck when loading values for synthetic source source based block loaders should use source filtering.
This change is mostly mechanical and pushes field names and ignore source format to
StoredFieldSpec
.Main changes:
sourcePaths
toStoredFieldsSpec
. This allows source loaders to apply aSourceFilter
.ShardContext#newSourceLoader(...)
method signature to accepts a set of fields to include.EsPhysicalOperationProviders.DefaultShardContext
class to create aSourceFilter
and push that down toSearchExecutionContext#newSourceLoader(...)
.BlockSourceReader.SourceBlockLoader#rowStrideStoredFieldSpec(...)
implementations to delegate toValueFetcher#storedFieldsSpec(...)
to fetch the stored field specification.ArraySourceValueFetcher
andSourceValueFetcher
implementations.The main advantage of delegating to
ValueFetcher#storedFieldsSpec(...)
is that the source paths are already know there. Only ignored source format needed to be added as field toArraySourceValueFetcher
andSourceValueFetcher
. Additionally in a follow up change, the search APIs can now also opt into use source loader to uses aSourceFilter
.