Skip to content

Conversation

martijnvg
Copy link
Member

@martijnvg martijnvg commented Oct 10, 2025

In order to avoid a performance bottleneck when loading values for synthetic source source based block loaders should use source filtering.

This change is mostly mechanical and pushes field names and ignore source format to StoredFieldSpec.

Main changes:

  • Added sourcePaths to StoredFieldsSpec. This allows source loaders to apply a SourceFilter.
  • Update compute engine ShardContext#newSourceLoader(...) method signature to accepts a set of fields to include.
  • Update the EsPhysicalOperationProviders.DefaultShardContext class to create a SourceFilter and push that down to SearchExecutionContext#newSourceLoader(...).
  • Update the BlockSourceReader.SourceBlockLoader#rowStrideStoredFieldSpec(...) implementations to delegate to ValueFetcher#storedFieldsSpec(...) to fetch the stored field specification.
  • Make ArraySourceValueFetcher and SourceValueFetcher return stored field spec with sourcePaths set to its own configured sourcePaths.
  • Introduce marker interface for both ArraySourceValueFetcher and SourceValueFetcher implementations.

The main advantage of delegating to ValueFetcher#storedFieldsSpec(...) is that the source paths are already know there. Only ignored source format needed to be added as field to ArraySourceValueFetcher and SourceValueFetcher. Additionally in a follow up change, the search APIs can now also opt into use source loader to uses a SourceFilter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants