Skip to content

Conversation

martijnvg
Copy link
Member

@martijnvg martijnvg commented Oct 10, 2025

In order to avoid a performance bottleneck when loading values for synthetic source source based block loaders should use source filtering.

This change is mostly mechanical and pushes field names and ignore source format to StoredFieldSpec.

Main changes:

  • Added sourcePaths to StoredFieldsSpec. This allows source loaders to apply a SourceFilter.
  • Update compute engine ShardContext#newSourceLoader(...) method signature to accepts a set of fields to include.
  • Update the EsPhysicalOperationProviders.DefaultShardContext class to create a SourceFilter and push that down to SearchExecutionContext#newSourceLoader(...).
  • Update the BlockSourceReader.SourceBlockLoader#rowStrideStoredFieldSpec(...) implementations to delegate to ValueFetcher#storedFieldsSpec(...) to fetch the stored field specification.
  • Make ArraySourceValueFetcher and SourceValueFetcher return stored field spec with sourcePaths set to its own configured sourcePaths.
  • Introduce marker interface for both ArraySourceValueFetcher and SourceValueFetcher implementations.

The main advantage of delegating to ValueFetcher#storedFieldsSpec(...) is that the source paths are already know there. Only ignored source format needed to be added as field to ArraySourceValueFetcher and SourceValueFetcher. Additionally in a follow up change, the search APIs can now also opt into use source loader to uses a SourceFilter.

Set<String> requiredStoredFields,
IgnoredFieldsSpec ignoredFieldsSpec
IgnoredFieldsSpec ignoredFieldsSpec,
Set<String> sourcePaths
Copy link
Member Author

@martijnvg martijnvg Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is overlap with IgnoredFieldsSpec.requiredIgnoredFields and sourcePaths and could be unified. But IgnoredFieldsSpec is only used with synthetic source and this change also works for stored source.

In a followup we should try to merge IgnoredFieldsSpec ignoredFieldsSpec with StoredFieldSpec. Not doing this in this PR now, as it would make this PR larger.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants