-
Notifications
You must be signed in to change notification settings - Fork 25.6k
BlockSourceReader should apply source filtering #136438
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
BlockSourceReader should apply source filtering #136438
Conversation
Set<String> requiredStoredFields, | ||
IgnoredFieldsSpec ignoredFieldsSpec | ||
IgnoredFieldsSpec ignoredFieldsSpec, | ||
Set<String> sourcePaths |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is overlap with IgnoredFieldsSpec.requiredIgnoredFields
and sourcePaths
and could be unified. But IgnoredFieldsSpec
is only used with synthetic source and this change also works for stored source.
In a followup we should try to merge IgnoredFieldsSpec ignoredFieldsSpec
with StoredFieldSpec
. Not doing this in this PR now, as it would make this PR larger.
In order to avoid a performance bottleneck when loading values for synthetic source source based block loaders should use source filtering.
This change is mostly mechanical and pushes field names and ignore source format to
StoredFieldSpec
.Main changes:
sourcePaths
toStoredFieldsSpec
. This allows source loaders to apply aSourceFilter
.ShardContext#newSourceLoader(...)
method signature to accepts a set of fields to include.EsPhysicalOperationProviders.DefaultShardContext
class to create aSourceFilter
and push that down toSearchExecutionContext#newSourceLoader(...)
.BlockSourceReader.SourceBlockLoader#rowStrideStoredFieldSpec(...)
implementations to delegate toValueFetcher#storedFieldsSpec(...)
to fetch the stored field specification.ArraySourceValueFetcher
andSourceValueFetcher
implementations.The main advantage of delegating to
ValueFetcher#storedFieldsSpec(...)
is that the source paths are already know there. Only ignored source format needed to be added as field toArraySourceValueFetcher
andSourceValueFetcher
. Additionally in a follow up change, the search APIs can now also opt into use source loader to uses aSourceFilter
.