Alternate approach to speed up ignored_source access #133839
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jordan-powers
added
>non-issue
Team:StorageEngine
:StorageEngine/Mapping
|
Here are the results of running the INSIST_🐔 benchmark comparing against the current main (with the feature flag enabled, splitting up Summary:
Looks like the new approach is comparable for limit_500 chicken_3, and chicken_3_with_where, slightly worse for chicken_1, and much worse for chicken_2 and chicken_4 |
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
elasticsearchmachine
added
the
serverless-linked
martijnvg
approved these changes
Sep 5, 2025
| } | ||
|
|
||
| return permissions.getFieldPermissions().filter(wrappedReader); | ||
| var indexVersionCreated = searchExecutionContextProvider.apply(shardId).indexVersionCreated(); |
There was a problem hiding this comment.
Passing down index version here is a little intrusive. However I don't think it is too bad. In other places of the code base passing down index version is a common pattern.
The alternative we talked about, which would store coalesced ignored source in a different field, would also be an intrusive change.
So I'm okay with this approach.
…d-fieldinfo-explosion
This was referenced Sep 8, 2025
jordan-powers
added a commit
that referenced
this pull request
Sep 26, 2025
Follow-up to #133839 to remove the feature flag and enable the feature in production.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In #132142 and #132428 we split up ignored_source entries into distinct lucene fields, then added an optimized field visitor to speed up retrieving unmapped values for INSIST_🐔.
However, since this approach creates a unique lucene field for every ignored_source entry, we can very quickly have a lot of lucene fields if there are a lot of unique unmapped fields per document. This can cause significant slowdowns in indexing throughput and merge time.
This PR addresses those limitations by reverting back to keeping all ignored_source entries under the same lucene field. However, we still keep some of the speedups from that prior work by continuing to coalesce multiple ignored_source entries for the same field into a single entry, allowing the field visitor to exit early.
Unfortunately, we do lose some time compared to the original optimizations because now the field visitor cannot look at the fieldInfo to decide whether or not to visit a field, and it instead needs to actually visit and materialize each ignored_source entry before it can decide whether or not to keep it.