Speed up stored field spec management #122645
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When you load stored fields from lucene you have get all of your ducks in a row first or it'll be really slow. You need to get a list of all of the fields you want so you can ignore the ones that you don't. We do this with the `StoredFieldsSpec` which is immutable and has a `merge` method. It's quite common to merge a few dozen of these specs together to prepare for the fetch phase or for a new segment when loading fields from ESQL. When I was loading thousands of fields in ESQL I noticed that the merge was slowing things down marginally. This skips a big chunk of the merge in the common case that we don't have to load any named stored fields.
nik9000
added
>non-issue
:Search Foundations/Search
elasticsearchmachine
added
the
Team:Search Foundations
|
Pinging @elastic/es-search-foundations (Team:Search Foundations) |
javanna
approved these changes
Feb 15, 2025
Yeah. I'll do both. |
nik9000
added
auto-backport
💚 Backport successful
|
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this pull request
Mar 3, 2025
When you load stored fields from lucene you have get all of your ducks in a row first or it'll be really slow. You need to get a list of all of the fields you want so you can ignore the ones that you don't. We do this with the `StoredFieldsSpec` which is immutable and has a `merge` method. It's quite common to merge a few dozen of these specs together to prepare for the fetch phase or for a new segment when loading fields from ESQL. When I was loading thousands of fields in ESQL I noticed that the merge was slowing things down marginally. This skips a big chunk of the merge in the common case that we don't have to load any named stored fields.
elasticsearchmachine
pushed a commit
that referenced
this pull request
Mar 3, 2025
When you load stored fields from lucene you have get all of your ducks in a row first or it'll be really slow. You need to get a list of all of the fields you want so you can ignore the ones that you don't. We do this with the `StoredFieldsSpec` which is immutable and has a `merge` method. It's quite common to merge a few dozen of these specs together to prepare for the fetch phase or for a new segment when loading fields from ESQL. When I was loading thousands of fields in ESQL I noticed that the merge was slowing things down marginally. This skips a big chunk of the merge in the common case that we don't have to load any named stored fields.
georgewallace
pushed a commit
to georgewallace/elasticsearch
that referenced
this pull request
Mar 11, 2025
When you load stored fields from lucene you have get all of your ducks in a row first or it'll be really slow. You need to get a list of all of the fields you want so you can ignore the ones that you don't. We do this with the `StoredFieldsSpec` which is immutable and has a `merge` method. It's quite common to merge a few dozen of these specs together to prepare for the fetch phase or for a new segment when loading fields from ESQL. When I was loading thousands of fields in ESQL I noticed that the merge was slowing things down marginally. This skips a big chunk of the merge in the common case that we don't have to load any named stored fields.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When you load stored fields from lucene you have get all of your ducks in a row first or it'll be really slow. You need to get a list of all of the fields you want so you can ignore the ones that you don't. We do this with the
StoredFieldsSpecwhich is immutable and has amergemethod. It's quite common to merge a few dozen of these specs together to prepare for the fetch phase or for a new segment when loading fields from ESQL.When I was loading thousands of fields in ESQL I noticed that the merge was slowing things down marginally. This skips a big chunk of the merge in the common case that we don't have to load any named stored fields.