Prepare tsdb doc values format for merging optimizations. #125933
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The change contains the following changes: * The numDocsWithField field moved from SortedNumericEntry to NumericEntry. Making this statistic always available. * Store jump table after values in writeField(...) Currently it is stored before storing values. This change will allow all the optimizations that are listed in elastic#125403
…denseRankPower metadata properties in the format to be after values metadata.
martijnvg
force-pushed
the
es87_tsdb_doc_values_format_change_1
branch
from
ece3a8f to
7597820
Compare
Collaborator
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
dnhatn
reviewed
Mar 31, 2025
| entry.jumpTableEntryCount = meta.readShort(); | ||
| entry.denseRankPower = meta.readByte(); | ||
| private static void readNumeric(IndexInput meta, NumericEntry entry, int version) throws IOException { | ||
| if (version < ES87TSDBDocValuesFormat.VERSION_TWO) { |
Member
There was a problem hiding this comment.
Since these branches are far away, should we fork a new consumer and producer instead?
dnhatn
approved these changes
Apr 1, 2025
Member
dnhatn
left a comment
dnhatn
left a comment
There was a problem hiding this comment.
Can we move ES87TSDBDocValuesConsumer to tests only and throw UOE in ES87TSDBDocValuesFormat#fieldsConsumer?
LGTM. Thanks Martijn!
5 tasks
elasticsearchmachine
added
the
serverless-linked
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this pull request
Apr 4, 2025
Backporting elastic#125933 to 8.x branch. The change contains the following changes: - The numDocsWithField field moved from SortedNumericEntry to NumericEntry. Making this statistic always available. - Store jump table after values in ES87TSDBDocValuesConsumer#writeField(...). Currently it is stored before storing values. This will allow us later to iterate over the SortedNumericDocValues once. When merging, this is expensive as a merge sort on the fly is being executed. This change will allow all the optimizations that are listed in elastic#125403
martijnvg
added a commit
that referenced
this pull request
Apr 4, 2025
) Backporting #125933 to 8.x branch. The change contains the following changes: - The numDocsWithField field moved from SortedNumericEntry to NumericEntry. Making this statistic always available. - Store jump table after values in ES87TSDBDocValuesConsumer#writeField(...). Currently it is stored before storing values. This will allow us later to iterate over the SortedNumericDocValues once. When merging, this is expensive as a merge sort on the fly is being executed. This change will allow all the optimizations that are listed in #125403
andreidan
pushed a commit
to andreidan/elasticsearch
that referenced
this pull request
Apr 9, 2025
…5933) The change contains the following changes: - The numDocsWithField field moved from SortedNumericEntry to NumericEntry. Making this statistic always available. - Store jump table after values in ES87TSDBDocValuesConsumer#writeField(...). Currently it is stored before storing values. This will allow us later to iterate over the SortedNumericDocValues once. When merging, this is expensive as a merge sort on the fly is being executed. This change will allow all the optimizations that are listed in elastic#125403
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The change contains the following changes:
numDocsWithFieldfield moved fromSortedNumericEntrytoNumericEntry. Making this statistic always available.ES87TSDBDocValuesConsumer#writeField(...). Currently it is stored before storing values. This will allow us later to iterate over theSortedNumericDocValuesonce. When merging, this is expensive as an merge sort on the fly is being executed.This change will allow all the optimizations that are listed in #125403
Note that most of the change is test code.