Write prefix partition for tsid in tsdb codec#144617
Open
dnhatn wants to merge 4 commits intoelastic:mainfrom
Open
Write prefix partition for tsid in tsdb codec#144617dnhatn wants to merge 4 commits intoelastic:mainfrom
dnhatn wants to merge 4 commits intoelastic:mainfrom
Conversation
dnhatn
added
:StorageEngine/TSDB
kkrik-es
reviewed
Mar 20, 2026
server/src/main/java/org/elasticsearch/index/codec/tsdb/es819/ES819TSDBDocValuesProducer.java
Show resolved
Hide resolved
kkrik-es
approved these changes
Mar 20, 2026
Contributor
kkrik-es
left a comment
kkrik-es
left a comment
There was a problem hiding this comment.
It'd be nice if @martijnvg can also take a look.
Member
Author
++ we should wait for a review from Martijn! |
kkrik-es
reviewed
Mar 20, 2026
| readNumeric(meta, entry.ordsEntry, numericBlockShift); | ||
| entry.termsDictEntry = new TermsDictEntry(); | ||
| readTermDict(meta, entry.termsDictEntry); | ||
| if (version >= ES819TSDBDocValuesFormat.VERSION_PREFIX_PARTITIONS) { |
Contributor
There was a problem hiding this comment.
Dod we have protection from the feature flag too?
Member
Author
There was a problem hiding this comment.
Yes, it's guarded by
Collaborator
|
Pinging @elastic/es-storage-engine (Team:StorageEngine) |
Member
Author
tsdb-metricsgen-270m
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #143955, which introduced a single-byte metric prefix in the tsid layout.
This PR writes prefix partition metadata for the
_tsidfield. The_tsidfield is grouped by its first 2 bytes - the metric prefix byte (byte-0) plus one random byte (byte-1) - yielding up to 256 partitions per metric. The partition records the starting document for each prefix group, allowing the query engine to slice data so that each slice contains only time-series sharing the same prefix.This enables ESQL to partition work across slices without splitting any individual time-series - a requirement for aggregations like rate. This should reduce memory usage and improve performance compared to time-interval partitioning, which requires multiple queries over fragmented data.
The compute engine is not wired up yet, so no improvements are expected yet, but this change may cause a small regression in indexing throughput and storage overhead, which is expected to be trivial.
Relates #143955