Skip to content

Commit 4f506d4

Browse files
authored
Optimize time-series source operator (elastic#127095)
This query against the TSDB track took 50 seconds and was reduced to 19 seconds with this changes. ``` TS tsdb | STATS sum(rate(kubernetes.container.memory.pagefaults)) by bucket(@timestamp, 5minute) ``` This change introduces several optimizations to improve the performance of the time-series source operator: - Split the leaf queue into two: one for `_tsid` and another for `@timestamp`. This avoids repeatedly comparing large `_tsid` values while iterating over a single `_tsid`. - Track the number of emitted documents per segment and use this data to build forward and backward document maps, reducing the need for expensive sorts. - Use ordinal blocks to avoid duplicating the same `_tsid` multiple times
1 parent 53780fe commit 4f506d4

File tree

2 files changed

+268
-171
lines changed

2 files changed

+268
-171
lines changed

x-pack/plugin/esql/compute/src/main/java/org/elasticsearch/compute/data/DocVector.java

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,12 @@ public DocVector(IntVector shards, IntVector segments, IntVector docs, Boolean s
6767
blockFactory().adjustBreaker(BASE_RAM_BYTES_USED);
6868
}
6969

70+
public DocVector(IntVector shards, IntVector segments, IntVector docs, int[] docMapForwards, int[] docMapBackwards) {
71+
this(shards, segments, docs, null);
72+
this.shardSegmentDocMapForwards = docMapForwards;
73+
this.shardSegmentDocMapBackwards = docMapBackwards;
74+
}
75+
7076
public IntVector shards() {
7177
return shards;
7278
}
@@ -209,8 +215,12 @@ protected void swap(int i, int j) {
209215
}
210216
}
211217

218+
public static long sizeOfSegmentDocMap(int positionCount) {
219+
return 2 * (((long) RamUsageEstimator.NUM_BYTES_ARRAY_HEADER) + ((long) Integer.BYTES) * positionCount);
220+
}
221+
212222
private long sizeOfSegmentDocMap() {
213-
return 2 * (((long) RamUsageEstimator.NUM_BYTES_ARRAY_HEADER) + ((long) Integer.BYTES) * shards.getPositionCount());
223+
return sizeOfSegmentDocMap(getPositionCount());
214224
}
215225

216226
@Override

0 commit comments

Comments
 (0)