Possible performance regression in 9.1.x (TSDS)

[sherman]

Hi, guys!

On a multi-node cluster, indexing of TSDS (otel-format metrics) is stable on 9.0.x.
After upgrading to 9.1.4, the indexing rate drops and the write queue grows, making the cluster unstable.

Profiling shows significant time spent in dv.writeField (see pic. 1)
Disabling the new 9.1.x optimization with:

-Dorg.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesConsumer.enableOptimizedMerge=false

restores normal performance.

Can you help figure out the root cause?

(using java 21)

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[kkrik-es]

Seems similar to #132597 that should be fixed in v.9.1.2. @martijnvg @dnhatn fyi.

[sherman]

@kkrik-es, hi!

I’m using 9.1.4 (flamegraph from it), which already includes this patch.

[martijnvg]

Thanks @sherman for reporting this. Looks like closing a temporary index output is much slower than expected. We will take a look at get back to you.

Looking at the DISIAccumulator, looks we close twice, while only once should be needed. I don't think this explains the regression, but that should also be addressed.