[CI] HeapAttackIT testFetchTooManyMvLongs failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #6724 / concurrent-search-tests
• elasticsearch-periodic-platform-support #6616 / debian-11_platform-support-unix

Reproduction Line:

./gradlew ":test:external-modules:test-esql-heap-attack:javaRestTest" --tests "org.elasticsearch.xpack.esql.heap_attack.HeapAttackIT.testHugeConcat" -Dtests.seed=FA1A265012D38730 -Dtests.jvm.argline="-Des.concurrent_search=true" -Dtests.locale=es-EC -Dtests.timezone=SystemV/PST8 -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.junit.TestCouldNotBeSkippedException: Test could not be skipped due to other failures

Issue Reasons:

• [8.x] 2 failures in test testFetchTooManyMvLongs (0.4% fail rate in 549 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[nik9000]

This looks to be a legitimate failure of testEnrichExplosionManyMatches:

2025-03-01T14:58:47,131][WARN ][o.e.i.b.request          ] [test-cluster-1] [request] New used memory 322126047 [307.2mb] for data of [<<enrich-result>>] would be larger than configured breaker: 322122547 [307.1mb], breaking
[2025-03-01T15:00:45,104][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1577] overhead, spent [261ms] collecting in the last [1s]
[2025-03-01T15:00:47,111][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1579] overhead, spent [266ms] collecting in the last [1s]
[2025-03-01T15:00:50,126][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1582] overhead, spent [282ms] collecting in the last [1s]
[2025-03-01T15:00:51,129][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1583] overhead, spent [252ms] collecting in the last [1s]
[2025-03-01T15:00:52,131][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1584] overhead, spent [285ms] collecting in the last [1s]
[2025-03-01T15:00:53,152][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1585] overhead, spent [291ms] collecting in the last [1s]
[2025-03-01T15:00:54,161][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1586] overhead, spent [268ms] collecting in the last [1s]
[2025-03-01T15:00:55,176][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1587] overhead, spent [257ms] collecting in the last [1s]
[2025-03-01T15:00:56,178][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1588] overhead, spent [259ms] collecting in the last [1s]
[2025-03-01T15:00:57,181][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1589] overhead, spent [258ms] collecting in the last [1s]
[2025-03-01T15:00:58,200][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1590] overhead, spent [251ms] collecting in the last [1s]
[2025-03-01T15:00:59,204][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1591] overhead, spent [277ms] collecting in the last [1s]
[2025-03-01T15:01:00,215][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1592] overhead, spent [274ms] collecting in the last [1s]
[2025-03-01T15:01:01,220][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1593] overhead, spent [267ms] collecting in the last [1s]
[2025-03-01T15:01:02,224][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1594] overhead, spent [286ms] collecting in the last [1s]
[2025-03-01T15:01:03,226][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1595] overhead, spent [253ms] collecting in the last [1s]
[2025-03-01T15:01:04,237][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1596] overhead, spent [276ms] collecting in the last [1s]
[2025-03-01T15:01:05,239][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1597] overhead, spent [289ms] collecting in the last [1s]
[2025-03-01T15:01:07,249][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1599] overhead, spent [261ms] collecting in the last [1s]
[2025-03-01T15:01:08,251][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1600] overhead, spent [251ms] collecting in the last [1s]
[2025-03-01T15:01:09,254][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1601] overhead, spent [281ms] collecting in the last [1s]
[2025-03-01T15:01:10,269][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1602] overhead, spent [272ms] collecting in the last [1s]
[2025-03-01T15:01:11,274][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1603] overhead, spent [256ms] collecting in the last [1s]
[2025-03-01T15:01:13,280][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1605] overhead, spent [272ms] collecting in the last [1s]
[2025-03-01T15:01:14,283][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1606] overhead, spent [271ms] collecting in the last [1s]
[2025-03-01T15:01:15,294][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1607] overhead, spent [257ms] collecting in the last [1s]
[2025-03-01T15:01:16,347][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1608] overhead, spent [277ms] collecting in the last [1s]
[2025-03-01T15:01:19,353][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-0] [gc][1611] overhead, spent [267ms] collecting in the last [1s]
[2025-03-01T15:01:23,855][INFO ][o.e.x.e.h.HeapAttackIT   ] [testEnrichExplosionManyMatches] --> test testEnrichExplosionManyMatches triggering OOM after 5m
[2025-03-01T15:01:23,885][ERROR][o.e.t.e.h.RestTriggerOutOfMemoryAction] [test-cluster-0] triggering out of memory

I'm going to dig through the heap dump.

[nik9000]

It's a legitimate failure of testEnrichExplosionManyMatches. It looks like the thing isn't let go. And it doesn't reproduce for me. I'll add it to a backlog to look through.

[idegtiarenko]

@nik9000, if you are planning to look into this issue, is there a chance you could also take a look on following aspects:

• HeapAttackIT relies on default shard settings (1 primary and 1 replica). Since CircuitBreaker is not fatal error, we might run query against one shard and if that fails with CircuitBreaker retry it against another one. While this is a nice feature for production use-case, it makes this test slower (when initializing index and when retrying) as well as making it a bit more resilient then we need here.

• HeapAttackIT reuses indices without cleaning them. For example initMvLongsIndex is currently populated 4 times. This means each next test sees more and more data and might implicitly depend on it and on the order of test execution. This makes it trickier to reproduce test failure when running test cases separately.

?

[nik9000]

• HeapAttackIT relies on default shard settings (1 primary and 1 replica). Since CircuitBreaker is not fatal error, we might run query against one shard and if that fails with CircuitBreaker retry it against another one. While this is a nice feature for production use-case, it makes this test slower (when initializing index and when retrying) as well as making it a bit more resilient then we need here.

Interesting! I don't think we should retry circuit breaker errors. If it fails once it's likely to fail again.

• HeapAttackIT reuses indices without cleaning them. For example initMvLongsIndex is currently populated 4 times. This means each next test sees more and more data and might implicitly depend on it and on the order of test execution. This makes it trickier to reproduce test failure when running test cases separately.

That really wasn't my intent. I had thought we were cleaning them. That'd account for some fun failures I wasn't expecting.