[CI] HeapAttackIT testTooManyEval failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #6511 / oraclelinux-7_platform-support-unix
• elasticsearch-periodic-platform-support #6469 / ubuntu-2204_platform-support-unix

Reproduction Line:

./gradlew ":test:external-modules:test-esql-heap-attack:javaRestTest" --tests "org.elasticsearch.xpack.esql.heap_attack.HeapAttackIT.testTooManyEval" -Dtests.seed=2621B4298B0075C4 -Dtests.locale=lag-Latn-TZ -Dtests.timezone=Australia/Melbourne -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.junit.TestCouldNotBeSkippedException: Test could not be skipped due to other failures

Issue Reasons:

• [8.x] 2 failures in test testTooManyEval (0.8% fail rate in 242 executions)
• [8.x] 2 failures in pipeline elasticsearch-periodic-platform-support (28.6% fail rate in 7 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[astefan]

I think this one is the same as #123432

[2025-02-20T18:02:18,256][INFO ][o.e.m.j.JvmGcMonitorService] [test-cluster-1] [gc][1527] overhead, spent [394ms] collecting in the last [1s]
[2025-02-20T18:02:18,345][INFO ][o.e.x.e.h.HeapAttackIT   ] [testEnrichExplosionManyMatches] --> test testEnrichExplosionManyMatches triggering OOM after 5m
[2025-02-20T18:02:18,349][ERROR][o.e.t.e.h.RestTriggerOutOfMemoryAction] [test-cluster-1] triggering out of memory
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1740031066411300062/elastic/elasticsearch-periodic-platform-support/test/external-modules/esql-heap-attack/build/testrun/javaRestTest/temp/test-cluster18305841853369123460/test-cluster-1/logs/java_pid360531.hprof ...
Heap dump file created [555011853 bytes in 1.021 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space
[2025-02-20T18:02:20,428][INFO ][o.e.c.c.Coordinator      ] [test-cluster-0] master node [{test-cluster-1}{MfaRBkPQSuWmbxHi-EaFiQ}{e3bE0eY7QgascFL6mzODeg}{test-cluster-1}{127.0.0.1}{127.0.0.1:45047}{cdfhilmrstw}{8.19.0}{7000099-8526000}] disconnected, restarting discovery [[test-cluster-1][127.0.0.1:45047][internal:coordination/fault_detection/leader_check] disconnected]
[2025-02-20T18:02:20,428][INFO ][o.e.t.ClusterConnectionManager] [test-cluster-0] transport connection to [{test-cluster-1}{MfaRBkPQSuWmbxHi-EaFiQ}{e3bE0eY7QgascFL6mzODeg}{test-cluster-1}{127.0.0.1}{127.0.0.1:45047}{cdfhilmrstw}{8.19.0}{7000099-8526000}] closed by remote; if unexpected, see [https://www.elastic.co/guide/en/elasticsearch/reference/master/troubleshooting-unstable-cluster.html#troubleshooting-unstable-cluster-network] for troubleshooting guidance
[2025-02-20T18:02:20,436][INFO ][o.e.c.s.ClusterApplierService] [test-cluster-0] master node changed {previous [{test-cluster-1}{MfaRBkPQSuWmbxHi-EaFiQ}{e3bE0eY7QgascFL6mzODeg}{test-cluster-1}{127.0.0.1}{127.0.0.1:45047}{cdfhilmrstw}{8.19.0}{7000099-8526000}], current []}, term: 2, version: 482, reason: becoming candidate: onLeaderFailure
[2025-02-20T18:02:20,453][ERROR][o.e.t.e.h.RestTriggerOutOfMemoryAction] [test-cluster-0] triggering out of memory
[2025-02-20T18:02:20,462][WARN ][o.e.x.e.a.EsqlResponseListener] [test-cluster-0] ESQL request failed with status [BAD_GATEWAY]: org.elasticsearch.transport.NodeDisconnectedException: [test-cluster-1][127.0.0.1:45047][internal:data/read/esql/exchange] disconnected
	Suppressed: org.elasticsearch.transport.NodeDisconnectedException: [test-cluster-1][127.0.0.1:45047][internal:data/read/esql/exchange] disconnected
	Suppressed: org.elasticsearch.transport.NodeDisconnectedException: [test-cluster-1][127.0.0.1:45047][internal:data/read/esql/exchange] disconnected
	Suppressed: org.elasticsearch.transport.NodeDisconnectedException: [test-cluster-1][127.0.0.1:45047][indices:data/read/esql/data] disconnected


....

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1740031066411300062/elastic/elasticsearch-periodic-platform-support/test/external-modules/esql-heap-attack/build/testrun/javaRestTest/temp/test-cluster18305841853369123460/test-cluster-0/logs/java_pid360526.hprof ...
Heap dump file created [400844821 bytes in 1.252 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space
[2025-02-20T18:02:22,560][INFO ][o.e.x.e.h.HeapAttackIT   ] [testEnrichExplosionManyMatches] after test
[2025-02-20T18:02:22,573][INFO ][o.e.x.e.h.HeapAttackIT   ] [testTooManyEval] before test
[2025-02-20T18:02:22,575][INFO ][o.e.x.e.h.HeapAttackIT   ] [testTooManyEval] after test

[idegtiarenko]

#124300 and corresponding 8.x backport should help to stabilize this test.