[CI] XPackRestIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #7265 / openjdk23_entitlements_false_checkpart3_java-matrix
• elasticsearch-periodic #7265 / openjdk21_checkpart3_java-fips-matrix
• elasticsearch-pull-request #65067 / part-3
• elasticsearch-pull-request #65063 / part-3
• elasticsearch-pull-request #65052 / part-3
• elasticsearch-pull-request #65024 / part-3
• elasticsearch-pull-request #65002 / part-3
• elasticsearch-pull-request #64995 / part-3
• elasticsearch-pull-request #64993 / part-3
• elasticsearch-pull-request #64993 / rest-compatibility

Reproduction Line:

./gradlew ":x-pack:plugin:yamlRestTest" --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=transform/transforms_stats/Test get multiple transform stats}" -Dtests.seed=CBA84393FF6F7BEB -Dtests.jvm.argline="-Des.entitlements.enabled=false" -Dtests.locale=dje-Latn-NE -Dtests.timezone=NZ -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 1 active tasks found:
reindex-data-stream[c]         Z5DSWTCvQlmEYMlAX7c3AQ:27890 cluster:32                   persistent 1743081869979 13:24:29 12.9m       127.0.0.1 yamlRestTest-0 Reindexing data stream my-data-stream
 expected:<0> but was:<1>

Issue Reasons:

• [main] 40 failures in class org.elasticsearch.xpack.test.rest.XPackRestIT (5.3% fail rate in 751 executions)
• [main] 6 failures in step rest-compatibility (2.9% fail rate in 206 executions)
• [main] 27 failures in step part-3 (13.9% fail rate in 194 executions)
• [main] 3 failures in step part3 (5.3% fail rate in 57 executions)
• [main] 3 failures in pipeline elasticsearch-periodic (42.9% fail rate in 7 executions)
• [main] 28 failures in pipeline elasticsearch-pull-request (13.1% fail rate in 213 executions)
• [main] 3 failures in pipeline elasticsearch-intake (5.2% fail rate in 58 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 60 failures in class org.elasticsearch.xpack.test.rest.XPackRestIT (6.0% fail rate in 999 executions)
• [main] 2 failures in step part3 (2.6% fail rate in 78 executions)
• [main] 40 failures in step rest-compatibility (12.1% fail rate in 330 executions)
• [main] 2 failures in step rest-compat (2.6% fail rate in 77 executions)
• [main] 16 failures in step part-3 (5.3% fail rate in 300 executions)
• [main] 3 failures in pipeline elasticsearch-intake (3.7% fail rate in 81 executions)
• [main] 51 failures in pipeline elasticsearch-pull-request (15.2% fail rate in 336 executions)

Build Scans:

• elasticsearch-intake #16353 / part3
• elasticsearch-pull-request #52612 / rest-compatibility
• elasticsearch-intake #16356 / rest-compat
• elasticsearch-intake #16353 / rest-compat
• elasticsearch-pull-request #52594 / rest-compatibility
• elasticsearch-pull-request #52588 / rest-compatibility
• elasticsearch-pull-request #52562 / rest-compatibility
• elasticsearch-pull-request #52554 / rest-compatibility
• elasticsearch-pull-request #52545 / rest-compatibility
• elasticsearch-pull-request #52542 / rest-compatibility

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[benwtrent]

Looking at the logs, looks like ML is crashing the yaml test nodes.

The final lines of the log include:

[2025-01-24T08:54:06,548][INFO ][o.e.c.m.MetadataCreateIndexService] [yamlRestTest-0] [.ml-notifications-000002] creating index, cause [auto(bulk api)], templates [.ml-notifications-000002], shards [1]/[1]
[2025-01-24T08:54:06,549][INFO ][o.e.c.r.a.AllocationService] [yamlRestTest-0] updating number_of_replicas to [0] for indices [.ml-notifications-000002]
[2025-01-24T08:54:06,563][WARN ][o.e.x.m.i.l.ModelLoadingService] [yamlRestTest-0] [a-classification-model] failed to load model definition
org.elasticsearch.ResourceNotFoundException: Could not find trained model definition [a-classification-model]
	at org.elasticsearch.xpack.ml.inference.persistence.TrainedModelProvider.lambda$getTrainedModelForInference$16(TrainedModelProvider.java:599) ~[?:?]
	at org.elasticsearch.xpack.ml.inference.persistence.ChunkedTrainedModelRestorer.doSearch(ChunkedTrainedModelRestorer.java:162) ~[?:?]
	at org.elasticsearch.xpack.ml.inference.persistence.ChunkedTrainedModelRestorer.lambda$restoreModelDefinition$0(ChunkedTrainedModelRestorer.java:134) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1575) ~[?:?]
[2025-01-24T08:54:06,566][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [yamlRestTest-0] fatal error in thread [elasticsearch[yamlRestTest-0][ml_utility][T#2]], exiting
java.lang.AssertionError: Used bytes: [-520] must be >= 0
	at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addWithoutBreaking(ChildMemoryCircuitBreaker.java:207) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.xpack.ml.inference.loadingservice.ModelLoadingService.lambda$loadModel$10(ModelLoadingService.java:490) ~[?:?]
	at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:64) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:265) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at org.elasticsearch.xpack.ml.inference.persistence.TrainedModelProvider.lambda$getTrainedModelForInference$16(TrainedModelProvider.java:601) ~[?:?]
	at org.elasticsearch.xpack.ml.inference.persistence.ChunkedTrainedModelRestorer.doSearch(ChunkedTrainedModelRestorer.java:162) ~[?:?]
	at org.elasticsearch.xpack.ml.inference.persistence.ChunkedTrainedModelRestorer.lambda$restoreModelDefinition$0(ChunkedTrainedModelRestorer.java:134) ~[?:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977) ~[elasticsearch-9.0.0-SNAPSHOT.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1575) ~[?:?]