[CI] TimeSeriesLifecycleActionsIT testWaitForSnapshot failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #11507 / openjdk25_checkpart5_java-matrix
• elasticsearch-pull-request #106298 / part-5
• elasticsearch-pull-request #105811 / part-5
• elasticsearch-pull-request #105024 / part-5
• elasticsearch-pull-request #104986 / part-5

Reproduction Line:

./gradlew ":x-pack:plugin:ilm:qa:multi-node:javaRestTest" --tests "org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testWaitForSnapshot" -Dtests.seed=5525928DA661DB2D -Dtests.locale=ak-Latn-GH -Dtests.timezone=Asia/Qyzylorda -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Index:{index=index-hxefxvqcxt, managed=true, policy=policy-DAZuI, index_creation_date_millis=1764164381926, time_since_index_creation=12.1s, lifecycle_date_millis=1764164381926, age=12.1s, age_in_millis=12102, phase=delete, phase_time_millis=1764164393138, action=wait_for_snapshot, action_time_millis=1764164383272, step=wait-for-snapshot, step_time_millis=1764164393138, is_auto_retryable_error=true, failed_step_retry_count=5, previous_step_info={type=illegal_state_exception, reason=configured policy 'kqqfuSsp' not found}, phase_execution={policy=policy-DAZuI, phase_definition={min_age=0ms, actions={wait_for_snapshot={policy=kqqfuSsp}}}, version=1, modified_date_in_millis=1764164382833}, skip=false}
SLM:{}

Issue Reasons:

• [main] 5 failures in test testWaitForSnapshot (0.6% fail rate in 841 executions)
• [main] 4 failures in step part-5 (0.8% fail rate in 497 executions)
• [main] 4 failures in pipeline elasticsearch-pull-request (0.8% fail rate in 511 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 5 failures in test testWaitForSnapshot (0.6% fail rate in 841 executions)
• [main] 4 failures in step part-5 (0.8% fail rate in 497 executions)
• [main] 4 failures in pipeline elasticsearch-pull-request (0.8% fail rate in 511 executions)

Build Scans:

• elasticsearch-periodic #11507 / openjdk25_checkpart5_java-matrix
• elasticsearch-pull-request #106298 / part-5
• elasticsearch-pull-request #105811 / part-5
• elasticsearch-pull-request #105024 / part-5
• elasticsearch-pull-request #104986 / part-5

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

I initially thought this might be related to #133954, which made the searchable snapshot action take longer, but I also see similar failures on branches other than 9.2, so this issue definitely existed before this PR: https://es-delivery-stats.elastic.dev/app/r/s/txOE4.

Posting a full error here for completeness and searchability:

java.lang.AssertionError: Index:{index=index-xydyfpbprv, managed=true, policy=policy-TONZB, index_creation_date_millis=1751893552755, time_since_index_creation=10.57s, lifecycle_date_millis=1751893552755, age=10.57s, phase=delete, phase_time_millis=1751893563242, action=wait_for_snapshot, action_time_millis=1751893553212, step=wait-for-snapshot, step_time_millis=1751893563242, is_auto_retryable_error=true, failed_step_retry_count=5, previous_step_info={type=illegal_state_exception, reason=configured policy 'mUtOPTcxkK' not found}, phase_execution={policy=policy-TONZB, phase_definition={min_age=0ms, actions={wait_for_snapshot={policy=mUtOPTcxkK}}}, version=1, modified_date_in_millis=1751893553059}}	
SLM:{}	
at __randomizedtesting.SeedInfo.seed([B81AE29F3E20C6CD:CA1363EE5CD0DB]:0)	
at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.lambda$assertBusy$62(TimeSeriesLifecycleActionsIT.java:1284)	
•••
at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.assertBusy(TimeSeriesLifecycleActionsIT.java:1274)	
at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testWaitForSnapshot(TimeSeriesLifecycleActionsIT.java:282)	
•••
Caused by: java.lang.AssertionError: 	
Expected: is "wait-for-snapshot"	
     but: was null	
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)	
•••
at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.lambda$testWaitForSnapshot$10(TimeSeriesLifecycleActionsIT.java:285)	
at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.lambda$assertBusy$62(TimeSeriesLifecycleActionsIT.java:1276)	
•••
at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.assertBusy(TimeSeriesLifecycleActionsIT.java:1274)	
at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testWaitForSnapshot(TimeSeriesLifecycleActionsIT.java:282)

Edit: this test doesn't even have anything to do with searchable snapshots, ignore my comment 🙃.

[nielsbauman]

I unassigned myself, as I don't think this is directly caused by my previous work.

[nielsbauman]

Also note that this failure doesn't seem to be unique to testWaitForSnapshot; other tests in this class have similar failures: https://es-delivery-stats.elastic.dev/app/r/s/oseyH.