[CI] CrossClusterQueryWithFiltersIT testTimestampFilterFromQuery failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #9329 / rhel-9_platform-support-unix
• elasticsearch-periodic-platform-support #9301 / amazonlinux-2_platform-support-aws
• elasticsearch-intake #25335 / part3

Reproduction Line:

./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.action.CrossClusterQueryWithFiltersIT.testTimestampFilterFromQuery" -Dtests.seed=4B2AE5933E1429B4 -Dtests.locale=fr-SY -Dtests.timezone=Australia/Currie -Druntime.java=24

Applicable branches:
8.19

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: <0>
     but: was <1>

Issue Reasons:

• [8.19] 3 failures in test testTimestampFilterFromQuery (1.2% fail rate in 251 executions)
• [8.19] 2 failures in pipeline elasticsearch-periodic-platform-support (25.0% fail rate in 8 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testTimestampFilterFromQuery (1.1% fail rate in 180 executions)

Build Scans:

• elasticsearch-intake #22302 / part3
• elasticsearch-pull-request #68642 / part-3

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[fang-xing-esql]

@smalyshev Could you help take a look at this test? Thank you!

[smalyshev]

Something weird is happening here - the error basically says one shard is skipped where there shouldn't be any skipped shards, and I am not sure why exactly this happens. Seems to be very rare occurrence and is not reproducible locally. Generally having a skipped shard is no big deal unless it indicates some deeper problem - what worries me I still don't understand where it comes from. Let's see if I get more failures to see if there's any pattern.

[smalyshev]

One failure on 4th - looks like the fail is real, though very rare.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testTimestampFilterFromQuery (0.3% fail rate in 705 executions)

Build Scans:

• elasticsearch-periodic-platform-support #8106 / ubuntu-2004-aarch64_checkpart3_platform-support-arm
• elasticsearch-pull-request #70417 / part-3