[CI] NodeEnvironmentTests testGetBestDowngradeVersion failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #16937 / part1
• elasticsearch-intake #16935 / part1
• elasticsearch-intake #16936 / part1
• elasticsearch-intake #16939 / part1
• elasticsearch-intake #16940 / part1
• elasticsearch-intake #16938 / part1
• elasticsearch-intake #16934 / part1
• elasticsearch-intake #16932 / part1
• elasticsearch-pull-request #54268 / part-1
• elasticsearch-pull-request #54266 / part-1

Reproduction Line:

./gradlew ":server:test" --tests "org.elasticsearch.env.NodeEnvironmentTests.testGetBestDowngradeVersion" -Dtests.seed=3968A5F46FA01596 -Dtests.locale=mzn-IR -Dtests.timezone=America/Adak -Druntime.java=23

Applicable branches:
9.0

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: 
Expected: <8.18.0>
     but: was <8.19.0>

Issue Reasons:

• [9.0] 6 consecutive failures in test testGetBestDowngradeVersion
• [9.0] 15 consecutive failures in step part1
• [9.0] 6 consecutive failures in step part-1
• [9.0] 2 consecutive failures in step checkpart1_release-tests
• [9.0] 15 consecutive failures in pipeline elasticsearch-intake
• [9.0] 6 consecutive failures in pipeline elasticsearch-pull-request
• [9.0] 24 failures in test testGetBestDowngradeVersion (37.5% fail rate in 64 executions)
• [9.0] 15 failures in step part1 (88.2% fail rate in 17 executions)
• [9.0] 6 failures in step part-1 (85.7% fail rate in 7 executions)
• [9.0] 2 failures in step checkpart1_release-tests (66.7% fail rate in 3 executions)
• [9.0] 15 failures in pipeline elasticsearch-intake (88.2% fail rate in 17 executions)
• [9.0] 6 failures in pipeline elasticsearch-pull-request (85.7% fail rate in 7 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 4 consecutive failures in test testGetBestDowngradeVersion
• [main] 2 consecutive failures in step part-1
• [main] 2 consecutive failures in pipeline elasticsearch-pull-request
• [main] 4 failures in test testGetBestDowngradeVersion (1.6% fail rate in 246 executions)
• [main] 2 failures in step part-1 (2.0% fail rate in 99 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (2.0% fail rate in 100 executions)

Build Scans:

• elasticsearch-intake #16889 / part1
• elasticsearch-pull-request #54156 / part-1
• elasticsearch-pull-request #54154 / part-1
• elasticsearch-pull-request #54154 / part-1-fips

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[ldematte]

I think the recent introduction of 8.19/the 8.19 branch may have broken some assumptions @alexey-ivanov-es
Can you take a look at these test to understand it this is just a test issue and needs a minor adjustment, or if this needs more reworking?

[thecoop]

This is also broken on 9.0.

This is because assumptions based on min compat being 'last version of previous minor' do not apply for the planned 8.19 release, where 9.0 min compat will be 8.18. This is breaking tests around this

[thecoop]

#121394 mutes it on 9.0 too