[CI] TransportServiceHandshakeTests testAcceptsMismatchedServerlessBuildHash failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #6468 / openjdk24_entitlements_false_checkpart1_java-matrix
• elasticsearch-periodic #6468 / openjdk24_entitlements_true_checkpart1_java-matrix
• elasticsearch-periodic #6461 / openjdk24_entitlements_true_checkpart1_java-matrix
• elasticsearch-periodic #6461 / openjdk24_entitlements_false_checkpart1_java-matrix
• elasticsearch-periodic #6454 / openjdk24_entitlements_true_checkpart1_java-matrix
• elasticsearch-periodic #6454 / openjdk24_entitlements_false_checkpart1_java-matrix
• elasticsearch-periodic #6447 / openjdk24_entitlements_false_checkpart1_java-matrix
• elasticsearch-periodic #6447 / openjdk24_entitlements_true_checkpart1_java-matrix
• elasticsearch-periodic #6440 / openjdk24_entitlements_true_checkpart1_java-matrix
• elasticsearch-periodic #6440 / openjdk24_entitlements_false_checkpart1_java-matrix

Reproduction Line:

./gradlew ":server:test" --tests "org.elasticsearch.transport.TransportServiceHandshakeTests.testAcceptsMismatchedServerlessBuildHash" -Dtests.seed=B13DEB236424409 -Dtests.jvm.argline="-Des.entitlements.enabled=false" -Dtests.locale=yo-NG -Dtests.timezone=Europe/Jersey -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: safeGet: listener was completed exceptionally

Issue Reasons:

• [main] 6 consecutive failures in test testAcceptsMismatchedServerlessBuildHash
• [main] 19 consecutive failures in step openjdk24_entitlements_false_checkpart1_java-matrix
• [main] 17 consecutive failures in step openjdk24_entitlements_true_checkpart1_java-matrix
• [main] 2 consecutive failures in step openjdk24_checkpart1_java-matrix
• [main] 21 consecutive failures in pipeline elasticsearch-periodic
• [main] 38 failures in test testAcceptsMismatchedServerlessBuildHash (92.7% fail rate in 41 executions)
• [main] 19 failures in step openjdk24_entitlements_false_checkpart1_java-matrix (100.0% fail rate in 19 executions)
• [main] 17 failures in step openjdk24_entitlements_true_checkpart1_java-matrix (100.0% fail rate in 17 executions)
• [main] 2 failures in step openjdk24_checkpart1_java-matrix (100.0% fail rate in 2 executions)
• [main] 21 failures in pipeline elasticsearch-periodic (100.0% fail rate in 21 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[ywangd]

OK. #122332 is meant to fix this. But it did not work because when the system property is set in the test, it is already too late because the static feature flag is already initialized with the false value if there are other tests running before it. There is no pretty way to make this work reliably other than breaking it into its own test job similar to this.

[mhl-b]

:(

Does this test belong here? Should it be in the serverless repo?

[ywangd]

Good question. I think we can look into moving this across as the next step. For the time being, can I get review on #122570 which should fix this as is? Thanks!