[CI] SearchHitsTests testConcurrentEquals failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #8412 / ubuntu-2404-aarch64_checkpart1_platform-support-arm
• elasticsearch-periodic-platform-support #8388 / ubuntu-2404-aarch64_checkpart1_platform-support-arm
• elasticsearch-periodic-platform-support #8346 / almalinux-8-aarch64_checkpart1_platform-support-arm
• elasticsearch-periodic-platform-support #8334 / ubuntu-2004-aarch64_checkpart1_platform-support-arm
• elasticsearch-periodic-platform-support #8310 / ubuntu-2004-aarch64_checkpart1_platform-support-arm

Reproduction Line:

./gradlew ":server:test" --tests "org.elasticsearch.search.SearchHitsTests.testConcurrentEquals" -Dtests.seed=EB9D31B36D01F752 -Dtests.locale=ast-ES -Dtests.timezone=Asia/Bishkek -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: expected: org.elasticsearch.search.SearchHits<org.elasticsearch.search.SearchHits@3814ebc8> but was: org.elasticsearch.search.SearchHits<org.elasticsearch.search.SearchHits@3814ebc8>

Issue Reasons:

• [main] 5 failures in test testConcurrentEquals (0.6% fail rate in 904 executions)
• [main] 2 failures in step ubuntu-2404-aarch64_checkpart1_platform-support-arm (11.1% fail rate in 18 executions)
• [main] 2 failures in step ubuntu-2004-aarch64_checkpart1_platform-support-arm (11.8% fail rate in 17 executions)
• [main] 5 failures in pipeline elasticsearch-periodic-platform-support (27.8% fail rate in 18 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[cbuescher]

I can repro the same error on "main" with

./gradlew ":server:test" --tests "org.elasticsearch.search.SearchHitsTests.testConcurrentEquals" -Dtests.seed=8FE8D9203AB75971:6C073DA4F1474D7E -Dtests.iters=1000

But it fails only a few times (i.e. 2 times) with -Dtests.iters=1000 so my suspicion is that the failure doesn't have to do with this particular random input data but that its not deterministic. The test here apparently forks several threads and calls Object#equals many times(several hundreds of iterations) in parallel on the input and a copy to shake out weird behavior of "equals" in parallel execution.
I'll do some bisecting and look at the test history to see when this started to happen and will also try to narrow down what's actually different in the objects compared here.