[CI] ConcurrentSeqNoVersioningIT testSeqNoCASLinearizability failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #11294 / oraclelinux-8_platform-support-unix
• elasticsearch-periodic-platform-support #11284 / rhel-8_platform-support-unix
• elasticsearch-periodic-platform-support #11279 / rhel-8_platform-support-unix
• elasticsearch-periodic #11269 / concurrent-search-tests
• elasticsearch-periodic #11264 / single-processor-node-tests
• elasticsearch-periodic-platform-support #11217 / oraclelinux-9_platform-support-unix

Reproduction Line:

./gradlew ":server:internalClusterTest" --tests "org.elasticsearch.versioning.ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability" -Dtests.seed=CA4A191CB6E201CD -Dtests.locale=en-AS -Dtests.timezone=Africa/Monrovia -Druntime.java=25

Applicable branches:
9.2

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: primary term does not match, luceneOp=[Index{id='ID:0', seqNo=245, primaryTerm=1, version=22, autoGeneratedIdTimestamp=-1}], translogOp=[Index{id='ID:8', seqNo=245, primaryTerm=3, version=28, autoGeneratedIdTimestamp=-1}]
Expected: <3L>
     but: was <1L>

Issue Reasons:

• [9.2] 6 failures in test testSeqNoCASLinearizability (1.5% fail rate in 392 executions)
• [9.2] 2 failures in step rhel-8_platform-support-unix (13.3% fail rate in 15 executions)
• [9.2] 4 failures in pipeline elasticsearch-periodic-platform-support (25.0% fail rate in 16 executions)
• [9.2] 2 failures in pipeline elasticsearch-periodic (13.3% fail rate in 15 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testSeqNoCASLinearizability (0.2% fail rate in 825 executions)

Build Scans:

• elasticsearch-periodic-platform-support #4946 / rhel-9_platform-support-unix
• elasticsearch-pull-request #41604 / part-1

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[henningandersen]

Might relate to #117024 they seem to have started failing at around the same time (16th/17th).

[ankikuma]

I will need to investigate this further but definitely does not look low-risk

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 3 failures in test testSeqNoCASLinearizability (1.8% fail rate in 169 executions)
• [8.x] 2 failures in pipeline elasticsearch-periodic-platform-support (50.0% fail rate in 4 executions)

Build Scans:

• elasticsearch-intake #14877 / part1
• elasticsearch-periodic-platform-support #5451 / almalinux-8-aarch64_checkpart1_platform-support-arm
• elasticsearch-periodic-platform-support #5441 / ubuntu-2204_platform-support-unix