[CI] S3RepositoryAnalysisRestIT testRepositoryAnalysis failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #6483 / oraclelinux-8_platform-support-unix
• elasticsearch-periodic-platform-support #6483 / rhel-9_platform-support-unix
• elasticsearch-periodic #6494 / openjdk17_checkpart2_java-matrix
• elasticsearch-intake #18368 / part2
• elasticsearch-intake #18315 / part2

Reproduction Line:

./gradlew ":x-pack:plugin:snapshot-repo-test-kit:qa:s3:javaRestTest" --tests "org.elasticsearch.repositories.blobstore.testkit.analyze.S3RepositoryAnalysisRestIT.testRepositoryAnalysis" -Dtests.seed=129F849E2DF6C55B -Dtests.locale=en-VU -Dtests.timezone=Pacific/Tarawa -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:40217], URI [/_snapshot/repository/_analyze?blob_count=10&seed=3000866559835271268&max_blob_size=1mb&timeout=120s&concurrency=4], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"amazon_s3_exception","reason":"No such upload (Service: Amazon S3; Status Code: 200; Error Code: NoSuchUpload; Request ID: test-request-id; S3 Extended Request ID: test-host-id; Proxy: null)"}],"type":"repository_verification_exception","reason":"[repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. Typically this happens when using storage other than AWS S3 which incorrectly claims to be S3-compatible. If so, please report this incompatibility to your storage supplier. Do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demons
[truncated]

Issue Reasons:

• [8.x] 2 consecutive failures in test testRepositoryAnalysis
• [8.x] 5 failures in test testRepositoryAnalysis (2.9% fail rate in 174 executions)
• [8.x] 2 failures in step part2 (12.5% fail rate in 16 executions)
• [8.x] 2 failures in pipeline elasticsearch-intake (12.5% fail rate in 16 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testRepositoryAnalysis (1.6% fail rate in 190 executions)
• [main] 2 failures in step part-2 (3.2% fail rate in 63 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (3.1% fail rate in 64 executions)

Build Scans:

• elasticsearch-intake #18314 / part2
• elasticsearch-pull-request #57690 / part-2
• elasticsearch-pull-request #57691 / part-2

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[ywangd]

Marking this low-risk since the test failure is with the s3 fixture.

[ywangd]

I think this is likely caused by the recent sdk upgrade #122431 which may introduced a new behaviour for translating the server side error for completeMultipartUpload request. It was previoulsy status code 0 (#116212) and now it is 200.

[ywangd]

This is an actual issue due to behaviour change in the upgrade SDK. I raised #122815 for a fix. Leaving it to low-risk since the code with upgraded SDK is not relased.