[CI] GCSRepositoryAnalysisRestIT testRepositoryAnalysis failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #7312 / third-party_gcs
• elasticsearch-periodic-platform-support #7313 / rhel-8_platform-support-unix
• elasticsearch-pull-request #65096 / part-2

Reproduction Line:

./gradlew ":x-pack:plugin:snapshot-repo-test-kit:qa:gcs:javaRestTest" --tests "org.elasticsearch.repositories.blobstore.testkit.analyze.GCSRepositoryAnalysisRestIT.testRepositoryAnalysis" -Dtests.seed=E956DE65A2C7CA51 -Dtests.locale=en-KE -Dtests.timezone=Asia/Dhaka -Druntime.java=24

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:44177], URI [/_snapshot/repository/_analyze?blob_count=10&seed=5856317475903747225&max_blob_size=10mb&timeout=120s&concurrency=4], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository] register [test-register-contended-CbVb9VARRria-l-j_Xc3qg] should have value [10] but instead had value [OptionalBytesReference[MISSING]]"}],"type":"repository_verification_exception","reason":"[repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. See [https://www.elastic.co/guide/en/elasticsearch/reference/master/repo-analysis-api.html] for further information. Elasticsearch attempted to remove the data it wrote at [temp-analysis-OcWnoI0QSvm-2kJTsnP86A] but may have left some behind. If so, please now remove this data manual
[truncated]

Issue Reasons:

• [8.x] 3 failures in test testRepositoryAnalysis (1.2% fail rate in 246 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testRepositoryAnalysis (1.1% fail rate in 182 executions)

Build Scans:

• elasticsearch-periodic #7244 / openjdk23_entitlements_false_checkpart2_java-matrix
• elasticsearch-intake #20773 / part2

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[nicktindall]

This is a blocker, we shouldn't release with the GCP client in this state. I'm working to fix the issue, but will revert if that can't be done.

[DaveCTurner]

This eventually failed on my CI machine, and with #125748 it logged this line:

[2025-03-27T14:16:40,111][ERROR][o.e.c.b.s.BlobContainerUtils] [test-cluster-0] getRegisterUsingConsistentRead including trailing data (gzip compressed and base64-encoded; for details see https://www.elastic.co/docs/deploy-manage/monitor/logging-configuration/elasticsearch-deprecation-logs?version=master): H4sIAAAAAAAA/2NgwAoYAbb6xtQYAAAA

Decoding that yields the following 24 bytes which suggests maybe data was appended to the blob rather than overwriting?

$ echo 'H4sIAAAAAAAA/2NgwAoYAbb6xtQYAAAA' | base64 -d | gzip -d | xxd
00000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000010: 0000 0000 0000 0001                      ........