repository: increase test timeout for repository analysis

[schase-es]

The MinIO repository analyze test is failing in builds sometimes because of
a timeout. This change increases it from 2 minutes to 5 minutes.

Closes: #134853

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DiannaHohensee]

@schase-es could you link the test failure(s)?

[schase-es]

Yes -- #134853

[DaveCTurner]

Hmm I suspect this test failure might represents a continuation of the saga in MinIO around concurrent UploadPart and AbortMultipartUpload calls:

• minio/minio#21189
• minio/minio#21377
• minio/minio#21456

The change that closed minio/minio#21456 might not actually have fixed the issue, it might have just made it so that retrying the AbortMultipartUpload call eventually cleans things up. I'd rather we ruled out such issues in MinIO before just blindly increasing the timeout. 2 minutes should already be ample time to complete this test.

[schase-es]

@DaveCTurner thanks for the added context -- I'm realizing I need to do some searching around and git-blaming to look at things like this more. I did need to get docker setup before I could test this locally -- so I was very blind.

The docker image for testing this is the 9/07 release. Most of the tests are around 20 seconds. But the reproduction string is running at about 3'20":

./gradlew ":x-pack:plugin:snapshot-repo-test-kit:qa:minio:javaRestTest" --tests "org.elasticsearch.repositories.blobstore.testkit.analyze.MinioRepositoryAnalysisRestIT.testRepositoryAnalysis" -Dtests.seed=43BF8E5368D66EDC -Dtests.locale=asa-TZ -Dtests.timezone=America/Thunder_Bay -Druntime.java=24

I tried:

• adding back in the settings from the PR you linked: time_to_live and anti_contention_delay
• increasing settings like throttled_delete_retry.delay_increment and throttled_delete_retry.delay_increment
• lowering the cooldown_period to one minute (three minutes and 20 would be perfect for hitting this)
• in S3BlobContainer::get_register, temporarily removing the noBackoff policy for the analysis pathway

I'm still seeing 3'20" with various combinations of these.

Do you have any ideas about what this could be hitting?