Less lenience during snapshot deletion

[DaveCTurner]

Today all the shard-level operations during snapshot deletion will log exceptions on failure, but the deletion process continues regardless. This makes sense in the pre-7.6.0 repository format because the shard-level operations happen after updating the root RepositoryData blob, at which point the deletion cannot really fail. But since 7.6.0 we do the shard-level operations first, in order to obtain the names of all the new BlobStoreIndexShardSnapshots blobs. That means we could choose to be stricter and bail out, failing the deletion process before updating the root RepositoryData blob. IMO there's no great reason to be lenient here, a failure to update the shard-level metadata is surely serious enough to halt the process, and stopping on failure avoids bringing the repository into a state where the shard-level metadata is inconsistent with the root.

Opening this for discussion: should we treat these exceptions more seriously now?

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

We (@elastic/es-distributed) discussed this as a team yesterday and although there were no objections to doing this in principle we decided to look at production logs to see if there's any evidence that we are relying on the lenience today. It looks like we are, and that suggests a bug. We need to address bugs in this area before we can get stricter.

[DaveCTurner]

If we had a way to check repo metadata for integrity (#93735) then we would be picking up these problems even without stricter deletes.

[DaveCTurner]

One common failure we encounter is that the shard-level listing of blobs may time out because of a poorly-optimized implementation of the listing API. This seems particularly common for storage accessed with repository-s3 using S3-like HTTP APIs but without the S3-like scalability needed to support a claim that it is "S3-compatible". This isn't actually fatal to a delete (as long as shard generations are in use) it just means we may leak some blobs. I think we should handle this more explicitly: if we set org.elasticsearch.repositories.blobstore.BlobStoreRepository.SnapshotsDeletion.IndexSnapshotsDeletion.ShardSnapshotsDeletion#originalShardBlobs to Set.of() then we can still update the shard generation blob (and detect other failures). We can also improve the log message here to indicate that we may have leaked some blobs but that the overall deletion is still going ahead.

[DaveCTurner]

One common failure we encounter is that the shard-level listing of blobs may time out because of a poorly-optimized implementation of the listing API.

Similarly the top-level listing of blobs in the repository root, and the listing of index path prefixes, may also time out. These failures are currently fatal to a snapshot deletion, but like with shard-level listing failures the only consequence of not being able to list these things is leaked blobs. The deletion itself can still go ahead (i.e. we can update the repository metadata) regardless.

[DaveCTurner]

We can also improve the log message here to indicate that we may have leaked some blobs but that the overall deletion is still going ahead.

We should also improve the exception message in repository-s3 to indicate that a timed-out call to the ListObjects API is indicative of a not-really-S3-compatible repository implementation and to guide users towards seeking support from their storage supplier.