Ensure S3Service is STARTED when creating client

It's possible for another component to request a S3 client after the
node has started to shut down, and today the `S3Service` will dutifully
attempt to create a fresh client instance even if it is closed. Such
clients will then leak, resulting in test failures.

With this commit we refuse to create new S3 clients once the service has
started to shut down.

Author: DaveCTurner

modules/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Service.java

@@ -37,6 +37,7 @@
 import org.apache.http.conn.DnsResolver;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
+import org.apache.lucene.store.AlreadyClosedException;
 import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.cluster.coordination.stateless.StoreHeartbeatService;
 import org.elasticsearch.cluster.metadata.RepositoryMetadata;
@@ -176,6 +177,13 @@ public AmazonS3Reference client(RepositoryMetadata repositoryMetadata) {
             if (existing != null && existing.tryIncRef()) {
                 return existing;
             }
+
+            if (lifecycle.started() == false) {
+                // doClose() calls releaseCachedClients() which is also synchronized (this) so if we're STARTED here then the client we
+                // create will definitely not leak on close.
+                throw new AlreadyClosedException("S3Service is in state [" + lifecycle + "]");
+            }
+
             final SdkHttpClient httpClient = buildHttpClient(clientSettings, getCustomDnsResolver());
             Releasable toRelease = httpClient::close;
             try {