Make CacheService.get() throws AlreadyClosedException when service is stopped

This is caught thanks to #121210: if shard files are verified/checksumed
while the node is stopping, an IllegalStateException is throw by
 CacheService.get() when it attempts to read data from the cache. This
 exception later caused the verification to fail and then the Lucene
 index to be marked as corrupted (which nows fails for searchable
 snapshots shards that are read-only and should not be corrupted at
 all).

 This pull request changes ensureLifecycleStarted(), which is called
 during CacheService.get(), to throw an AlreadyClosedException when
 the service is stopped (note that ACE extends IllegalStateException,
 which is convenient here). This ACE will be later specially handlded
 in the checksumIndex method to not mark the shard as corrupted (see
  #121210).

Closes #121927

Author: tlrx

muted-tests.yml

@@ -380,9 +380,6 @@ tests:
   issue: https://github.com/elastic/elasticsearch/issues/121867
 - class: org.elasticsearch.xpack.searchablesnapshots.minio.MinioSearchableSnapshotsIT
   issue: https://github.com/elastic/elasticsearch/issues/121882
-- class: org.elasticsearch.xpack.searchablesnapshots.FrozenSearchableSnapshotsIntegTests
-  method: testCreateAndRestorePartialSearchableSnapshot
-  issue: https://github.com/elastic/elasticsearch/issues/121927
 - class: org.elasticsearch.analysis.common.CommonAnalysisClientYamlTestSuiteIT
   method: test {yaml=analysis-common/40_token_filters/stemmer_override file access}
   issue: https://github.com/elastic/elasticsearch/issues/121625

x-pack/plugin/searchable-snapshots/src/main/java/org/elasticsearch/xpack/searchablesnapshots/cache/full/CacheService.java

@@ -245,7 +245,11 @@ private void ensureLifecycleStarted() {
         final Lifecycle.State state = lifecycleState();
         assert state != Lifecycle.State.INITIALIZED : state;
         if (state != Lifecycle.State.STARTED) {
-            throw new IllegalStateException("Failed to read data from cache: cache service is not started [" + state + "]");
+            if (state == Lifecycle.State.STOPPED) {
+                throw new AlreadyClosedException("Failed to read data from cache: cache service is stopped");
+            } else {
+                throw new IllegalStateException("Failed to read data from cache: cache service is not started [" + state + "]");
+            }
         }
     }