Fix PolicyStepsRegistry cache concurrency issue

The following order of events was possible:
- An ILM policy update cleared `cachedSteps`
- ILM retrieves the step definition for an index, this populates `cachedSteps` with the outdated policy
- The updated policy is put in `lifecyclePolicyMap`

Any subsequent cache retrievals will see the old step definition.

By clearing `cachedSteps` _after_ we update `lifecyclePolicyMap`, we
ensure eventual consistency between the policy and the cache.

Fixes #118406

Author: nielsbauman

muted-tests.yml

@@ -135,9 +135,6 @@ tests:
 - class: org.elasticsearch.xpack.test.rest.XPackRestIT
   method: test {p0=transform/transforms_unattended/Test unattended put and start}
   issue: https://github.com/elastic/elasticsearch/issues/120019
-- class: org.elasticsearch.xpack.ilm.actions.SearchableSnapshotActionIT
-  method: testUpdatePolicyToAddPhasesYieldsInvalidActionsToBeSkipped
-  issue: https://github.com/elastic/elasticsearch/issues/118406
 - class: org.elasticsearch.xpack.security.QueryableReservedRolesIT
   method: testConfiguredReservedRolesAfterClosingAndOpeningIndex
   issue: https://github.com/elastic/elasticsearch/issues/120127

x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/PolicyStepsRegistry.java

@@ -105,9 +105,6 @@ Map<String, Map<Step.StepKey, Step>> getStepMap() {
     public void update(IndexLifecycleMetadata meta) {
         assert meta != null : "IndexLifecycleMetadata cannot be null when updating the policy steps registry";
 
-        // since the policies (may have) changed, the whole steps cache needs to be thrown out
-        cachedSteps.clear();
-
         DiffableUtils.MapDiff<String, LifecyclePolicyMetadata, Map<String, LifecyclePolicyMetadata>> mapDiff = DiffableUtils.diff(
             lifecyclePolicyMap,
             meta.getPolicyMetadatas(),
@@ -163,6 +160,11 @@ public LifecyclePolicyMetadata read(StreamInput in, String key) {
                 }
             }
         }
+
+        // Since the policies (may have) changed, the whole steps cache needs to be thrown out.
+        // We do this after we update `lifecyclePolicyMap` to ensure `cachedSteps` does not contain outdated data.
+        // This means we may clear up-to-date data, but that's a lot better than the cache containing outdated entries indefinitely.
+        cachedSteps.clear();
     }
 
     /**