Fix PolicyStepsRegistry cache concurrency issue (#126840) (#126913)

The following order of events was possible:
- An ILM policy update cleared `cachedSteps`
- ILM retrieves the step definition for an index, this populates `cachedSteps` with the outdated policy
- The updated policy is put in `lifecyclePolicyMap`

Any subsequent cache retrievals will see the old step definition.

By clearing `cachedSteps` _after_ we update `lifecyclePolicyMap`, we
ensure eventual consistency between the policy and the cache.

Fixes #118406

(cherry picked from commit 5383f0f)

# Conflicts:
#	muted-tests.yml

Author: nielsbauman

docs/changelog/126840.yaml

@@ -0,0 +1,6 @@
+pr: 126840
+summary: Fix `PolicyStepsRegistry` cache concurrency issue
+area: ILM+SLM
+type: bug
+issues:
+ - 118406

x-pack/plugin/ilm/src/main/java/org/elasticsearch/xpack/ilm/PolicyStepsRegistry.java

@@ -105,9 +105,6 @@ Map<String, Map<Step.StepKey, Step>> getStepMap() {
     public void update(IndexLifecycleMetadata meta) {
         assert meta != null : "IndexLifecycleMetadata cannot be null when updating the policy steps registry";
 
-        // since the policies (may have) changed, the whole steps cache needs to be thrown out
-        cachedSteps.clear();
-
         DiffableUtils.MapDiff<String, LifecyclePolicyMetadata, Map<String, LifecyclePolicyMetadata>> mapDiff = DiffableUtils.diff(
             lifecyclePolicyMap,
             meta.getPolicyMetadatas(),
@@ -163,6 +160,11 @@ public LifecyclePolicyMetadata read(StreamInput in, String key) {
                 }
             }
         }
+
+        // Since the policies (may have) changed, the whole steps cache needs to be thrown out.
+        // We do this after we update `lifecyclePolicyMap` to ensure `cachedSteps` does not contain outdated data.
+        // This means we may clear up-to-date data, but that's a lot better than the cache containing outdated entries indefinitely.
+        cachedSteps.clear();
     }
 
     /**