Don't update system index mappings in mixed clusters (#101778) (#101798)

* Don't update system index mappings in mixed clusters

#99668 seems to have introduced a bug where
SystemIndexMappingUpdateService updates system index mappings
even in mixed clusters. This PR restores the old version-based
check in order to be sure that there's no update until the
cluster is fully upgraded.

The timing of the mapping update seems to be causing worse
problems, corrupting persisted cluster state.

Fixes #99778, #101331

* Remove broken assertion

The compatibility versions objects are not showing up
correctly, so we shouldn't assert on them.

Author: williamrandolph

docs/changelog/101778.yaml

@@ -0,0 +1,7 @@
+pr: 101778
+summary: Don't update system index mappings in mixed clusters
+area: Infra/Core
+type: bug
+issues:
+ - 101331
+ - 99778

qa/rolling-upgrade-legacy/src/test/java/org/elasticsearch/upgrades/RecoveryIT.java

@@ -8,7 +8,6 @@
 package org.elasticsearch.upgrades;
 
 import org.apache.http.util.EntityUtils;
-import org.apache.lucene.tests.util.LuceneTestCase;
 import org.elasticsearch.Version;
 import org.elasticsearch.action.support.PlainActionFuture;
 import org.elasticsearch.client.Request;
@@ -53,7 +52,6 @@
 /**
  * In depth testing of the recovery mechanism during a rolling restart.
  */
-@LuceneTestCase.AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/99778")
 public class RecoveryIT extends AbstractRollingTestCase {
 
     private static String CLUSTER_NAME = System.getProperty("tests.clustername");

server/src/main/java/org/elasticsearch/indices/SystemIndexMappingUpdateService.java

@@ -92,7 +92,7 @@ public void clusterChanged(ClusterChangedEvent event) {
         }
 
         // if we're in a mixed-version cluster, exit
-        if (state.hasMixedSystemIndexVersions()) {
+        if (state.nodes().getMaxNodeVersion().after(state.nodes().getSmallestNonClientNodeVersion())) {
             logger.debug("Skipping system indices up-to-date check as cluster has mixed versions");
             return;
         }