Periodic logging in autosharding service

This enhances DataStreamAutoShardingService so that it periodically
logs at INFO level the most 'interesting' results it has produced in
the last period.

In this PR, the most 'interesting' results are considered to be the
ones with the highest load, keeping track separately of the top 10
which resulting in an increase decision and the top 10 which did
not. In practice, increase recommendations are sufficiently rare that
the top 10 will often be 'all of them', and they are all potentially
interesting (but we cap it to protect against taking up an unbounded
amount of memory). We keep the high load non-increase recommendations
as well, since these are likely to be the interesting ones to look at
when investigating why some data stream did not get an increase shards
recommendation when we might have expected it.

The mechanism would be easily extended to add in other categories. For
example, we may find that there are some categories of decrease
decisions we consider 'interesting'. (N.B. The rollover service will
not roll over just because the auto-sharding service recommended
down-sharding — down-sharding only happens if the rollover was going
to happen for some other reason (age, size, etc.) So it's normal for
the auto-sharding service to return decrease recommendations for the
same data streams every 5 minutes until those other conditions are
met. Which makes things a bit more complicated.) This PR just covers
the cases that seem likely to be useful in the support cases we have
seen.

The existing DEBUG and TRACE log lines in the service are replaced
with a single DEBUG log which pulls together all the data. This is an
improvement, since at the moment it is hard to figure out from the
logs which lines refer to the same data stream (they are interleaved,
and don't all include the data stream name).

The write load field in the AutoShardingResult was unused, and is
removed.

Author: PeteGillinElastic

server/src/main/java/org/elasticsearch/action/datastreams/autosharding/AutoShardingResult.java

@@ -9,7 +9,7 @@
 
 package org.elasticsearch.action.datastreams.autosharding;
 
-import org.elasticsearch.core.Nullable;
+import org.elasticsearch.common.Strings;
 import org.elasticsearch.core.TimeValue;
 
 import java.util.Arrays;
@@ -25,13 +25,7 @@
  * {@link DataStreamAutoShardingService#DATA_STREAMS_AUTO_SHARDING_EXCLUDES_SETTING}) the target number of shards will be -1 and cool down
  * remaining {@link TimeValue#MAX_VALUE}.
  */
-public record AutoShardingResult(
-    AutoShardingType type,
-    int currentNumberOfShards,
-    int targetNumberOfShards,
-    TimeValue coolDownRemaining,
-    @Nullable Double writeLoad
-) {
+public record AutoShardingResult(AutoShardingType type, int currentNumberOfShards, int targetNumberOfShards, TimeValue coolDownRemaining) {
 
     static final String COOLDOWN_PREVENTING_TYPES = Arrays.toString(
         new AutoShardingType[] { COOLDOWN_PREVENTED_DECREASE, COOLDOWN_PREVENTED_INCREASE }
@@ -53,8 +47,36 @@ public record AutoShardingResult(
         AutoShardingType.NOT_APPLICABLE,
         -1,
         -1,
-        TimeValue.MAX_VALUE,
-        null
+        TimeValue.MAX_VALUE
     );
 
+    @Override
+    public String toString() {
+        return switch (type) {
+            case INCREASE_SHARDS -> Strings.format(
+                "Recommendation to increase shards from %d to %d",
+                currentNumberOfShards,
+                targetNumberOfShards
+            );
+            case DECREASE_SHARDS -> Strings.format(
+                "Recommendation to decrease shards from %d to %d",
+                currentNumberOfShards,
+                targetNumberOfShards
+            );
+            case COOLDOWN_PREVENTED_INCREASE -> Strings.format(
+                "Deferred recommendation to increase shards from %d to %d after cooldown period %s",
+                currentNumberOfShards,
+                targetNumberOfShards,
+                coolDownRemaining
+            );
+            case COOLDOWN_PREVENTED_DECREASE -> Strings.format(
+                "Deferred recommendation to decrease shards from %d to %d after cooldown period %s",
+                currentNumberOfShards,
+                targetNumberOfShards,
+                coolDownRemaining
+            );
+            case NO_CHANGE_REQUIRED -> Strings.format("Recommendation to leave shards unchanged at %d", currentNumberOfShards);
+            case NOT_APPLICABLE -> "No recommendation as auto-sharding not enabled";
+        };
+    }
 }