ES-10037 Periodic logging in autosharding service

This enhances `DataStreamAutoShardingService` so that it periodically
logs at `INFO` level the most 'interesting' results it has produced in
the last period.

In this PR, the most 'interesting' results are considered to be the
ones with the highest load, keeping track separately of the top 10
which resulting in an increase decision and the top 10 which did
not. In practice, increase recommendations are sufficiently rare that
the top 10 will often be 'all of them', and they are all potentially
interesting (but we cap it to protect against taking up an unbounded
amount of memory). We keep the high load non-increase recommendations
as well, since these are likely to be the interesting ones to look at
when investigating why some data stream did not get an increase shards
recommendation when we might have expected it.

The mechanism would be easily extended to add in other categories.

The existing `DEBUG` and `TRACE` log lines in the service are replaced
with a single `DEBUG` log which pulls together all the data. This is
an improvement, since at the moment it is hard to figure out from the
logs which lines refer to the same data stream (they are interleaved,
and don't all include the data stream name).

The write load field in the `AutoShardingResult` was unused, and is
removed.

Author: PeteGillinElastic

server/src/main/java/org/elasticsearch/action/datastreams/autosharding/AutoShardingResult.java

@@ -9,7 +9,7 @@
 
 package org.elasticsearch.action.datastreams.autosharding;
 
-import org.elasticsearch.core.Nullable;
+import org.elasticsearch.common.Strings;
 import org.elasticsearch.core.TimeValue;
 
 import java.util.Arrays;
@@ -25,13 +25,7 @@
  * {@link DataStreamAutoShardingService#DATA_STREAMS_AUTO_SHARDING_EXCLUDES_SETTING}) the target number of shards will be -1 and cool down
  * remaining {@link TimeValue#MAX_VALUE}.
  */
-public record AutoShardingResult(
-    AutoShardingType type,
-    int currentNumberOfShards,
-    int targetNumberOfShards,
-    TimeValue coolDownRemaining,
-    @Nullable Double writeLoad
-) {
+public record AutoShardingResult(AutoShardingType type, int currentNumberOfShards, int targetNumberOfShards, TimeValue coolDownRemaining) {
 
     static final String COOLDOWN_PREVENTING_TYPES = Arrays.toString(
         new AutoShardingType[] { COOLDOWN_PREVENTED_DECREASE, COOLDOWN_PREVENTED_INCREASE }
@@ -53,8 +47,36 @@ public record AutoShardingResult(
         AutoShardingType.NOT_APPLICABLE,
         -1,
         -1,
-        TimeValue.MAX_VALUE,
-        null
+        TimeValue.MAX_VALUE
     );
 
+    @Override
+    public String toString() {
+        return switch (type) {
+            case INCREASE_SHARDS -> Strings.format(
+                "Recommendation to increase shards from %d to %d",
+                currentNumberOfShards,
+                targetNumberOfShards
+            );
+            case DECREASE_SHARDS -> Strings.format(
+                "Recommendation to decrease shards from %d to %d",
+                currentNumberOfShards,
+                targetNumberOfShards
+            );
+            case COOLDOWN_PREVENTED_INCREASE -> Strings.format(
+                "Deferred recommendation to increase shards from %d to %d after cooldown period %s",
+                currentNumberOfShards,
+                targetNumberOfShards,
+                coolDownRemaining
+            );
+            case COOLDOWN_PREVENTED_DECREASE -> Strings.format(
+                "Deferred recommendation to decrease shards from %d to %d after cooldown period %s",
+                currentNumberOfShards,
+                targetNumberOfShards,
+                coolDownRemaining
+            );
+            case NO_CHANGE_REQUIRED -> Strings.format("Recommendation to leave shards unchanged at %d", currentNumberOfShards);
+            case NOT_APPLICABLE -> "No recommendation as auto-sharding not enabled";
+        };
+    }
 }