(API) Cluster Health report unassigned_primary_shards (#111727) (#112024)

This PR adds a count of currently unassigned primary shards to both the
`/_cat/health` and `/_cluster/health` endpoints. This is to aid cluster
administrators in estimating the time remaining for a cluster to go from
RED to YELLOW status as per enchancement request #111727.

Tests and doc updates are in place with this PR and manual testing with
`./gradlew run` has been conducted on the endpoints to ensure correct
output.

## Known Limitations * Testing   * Due to limitations in the YAML REST
test framework skip functionality, YAML REST tests for this endpoint are
disabled when running a mixed version cluster by using a cluster version
number synthetic feature to skip when any member of the cluster is not
at a version greater than when this change is due to be introduced

Author: lukewhiting

docs/changelog/112024.yaml

@@ -0,0 +1,5 @@
+pr: 112024
+summary: (API) Cluster Health report `unassigned_primary_shards`
+area: Health
+type: enhancement
+issues: []

docs/reference/cat/health.asciidoc

@@ -6,8 +6,8 @@
 
 [IMPORTANT]
 ====
-cat APIs are only intended for human consumption using the command line or {kib} 
-console. They are _not_ intended for use by applications. For application 
+cat APIs are only intended for human consumption using the command line or {kib}
+console. They are _not_ intended for use by applications. For application
 consumption, use the <<cluster-health,cluster health API>>.
 ====
 
@@ -87,8 +87,8 @@ The API returns the following response:
 
 [source,txt]
 --------------------------------------------------
-epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
-1475871424 16:17:04  elasticsearch green           1         1      1   1    0    0        0             0                  -                100.0%
+epoch      timestamp cluster       status node.total node.data shards pri relo init unassign unassign.pri pending_tasks max_task_wait_time active_shards_percent
+1475871424 16:17:04  elasticsearch green           1         1      1   1    0    0        0            0             0                  -                100.0%
 --------------------------------------------------
 // TESTRESPONSE[s/1475871424 16:17:04/\\d+ \\d+:\\d+:\\d+/]
 // TESTRESPONSE[s/elasticsearch/[^ ]+/ s/0                  -/\\d+ (-|\\d+(\\.\\d+)?[ms]+)/ non_json]
@@ -107,11 +107,13 @@ The API returns the following response:
 
 [source,txt]
 --------------------------------------------------
-cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
-elasticsearch green           1         1      1   1    0    0        0             0                  -                100.0%
+cluster       status node.total node.data shards pri relo init unassign unassign.pri pending_tasks max_task_wait_time active_shards_percent
+elasticsearch green           1         1      1   1    0    0        0            0             0                  -                100.0%
 --------------------------------------------------
 // TESTRESPONSE[s/elasticsearch/[^ ]+/ s/0                  -/\\d+ (-|\\d+(\\.\\d+)?[ms]+)/ non_json]
 
+**Note**: The reported number of unassigned primary shards may be lower than the true value if your cluster contains nodes running a version below 8.16. For a more accurate count in this scenario, please use the <<cluster-health,cluster health API>>.
+
 [[cat-health-api-example-across-nodes]]
 ===== Example across nodes
 You can use the cat health API to verify the health of a cluster across nodes.
@@ -121,11 +123,11 @@ For example:
 --------------------------------------------------
 % pssh -i -h list.of.cluster.hosts curl -s localhost:9200/_cat/health
 [1] 20:20:52 [SUCCESS] es3.vm
-1384309218 18:20:18 foo green 3 3 3 3 0 0 0 0
+1384309218 18:20:18 foo green 3 3 3 3 0 0 0 0 0
 [2] 20:20:52 [SUCCESS] es1.vm
-1384309218 18:20:18 foo green 3 3 3 3 0 0 0 0
+1384309218 18:20:18 foo green 3 3 3 3 0 0 0 0 0
 [3] 20:20:52 [SUCCESS] es2.vm
-1384309218 18:20:18 foo green 3 3 3 3 0 0 0 0
+1384309218 18:20:18 foo green 3 3 3 3 0 0 0 0 0
 --------------------------------------------------
 // NOTCONSOLE
 
@@ -138,15 +140,15 @@ in a delayed loop. For example:
 [source,sh]
 --------------------------------------------------
 % while true; do curl localhost:9200/_cat/health; sleep 120; done
-1384309446 18:24:06 foo red 3 3 20 20 0 0 1812 0
-1384309566 18:26:06 foo yellow 3 3 950 916 0 12 870 0
-1384309686 18:28:06 foo yellow 3 3 1328 916 0 12 492 0
-1384309806 18:30:06 foo green 3 3 1832 916 4 0 0
+1384309446 18:24:06 foo red 3 3 20 20 0 0 1812 1121 0
+1384309566 18:26:06 foo yellow 3 3 950 916 0 12 870 421 0
+1384309686 18:28:06 foo yellow 3 3 1328 916 0 12 492 301 0
+1384309806 18:30:06 foo green 3 3 1832 916 4 0 0 0
 ^C
 --------------------------------------------------
 // NOTCONSOLE
 
 In this example, the recovery took roughly six minutes, from `18:24:06` to
 `18:30:06`. If this recovery took hours, you could continue to monitor the
 number of `UNASSIGNED` shards, which should drop. If the number of `UNASSIGNED`
-shards remains static, it would indicate an issue with the cluster recovery.
+shards remains static, it would indicate an issue with the cluster recovery.

docs/reference/cluster/health.asciidoc

@@ -20,22 +20,22 @@ Returns the health status of a cluster.
 [[cluster-health-api-desc]]
 ==== {api-description-title}
 
-The cluster health API returns a simple status on the health of the 
+The cluster health API returns a simple status on the health of the
 cluster. You can also use the API to get the health status of only specified
 data streams and indices. For data streams, the API retrieves the health status
 of the stream's backing indices.
 
-The cluster health status is: `green`, `yellow` or `red`. On the shard level, a 
-`red` status indicates that the specific shard is not allocated in the cluster, 
-`yellow` means that the primary shard is allocated but replicas are not, and 
-`green` means that all shards are allocated. The index level status is 
-controlled by the worst shard status. The cluster status is controlled by the 
+The cluster health status is: `green`, `yellow` or `red`. On the shard level, a
+`red` status indicates that the specific shard is not allocated in the cluster,
+`yellow` means that the primary shard is allocated but replicas are not, and
+`green` means that all shards are allocated. The index level status is
+controlled by the worst shard status. The cluster status is controlled by the
 worst index status.
 
-One of the main benefits of the API is the ability to wait until the cluster 
-reaches a certain high water-mark health level. For example, the following will 
-wait for 50 seconds for the cluster to reach the `yellow` level (if it reaches 
-the `green` or `yellow` status before 50 seconds elapse, it will return at that 
+One of the main benefits of the API is the ability to wait until the cluster
+reaches a certain high water-mark health level. For example, the following will
+wait for 50 seconds for the cluster to reach the `yellow` level (if it reaches
+the `green` or `yellow` status before 50 seconds elapse, it will return at that
 point):
 
 [source,console]
@@ -58,31 +58,31 @@ To target all data streams and indices in a cluster, omit this parameter or use
 ==== {api-query-parms-title}
 
 `level`::
-    (Optional, string) Can be one of `cluster`, `indices` or `shards`. Controls 
+    (Optional, string) Can be one of `cluster`, `indices` or `shards`. Controls
     the details level of the health information returned. Defaults to `cluster`.
-    
+
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=local]
-    
+
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
 
 `wait_for_active_shards`::
-    (Optional, string) A number controlling to how many active shards to wait 
-    for, `all` to wait for all shards in the cluster to be active, or `0` to not 
+    (Optional, string) A number controlling to how many active shards to wait
+    for, `all` to wait for all shards in the cluster to be active, or `0` to not
     wait. Defaults to `0`.
-    
+
 `wait_for_events`::
-    (Optional, string) Can be one of `immediate`, `urgent`, `high`, `normal`, 
-    `low`, `languid`. Wait until all currently queued events with the given 
+    (Optional, string) Can be one of `immediate`, `urgent`, `high`, `normal`,
+    `low`, `languid`. Wait until all currently queued events with the given
     priority are processed.
 
 `wait_for_no_initializing_shards`::
-    (Optional, Boolean) A boolean value which controls whether to wait (until 
-    the timeout provided) for the cluster to have no shard initializations. 
+    (Optional, Boolean) A boolean value which controls whether to wait (until
+    the timeout provided) for the cluster to have no shard initializations.
     Defaults to false, which means it will not wait for initializing shards.
 
 `wait_for_no_relocating_shards`::
-    (Optional, Boolean) A boolean value which controls whether to wait (until 
-    the timeout provided) for the cluster to have no shard relocations. Defaults 
+    (Optional, Boolean) A boolean value which controls whether to wait (until
+    the timeout provided) for the cluster to have no shard relocations. Defaults
     to false, which means it will not wait for relocating shards.
 
 `wait_for_nodes`::
@@ -92,7 +92,7 @@ include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
     `lt(N)` notation.
 
 `wait_for_status`::
-    (Optional, string) One of `green`, `yellow` or `red`. Will wait (until the 
+    (Optional, string) One of `green`, `yellow` or `red`. Will wait (until the
     timeout provided) until the status of the cluster changes to the one
     provided or better, i.e. `green` > `yellow` > `red`. By default, will not
     wait for any status.
@@ -107,7 +107,7 @@ include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
 include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=cluster-health-status]
 
 `timed_out`::
-    (Boolean) If `false` the response returned within the period of 
+    (Boolean) If `false` the response returned within the period of
     time that is specified by the `timeout` parameter (`30s` by default).
 
 `number_of_nodes`::
@@ -131,23 +131,26 @@ include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=cluster-health-status]
 `unassigned_shards`::
     (integer) The number of shards that are not allocated.
 
+`unassigned_primary_shards`::
+    (integer) The number of shards that are primary but not allocated. **Note**: This number may be lower than the true value if your cluster contains nodes running a version below 8.16. For a more accurate count in this scenario, please use the <<cluster-health,cluster health API>>.
+
 `delayed_unassigned_shards`::
-    (integer) The number of shards whose allocation has been delayed by the 
+    (integer) The number of shards whose allocation has been delayed by the
     timeout settings.
 
 `number_of_pending_tasks`::
-    (integer) The number of cluster-level changes that have not yet been 
+    (integer) The number of cluster-level changes that have not yet been
     executed.
 
 `number_of_in_flight_fetch`::
     (integer) The number of unfinished fetches.
 
 `task_max_waiting_in_queue_millis`::
-    (integer) The time expressed in milliseconds since the earliest initiated task 
+    (integer) The time expressed in milliseconds since the earliest initiated task
     is waiting for being performed.
 
 `active_shards_percent_as_number`::
-    (float) The ratio of active shards in the cluster expressed as a percentage. 
+    (float) The ratio of active shards in the cluster expressed as a percentage.
 
 [[cluster-health-api-example]]
 ==== {api-examples-title}
@@ -158,7 +161,7 @@ GET _cluster/health
 --------------------------------------------------
 // TEST[s/^/PUT test1\n/]
 
-The API returns the following response in case of a quiet single node cluster 
+The API returns the following response in case of a quiet single node cluster
 with a single index with one shard and one replica:
 
 [source,console-result]
@@ -174,6 +177,7 @@ with a single index with one shard and one replica:
   "relocating_shards" : 0,
   "initializing_shards" : 0,
   "unassigned_shards" : 1,
+  "unassigned_primary_shards" : 0,
   "delayed_unassigned_shards": 0,
   "number_of_pending_tasks" : 0,
   "number_of_in_flight_fetch": 0,

rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/cat.health/10_basic.yml

@@ -1,32 +1,45 @@
 ---
 "Help":
+  - requires:
+      capabilities:
+        - method: GET
+          path: /_cluster/health
+          capabilities: [ unassigned_pri_shard_count ]
+      test_runner_features: capabilities
+      reason: Capability required to run test
   - do:
       cat.health:
         help: true
 
   - match:
       $body: |
-               /^  epoch         .+ \n
-                   timestamp     .+ \n
-                   cluster       .+ \n
-                   status        .+ \n
-                   node.total    .+ \n
-                   node.data     .+ \n
-                   shards        .+ \n
-                   pri           .+ \n
-                   relo          .+ \n
-                   init          .+ \n
-                   unassign      .+ \n
-                   pending_tasks .+ \n
-                   max_task_wait_time .+ \n
-                   active_shards_percent .+ \n
-
+               /^  epoch         .+\n
+                   timestamp     .+\n
+                   cluster       .+\n
+                   status        .+\n
+                   node.total    .+\n
+                   node.data     .+\n
+                   shards        .+\n
+                   pri           .+\n
+                   relo          .+\n
+                   init          .+\n
+                   unassign      .+\n
+                   unassign.pri  .+\n
+                   pending_tasks .+\n
+                   max_task_wait_time .+\n
+                   active_shards_percent .+\n
                $/
 
 
 ---
 "Empty cluster":
-
+  - requires:
+      capabilities:
+        - method: GET
+          path: /_cluster/health
+          capabilities: [ unassigned_pri_shard_count ]
+      test_runner_features: capabilities
+      reason: Capability required to run test
   - do:
       cat.health: {}
 
@@ -44,6 +57,7 @@
                 \d+            \s+ # relo
                 \d+            \s+ # init
                 \d+            \s+ # unassign
+                \d+            \s+ # unassign.pri
                 \d+            \s+ # pending_tasks
                 (-|\d+(?:[.]\d+)?m?s) \s+ # max task waiting time
                 \d+\.\d+%             # active shards percent
@@ -54,7 +68,13 @@
 
 ---
 "With ts parameter":
-
+  - requires:
+      capabilities:
+        - method: GET
+          path: /_cluster/health
+          capabilities: [ unassigned_pri_shard_count ]
+      test_runner_features: capabilities
+      reason: Capability required to run test
   - do:
       cat.health:
         ts: false
@@ -71,6 +91,7 @@
                 \d+            \s+ # relo
                 \d+            \s+ # init
                 \d+            \s+ # unassign
+                \d+            \s+ # unassign.pri
                 \d+            \s+ # pending_tasks
                 (-|\d+(?:[.]\d+)?m?s) \s+ # max task waiting time
                 \d+\.\d+%             # active shards percent