Introduce repository integrity verification API (#112348)

Adds an API which scans all the metadata (and optionally the raw data)
in a snapshot repository to look for corruptions or other
inconsistencies.

Closes #52622 Closes
ES-8560

Author: DaveCTurner

docs/changelog/112348.yaml

@@ -0,0 +1,6 @@
+pr: 112348
+summary: Introduce repository integrity verification API
+area: Snapshot/Restore
+type: enhancement
+issues:
+ - 52622

docs/reference/snapshot-restore/apis/snapshot-restore-apis.asciidoc

@@ -28,6 +28,7 @@ For more information, see <<snapshot-restore>>.
 include::put-repo-api.asciidoc[]
 include::verify-repo-api.asciidoc[]
 include::repo-analysis-api.asciidoc[]
+include::verify-repo-integrity-api.asciidoc[]
 include::get-repo-api.asciidoc[]
 include::delete-repo-api.asciidoc[]
 include::clean-up-repo-api.asciidoc[]

docs/reference/snapshot-restore/apis/verify-repo-integrity-api.asciidoc

@@ -0,0 +1,232 @@
+[role="xpack"]
+[[verify-repo-integrity-api]]
+=== Verify repository integrity API
+++++
+<titleabbrev>Verify repository integrity</titleabbrev>
+++++
+
+Verifies the integrity of the contents of a snapshot repository.
+
+////
+[source,console]
+----
+PUT /_snapshot/my_repository
+{
+  "type": "fs",
+  "settings": {
+    "location": "my_backup_location"
+  }
+}
+----
+// TESTSETUP
+////
+
+[source,console]
+----
+POST /_snapshot/my_repository/_verify_integrity
+----
+
+[[verify-repo-integrity-api-request]]
+==== {api-request-title}
+
+`POST /_snapshot/<repository>/_verify_integrity`
+
+[[verify-repo-integrity-api-prereqs]]
+==== {api-prereq-title}
+
+* If the {es} {security-features} are enabled, you must have the `manage`
+<<privileges-list-cluster,cluster privilege>> to use this API. For more
+information, see <<security-privileges>>.
+
+[[verify-repo-integrity-api-desc]]
+==== {api-description-title}
+
+This API allows you to perform a comprehensive check of the contents of a
+repository, looking for any anomalies in its data or metadata which might
+prevent you from restoring snapshots from the repository or which might cause
+future snapshot create or delete operations to fail.
+
+If you suspect the integrity of the contents of one of your snapshot
+repositories, cease all write activity to this repository immediately, set its
+`read_only` option to `true`, and use this API to verify its integrity. Until
+you do so:
+
+* It may not be possible to <<snapshots-restore-snapshot,restore some
+  snapshots>> from this repository.
+
+* <<searchable-snapshots>> may report errors when searched, or may have
+  unassigned shards.
+
+* <<snapshots-take-snapshot,Taking snapshots>> into this repository may fail,
+  or may appear to succeed having created a snapshot which cannot be restored.
+
+* <<delete-snapshot-api,Deleting snapshots>> from this repository may fail, or
+  may appear to succeed leaving the underlying data on disk.
+
+* Continuing to write to the repository while it is in an invalid state may
+  causing additional damage to its contents.
+
+If the <<verify-repo-integrity-api>> API finds any problems with the integrity
+of the contents of your repository, {es} will not be able to repair the damage.
+The only way to bring the repository back into a fully working state after its
+contents have been damaged is by restoring its contents from a
+<<snapshots-repository-backup,repository backup>> which was taken before the
+damage occurred. You must also identify what caused the damage and take action
+to prevent it from happening again.
+
+If you cannot restore a repository backup,
+<<snapshots-register-repository,register a new repository>> and use this for
+all future snapshot operations. In some cases it may be possible to recover
+some of the contents of a damaged repository, either by
+<<snapshots-restore-snapshot,restoring>> as many of its snapshots as needed and
+<<snapshots-take-snapshot,taking new snapshots>> of the restored data, or by
+using the <<docs-reindex>> API to copy data from any <<searchable-snapshots>>
+mounted from the damaged repository.
+
+Avoid all operations which write to the repository while the
+<<verify-repo-integrity-api>> API is running. If something changes the
+repository contents while an integrity verification is running then {es} may
+incorrectly report having detected some anomalies in its contents due to the
+concurrent writes. It may also incorrectly fail to report some anomalies that
+the concurrent writes prevented it from detecting.
+
+NOTE: This API is intended for exploratory use by humans. You should expect the
+request parameters and the response format to vary in future versions.
+
+NOTE: This API may not work correctly in a mixed-version cluster.
+
+[[verify-repo-integrity-api-path-params]]
+==== {api-path-parms-title}
+
+`<repository>`::
+(Required, string)
+Name of the snapshot repository whose integrity to verify.
+
+[[verify-repo-integrity-api-query-params]]
+==== {api-query-parms-title}
+
+The default values for the parameters of this API are designed to limit the
+impact of the integrity verification on other activities in your cluster. For
+instance, by default it will only use at most half of the `snapshot_meta`
+threads to verify the integrity of each snapshot, allowing other snapshot
+operations to use the other half of this thread pool.
+
+If you modify these parameters to speed up the verification process, you risk
+disrupting other snapshot-related operations in your cluster. For large
+repositories, consider setting up a separate single-node {es} cluster just for
+running the integrity verification API.
+
+`snapshot_verification_concurrency`::
+(Optional, integer) Specifies the number of snapshots to verify concurrently.
+Defaults to `0` which means to use at most half of the `snapshot_meta` thread
+pool at once.
+
+`index_verification_concurrency`::
+(Optional, integer) Specifies the number of indices to verify concurrently.
+Defaults to `0` which means to use the entire `snapshot_meta` thread pool.
+
+`meta_thread_pool_concurrency`::
+(Optional, integer) Specifies the maximum number of snapshot metadata
+operations to execute concurrently. Defaults to `0` which means to use at most
+half of the `snapshot_meta` thread pool at once.
+
+`index_snapshot_verification_concurrency`::
+(Optional, integer) Specifies the maximum number of index snapshots to verify
+concurrently within each index verification. Defaults to `1`.
+
+`max_failed_shard_snapshots`::
+(Optional, integer) Limits the number of shard snapshot failures to track
+during integrity verification, in order to avoid excessive resource usage. If
+your repository contains more than this number of shard snapshot failures then
+the verification will fail. Defaults to `10000`.
+
+`verify_blob_contents`::
+(Optional, boolean) Specifies whether to verify the checksum of every data blob
+in the repository. Defaults to `false`. If this feature is enabled, {es} will
+read the entire repository contents, which may be extremely slow and expensive.
+
+`blob_thread_pool_concurrency`::
+(Optional, integer) If `?verify_blob_contents` is `true`, this parameter
+specifies how many blobs to verify at once. Defaults to `1`.
+
+`max_bytes_per_sec`::
+(Optional, <<size-units, size units>>)
+If `?verify_blob_contents` is `true`, this parameter specifies the maximum
+amount of data that {es} will read from the repository every second. Defaults
+to `10mb`.
+
+[role="child_attributes"]
+[[verify-repo-integrity-api-response-body]]
+==== {api-response-body-title}
+
+The response exposes implementation details of the analysis which may change
+from version to version. The response body format is therefore not considered
+stable and may be different in newer versions.
+
+`log`::
+(array) A sequence of objects that report the progress of the analysis.
++
+.Properties of `log`
+[%collapsible%open]
+====
+`timestamp_in_millis`::
+(integer) The timestamp of this log entry, represented as the number of
+milliseconds since the {wikipedia}/Unix_time[Unix epoch].
+
+`timestamp`::
+(string) The timestamp of this log entry, represented as a string formatted
+according to {wikipedia}/ISO_8601[ISO 8601]. Only included if the
+<<common-options,`?human`>> flag is set.
+
+`snapshot`::
+(object) If the log entry pertains to a particular snapshot then the snapshot
+will be described in this object.
+
+`index`::
+(object) If the log entry pertains to a particular index then the index will be
+described in this object.
+
+`snapshot_restorability`::
+(object) If the log entry pertains to the restorability of an index then the
+details will be described in this object.
+
+`anomaly`::
+(string) If the log entry pertains to an anomaly in the repository contents then
+this string will describe the anomaly.
+
+`exception`::
+(object) If the log entry pertains to an exception that {es} encountered during
+the verification then the details will be included in this object.
+
+====
+
+`results`::
+(object) An object which describes the final results of the analysis.
++
+.Properties of `results`
+[%collapsible%open]
+====
+`status`::
+(object) The final status of the analysis task.
+
+`final_repository_generation`::
+(integer) The repository generation at the end of the analysis. If there were
+any writes to the repository during the analysis then this value will be
+different from the `generation` reported in the task status, and the analysis
+may have detected spurious anomalies due to the concurrent writes, or may even
+have failed to detect some anomalies in the repository contents.
+
+`total_anomalies`::
+(integer) The total number of anomalies detected during the analysis.
+
+`result`::
+(string) The final result of the analysis. If the repository contents appear to
+be intact then this will be the string `pass`. If this field is missing, or
+contains some other value, then the repository contents were not fully
+verified.
+
+====
+
+`exception`::
+(object) If the analysis encountered an exception which prevented it from
+completing successfully then this exception will be reported here.

docs/reference/snapshot-restore/register-repository.asciidoc

@@ -272,7 +272,9 @@ filesystem snapshot of this repository.
 When restoring a repository from a backup, you must not register the repository
 with {es} until the repository contents are fully restored. If you alter the
 contents of a repository while it is registered with {es} then the repository
-may become unreadable or may silently lose some of its contents.
+may become unreadable or may silently lose some of its contents. After
+restoring a repository from a backup, use the <<verify-repo-integrity-api>> API
+to verify its integrity before you start to use the repository.
 
 include::repository-azure.asciidoc[]
 include::repository-gcs.asciidoc[]

rest-api-spec/src/main/resources/rest-api-spec/api/snapshot.repository_verify_integrity.json

@@ -0,0 +1,65 @@
+{
+  "snapshot.repository_verify_integrity":{
+    "documentation":{
+      "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-snapshots.html",
+      "description":"Verifies the integrity of the contents of a snapshot repository"
+    },
+    "stability":"experimental",
+    "visibility":"private",
+    "headers": {
+      "accept": [
+        "application/json"
+      ]
+    },
+    "url":{
+      "paths":[
+        {
+          "path":"/_snapshot/{repository}/_verify_integrity",
+          "methods":[
+            "POST"
+          ],
+          "parts":{
+            "repository":{
+              "type":"string",
+              "description":"A repository name"
+            }
+          }
+        }
+      ]
+    },
+    "params":{
+      "meta_thread_pool_concurrency":{
+        "type":"number",
+        "description":"Number of threads to use for reading metadata"
+      },
+      "blob_thread_pool_concurrency":{
+        "type":"number",
+        "description":"Number of threads to use for reading blob contents"
+      },
+      "snapshot_verification_concurrency":{
+        "type":"number",
+        "description":"Number of snapshots to verify concurrently"
+      },
+      "index_verification_concurrency":{
+        "type":"number",
+        "description":"Number of indices to verify concurrently"
+      },
+      "index_snapshot_verification_concurrency":{
+        "type":"number",
+        "description":"Number of snapshots to verify concurrently within each index"
+      },
+      "max_failed_shard_snapshots":{
+        "type":"number",
+        "description":"Maximum permitted number of failed shard snapshots"
+      },
+      "verify_blob_contents":{
+        "type":"boolean",
+        "description":"Whether to verify the contents of individual blobs"
+      },
+      "max_bytes_per_sec":{
+        "type":"string",
+        "description":"Rate limit for individual blob verification"
+      }
+    }
+  }
+}

server/src/main/java/org/elasticsearch/repositories/RepositoryData.java

@@ -281,6 +281,13 @@ public Collection<SnapshotId> getSnapshotIds() {
         return snapshotIds.values();
     }
 
+    /**
+     * @return the number of index snapshots (i.e. the sum of the index count of each snapshot)
+     */
+    public long getIndexSnapshotCount() {
+        return indexSnapshots.values().stream().mapToLong(List::size).sum();
+    }
+
     /**
      * @return whether some of the {@link SnapshotDetails} of the given snapshot are missing, due to BwC, so that they must be loaded from
      * the {@link SnapshotInfo} blob instead.

x-pack/plugin/ml/qa/native-multi-node-tests/build.gradle

@@ -17,6 +17,7 @@ dependencies {
   javaRestTestImplementation project(path: xpackModule('rank-rrf'))
   javaRestTestImplementation project(path: xpackModule('esql-core'))
   javaRestTestImplementation project(path: xpackModule('esql'))
+  javaRestTestImplementation project(path: xpackModule('snapshot-repo-test-kit'))
 }
 
 // location for keys and certificates

x-pack/plugin/security/qa/operator-privileges-tests/src/javaRestTest/java/org/elasticsearch/xpack/security/operator/Constants.java

@@ -98,6 +98,7 @@ public class Constants {
         "cluster:admin/snapshot/restore",
         "cluster:admin/snapshot/status",
         "cluster:admin/snapshot/status[nodes]",
+        "cluster:admin/repository/verify_integrity",
         "cluster:admin/features/get",
         "cluster:admin/features/reset",
         "cluster:admin/tasks/cancel",

x-pack/plugin/snapshot-repo-test-kit/qa/rest/src/yamlRestTest/resources/rest-api-spec/test/20_verify_integrity.yml

@@ -0,0 +1,39 @@
+---
+setup:
+  - requires:
+      cluster_features: "snapshot.repository_verify_integrity"
+      reason: "required feature"
+
+  - do:
+      snapshot.create_repository:
+        repository: test_repo
+        body:
+          type: fs
+          settings:
+            location: "test_repo_loc"
+
+  - do:
+      bulk:
+        index: test
+        refresh: true
+        body:
+          - '{"index":{}}'
+          - '{}'
+
+  - do:
+      snapshot.create:
+        repository: test_repo
+        snapshot: snap
+        wait_for_completion: true
+
+---
+"Integrity verification":
+  - do:
+      snapshot.repository_verify_integrity:
+        repository: test_repo
+
+  - match: {results.result: pass}
+  - match: {results.status.snapshots.total: 1}
+  - match: {results.status.snapshots.verified: 1}
+  - match: {results.status.indices.total: 1}
+  - match: {results.status.indices.verified: 1}