Merge pull request #40026 from apinnick/oadp75-troubleshooting

OADP-75: Troubleshooting

Author: apinnick

_topic_maps/_topic_map.yml

@@ -2095,35 +2095,37 @@ Topics:
   File: graceful-cluster-shutdown
 - Name: Restarting a cluster gracefully
   File: graceful-cluster-restart
-# - Name: Application backup and restore
-#   Dir: application_backup_and_restore
-#   Topics:
-#   - Name: OADP features and plugins
-#     File: oadp-features-plugins
-#   - Name: Installing and configuring OADP
-#     Dir: installing
-#     Topics:
-#     - Name: About installing OADP
-#       File: about-installing-oadp
-#     - Name: Installing and configuring OADP with AWS
-#       File: installing-oadp-aws
-#     - Name: Installing and configuring OADP with Azure
-#       File: installing-oadp-azure
-#     - Name: Installing and configuring OADP with GCP
-#       File: installing-oadp-gcp
-#     - Name: Installing and configuring OADP with MCG
-#       File: installing-oadp-mcg
-#     - Name: Installing and configuring OADP with OCS
-#       File: installing-oadp-ocs
-#     - Name: Uninstalling OADP
-#       File: uninstalling-oadp
-#   - Name: Backing up and restoring
-#     Dir: backing_up_and_restoring
-#     Topics:
-#     - Name: Backing up applications
-#       File: backing-up-applications
-#     - Name: Restoring applications
-#       File: restoring-applications
+- Name: Application backup and restore
+  Dir: application_backup_and_restore
+  Topics:
+  - Name: OADP features and plugins
+    File: oadp-features-plugins
+  - Name: Installing and configuring OADP
+    Dir: installing
+    Topics:
+    - Name: About installing OADP
+      File: about-installing-oadp
+    - Name: Installing and configuring OADP with AWS
+      File: installing-oadp-aws
+    - Name: Installing and configuring OADP with Azure
+      File: installing-oadp-azure
+    - Name: Installing and configuring OADP with GCP
+      File: installing-oadp-gcp
+    - Name: Installing and configuring OADP with MCG
+      File: installing-oadp-mcg
+    - Name: Installing and configuring OADP with OCS
+      File: installing-oadp-ocs
+    - Name: Uninstalling OADP
+      File: uninstalling-oadp
+  - Name: Backing up and restoring
+    Dir: backing_up_and_restoring
+    Topics:
+    - Name: Backing up applications
+      File: backing-up-applications
+    - Name: Restoring applications
+      File: restoring-applications
+  - Name: Troubleshooting
+    File: troubleshooting
 - Name: Control plane backup and restore
   Dir: control_plane_backup_and_restore
   Topics:

backup_and_restore/application_backup_and_restore/backing_up_and_restoring/backing-up-applications.adoc

@@ -8,7 +8,7 @@ toc::[]
 
 You back up applications by creating a xref:../../../backup_and_restore/application_backup_and_restore/backing_up_and_restoring/backing-up-applications.adoc#oadp-creating-backup-cr_backing-up-applications[`Backup`] custom resource (CR).
 
-The `Backup` CR backs up Kubernetes resources and internal images by saving them as an archive file on S3 object storage.
+The `Backup` CR creates backup files for Kubernetes resources and internal images, on S3 object storage, and snapshots for persistent volumes (PVs), if the cloud provider uses a native snapshot API or the xref:../../../backup_and_restore/application_backup_and_restore/backing_up_and_restoring/backing-up-applications.adoc#oadp-backing-up-pvs-csi_backing-up-applications[Container Storage Interface (CSI)] to create snapshots, such as OpenShift Container Storage 4. For more information, see xref:../../../storage/container_storage_interface/persistent-storage-csi-snapshots.adoc#persistent-storage-csi-snapshots[CSI volume snapshots].
 
 :FeatureName: The `CloudStorage` API for S3 storage
 include::modules/technology-preview.adoc[]

backup_and_restore/application_backup_and_restore/troubleshooting.adoc

@@ -0,0 +1,28 @@
+:_content-type: ASSEMBLY
+[id="troubleshooting"]
+= Troubleshooting
+include::modules/common-attributes.adoc[]
+:context: oadp-troubleshooting
+:oadp-troubleshooting:
+:namespace: openshift-adp
+:local-product: OADP
+:must-gather: registry.access.redhat.com/oadp-operator/oadp-must-gather-rhel8:v1.0
+
+toc::[]
+
+You can debug Velero custom resources (CRs) by using the xref:../../backup_and_restore/application_backup_and_restore/troubleshooting.adoc#oadp-debugging-oc-cli_oadp-troubleshooting[OpenShift CLI tool] or the xref:../../backup_and_restore/application_backup_and_restore/troubleshooting.adoc#migration-debugging-velero-resources_oadp-troubleshooting[Velero CLI tool]. The Velero CLI tool provides more detailed logs and information.
+
+You can check xref:../../backup_and_restore/application_backup_and_restore/troubleshooting.adoc#oadp-installation-issues_oadp-troubleshooting[installation issues], xref:../../backup_and_restore/application_backup_and_restore/troubleshooting.adoc#oadp-backup-restore-cr-issues_oadp-troubleshooting[backup and restore CR issues], and xref:../../backup_and_restore/application_backup_and_restore/troubleshooting.adoc#oadp-restic-issues_oadp-troubleshooting[Restic issues].
+
+You can collect logs, CR information, and Prometheus metric data by using the xref:../../backup_and_restore/application_backup_and_restore/troubleshooting.adoc#migration-using-must-gather_oadp-troubleshooting[`must-gather` tool].
+
+include::modules/oadp-debugging-oc-cli.adoc[leveloffset=+1]
+include::modules/migration-debugging-velero-resources.adoc[leveloffset=+1]
+
+include::modules/oadp-installation-issues.adoc[leveloffset=+1]
+include::modules/oadp-backup-restore-cr-issues.adoc[leveloffset=+1]
+include::modules/oadp-restic-issues.adoc[leveloffset=+1]
+
+include::modules/migration-using-must-gather.adoc[leveloffset=+1]
+
+:oadp-troubleshooting!:

migrating_from_ocp_3_to_4/troubleshooting-3-4.adoc

@@ -4,6 +4,9 @@
 include::modules/common-attributes.adoc[]
 :context: troubleshooting-3-4
 :troubleshooting-3-4:
+:namespace: openshift-migration
+:local-product: {mtc-short}
+:must-gather: registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8:v{mtc-version}
 
 toc::[]
 

migration_toolkit_for_containers/troubleshooting-mtc.adoc

@@ -4,6 +4,9 @@
 include::modules/common-attributes.adoc[]
 :context: troubleshooting-mtc
 :troubleshooting-mtc:
+:namespace: openshift-migration
+:local-product: {mtc-short}
+:must-gather: registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8:v{mtc-version}
 
 toc::[]
 

modules/migration-debugging-velero-resources.adoc

@@ -1,59 +1,88 @@
 // Module included in the following assemblies:
 //
+// * backup_and_restore/application_backup_and_restore/troubleshooting.adoc
 // * migrating_from_ocp_3_to_4/troubleshooting-3-4.adoc
 // * migration_toolkit_for_containers/troubleshooting-mtc
 
 [id="migration-debugging-velero-resources_{context}"]
-= Using the Velero CLI to debug Backup and Restore CRs
+= Debugging Velero resources with the Velero CLI tool
 
-You can debug the `Backup` and `Restore` custom resources (CRs) and partial migration failures with the Velero command line interface (CLI). The Velero CLI runs in the `velero` pod.
+You can debug `Backup` and `Restore` custom resources (CRs) and retrieve logs with the Velero CLI tool.
 
+The Velero CLI tool provides more detailed information than the OpenShift CLI tool.
+
+[discrete]
 [id="velero-command-syntax_{context}"]
-== Velero command syntax
+== Syntax
+
+Use the `oc exec` command to run a Velero CLI command:
 
-Velero CLI commands use the following syntax:
-[source,terminal]
+[source,terminal,subs="attributes+"]
 ----
-$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero <resource> <command> <resource_id>
+$ oc exec $(oc get pods -n {namespace} -o name | grep velero) \
+  -- ./velero <backup_restore_cr> <command> <cr_name>
 ----
 
-You can specify `velero-<pod> -n openshift-migration` in place of `$(oc get pods -n openshift-migration -o name | grep velero)`.
+.Example
+[source,terminal,subs="attributes+"]
+----
+$ oc exec $(oc get pods -n {namespace} -o name | grep velero) \
+  -- ./velero backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
+----
 
-[id="help-command_{context}"]
-== Help command
+You can specify `velero-<pod> -n {namespace}` in place of `$(oc get pods -n {namespace} -o name | grep velero)`.
 
-The Velero `help` command lists all the Velero CLI commands:
-[source,terminal]
+.Example
+[source,terminal,subs="attributes+"]
 ----
-$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero --help
+$ oc exec velero-<pod> -n {namespace} -- ./velero backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
 ----
 
-[id="describe-command_{context}"]
+[discrete]
+[id="velero-help-option_{context}"]
+== Help option
+
+Use the `velero --help` option to list all Velero CLI commands:
+
+[source,terminal,subs="attributes+"]
+----
+$ oc exec $(oc get pods -n {namespace} -o name | grep velero) -- ./velero --help
+----
+
+[discrete]
+[id="velero-describe-command_{context}"]
 == Describe command
 
-The Velero `describe` command provides a summary of warnings and errors associated with a Velero resource:
-[source,terminal]
+Use the `velero describe` command to retrieve a summary of warnings and errors associated with a `Backup` or `Restore` CR:
+
+[source,terminal,subs="attributes+"]
 ----
-$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero  <resource> describe <resource_id>
+$ oc exec $(oc get pods -n {namespace} -o name | grep velero) \
+  -- ./velero <backup_restore_cr> describe <cr_name>
 ----
 
 .Example
-[source,terminal]
+[source,terminal,subs="attributes+"]
 ----
-$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
+$ oc exec $(oc get pods -n {namespace} -o name | grep velero) \
+  -- ./velero backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
 ----
 
-[id="logs-command_{context}"]
+[discrete]
+[id="velero-logs-command_{context}"]
 == Logs command
 
-The Velero `logs` command provides the logs associated with a Velero resource:
-[source,terminal]
+Use the `velero logs` command to retrieve the logs of a `Backup` or `Restore` CR:
+
+[source,terminal,subs="attributes+"]
 ----
-velero <resource> logs <resource_id>
+$ oc exec $(oc get pods -n {namespace} -o name | grep velero) \
+  -- ./velero <backup_restore_cr> logs <cr_name>
 ----
 
 .Example
-[source,terminal]
+[source,terminal,subs="attributes+"]
 ----
-$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
+$ oc exec $(oc get pods -n {namespace} -o name | grep velero) \
+  -- ./velero restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
 ----

modules/migration-using-must-gather.adoc

@@ -1,19 +1,28 @@
 // Module included in the following assemblies:
 //
 // * migrating_from_ocp_3_to_4/troubleshooting-3-4.adoc
-// * migration_toolkit_for_containers/troubleshooting-mtc
+// * migration_toolkit_for_containers/troubleshooting-mtc.adoc
+// * backup_and_restore/application_backup_and_restore/troubleshooting.adoc
 
+:_content-type: PROCEDURE
 [id="migration-using-must-gather_{context}"]
-
-
-
 = Using the must-gather tool
 
-You can collect logs, metrics, and information about {MTC-short} custom resources by using the `must-gather` tool.
+You can collect logs, metrics, and information about {local-product} custom resources by using the `must-gather` tool.
 
 The `must-gather` data must be attached to all customer cases.
 
+ifdef::troubleshooting-3-4,troubleshooting-mtc[]
 You can collect data for a one-hour or a 24-hour period and view the data with the Prometheus console.
+endif::[]
+ifdef::oadp-troubleshooting[]
+You can run the `must-gather` tool with the following data collection options:
+
+* Full `must-gather` data collection collects Prometheus metrics, pod logs, and Velero CR information for all namespaces where the OADP Operator is installed.
+* Essential `must-gather` data collection collects pod logs and Velero CR information for a specific duration of time, for example, one hour or 24 hours. Prometheus metrics and duplicate logs are not included.
+* `must-gather` data collection with timeout. Data collection can take a long time if there are many failed `Backup` CRs. You can improve performance by setting a timeout value.
+* Prometheus metrics data dump downloads an archive file containing the metrics data collected by Prometheus.
+endif::[]
 
 .Prerequisites
 
@@ -23,29 +32,70 @@ You can collect data for a one-hour or a 24-hour period and view the data with t
 .Procedure
 
 . Navigate to the directory where you want to store the `must-gather` data.
-. Run the `oc adm must-gather` command:
-
-* To gather data for the past hour:
+. Run the `oc adm must-gather` command for one of the following data collection options:
+
+ifdef::troubleshooting-3-4,troubleshooting-mtc[]
+* To collect data for the past hour:
+endif::[]
+ifdef::oadp-troubleshooting[]
+* Full `must-gather` data collection, including Prometheus metrics:
+endif::[]
 +
 [source,terminal,subs="attributes+"]
 ----
-$ oc adm must-gather --image=registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8:v{mtc-version}
+$ oc adm must-gather --image={must-gather}
 ----
 +
-The data is saved as `/must-gather/must-gather.tar.gz`. You can upload this file to a support case on the link:https://access.redhat.com/[Red Hat Customer Portal].
+The data is saved as `must-gather/must-gather.tar.gz`. You can upload this file to a support case on the link:https://access.redhat.com/[Red Hat Customer Portal].
 
-* To gather data for the past 24 hours:
+ifdef::oadp-troubleshooting[]
+* Essential `must-gather` data collection, without Prometheus metrics, for a specific time duration:
++
+[source,terminal,subs="attributes+"]
+----
+$ oc adm must-gather --image={must-gather} \
+  -- /usr/bin/gather_<time>_essential <1>
+----
+<1> Specify the time in hours. Allowed values are `1h`, `6h`, `24h`, `72h`, or `all`, for example, `gather_1h_essential` or `gather_all_essential`.
+
+* `must-gather` data collection with timeout:
++
+[source,terminal,subs="attributes+"]
+----
+$ oc adm must-gather --image={must-gather} \
+  -- /usr/bin/gather_with_timeout <timeout> <1>
+----
+<1> Specify a timeout value in seconds.
+endif::[]
+ifdef::troubleshooting-3-4,troubleshooting-mtc[]
+* To collect data for the past 24 hours:
+endif::[]
+ifdef::oadp-troubleshooting[]
+* Prometheus metrics data dump:
+endif::[]
 +
 [source,terminal,subs="attributes+"]
 ----
-$ oc adm must-gather --image= \
-  registry.redhat.io/rhmtc/openshift-migration-must-gather-rhel8: \
-  v{mtc-version} -- /usr/bin/gather_metrics_dump
+$ oc adm must-gather --image={must-gather} \
+  -- /usr/bin/gather_metrics_dump
 ----
 +
-This operation can take a long time. The data is saved as `/must-gather/metrics/prom_data.tar.gz`. You can view this file with the Prometheus console.
+This operation can take a long time. The data is saved as `must-gather/metrics/prom_data.tar.gz`.
 
-.To view data with the Prometheus console
+[discrete]
+[id="viewing-data-with-prometheus-console_{context}"]
+== Viewing metrics data with the Prometheus console
+
+You can view the metrics data with the Prometheus console.
+
+.Procedure
+
+. Decompress the `prom_data.tar.gz` file:
++
+[source,terminal]
+----
+$ tar -xvzf must-gather/metrics/prom_data.tar.gz
+----
 
 . Create a local Prometheus instance:
 +
@@ -54,7 +104,7 @@ This operation can take a long time. The data is saved as `/must-gather/metrics/
 $ make prometheus-run
 ----
 +
-The command outputs the Prometheus URL:
+The command outputs the Prometheus URL.
 +
 .Output
 [source,terminal]

modules/oadp-backing-up-applications-restic.adoc

@@ -2,6 +2,7 @@
 //
 // * backup_and_restore/application_backup_and_restore/backing_up_and_restoring/backing-up-applications.adoc
 
+:_content-type: PROCEDURE
 [id="oadp-backing-up-applications-restic_{context}"]
 = Backing up applications with Restic