Skip to content

Commit 7b07072

Browse files
apinnickapinnick
authored
Merge pull request #29638 from apinnick/mig527-velero-debug
mig527: Velero debug tool
2 parents 3d2a163 + 6baebe0 commit 7b07072

File tree

6 files changed

+152
-16
lines changed

6 files changed

+152
-16
lines changed

‎migration/migrating_3_4/troubleshooting-3-4.adoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ include::modules/migration-downloading-logs.adoc[leveloffset=+1]
2121
include::modules/migration-updating-deprecated-gvks.adoc[leveloffset=+1]
2222
include::modules/migration-error-messages.adoc[leveloffset=+1]
2323
include::modules/migration-dvm-error-node-selectors.adoc[leveloffset=+1]
24+
include::modules/migration-debugging-velero-resources.adoc[leveloffset=+1]
25+
include::modules/migration-partial-failure-velero.adoc[leveloffset=+2]
2426
include::modules/migration-using-must-gather.adoc[leveloffset=+1]
2527

2628
[id="rolling-back-migration_{context}"]

‎migration/migrating_4_1_4/troubleshooting-4-1-4.adoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ include::modules/migration-using-mig-log-reader.adoc[leveloffset=+1]
2020
include::modules/migration-downloading-logs.adoc[leveloffset=+1]
2121
include::modules/migration-error-messages.adoc[leveloffset=+1]
2222
include::modules/migration-dvm-error-node-selectors.adoc[leveloffset=+1]
23+
include::modules/migration-debugging-velero-resources.adoc[leveloffset=+1]
24+
include::modules/migration-partial-failure-velero.adoc[leveloffset=+2]
2325
include::modules/migration-using-must-gather.adoc[leveloffset=+1]
2426

2527
[id="rolling-back-migration_{context}"]

‎migration/migrating_4_2_4/troubleshooting-4-2-4.adoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ include::modules/migration-using-mig-log-reader.adoc[leveloffset=+1]
2020
include::modules/migration-downloading-logs.adoc[leveloffset=+1]
2121
include::modules/migration-error-messages.adoc[leveloffset=+1]
2222
include::modules/migration-dvm-error-node-selectors.adoc[leveloffset=+1]
23+
include::modules/migration-debugging-velero-resources.adoc[leveloffset=+1]
24+
include::modules/migration-partial-failure-velero.adoc[leveloffset=+2]
2325
include::modules/migration-using-must-gather.adoc[leveloffset=+1]
2426

2527
[id="rolling-back-migration_{context}"]

‎modules/migration-debugging-velero-resources.adoc‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
// Module included in the following assemblies:
2+
// * migration/migrating_3_4/troubleshooting-3-4.adoc
3+
// * migration/migrating_4_1_4/troubleshooting-4-1-4.adoc
4+
// * migration/migrating_4_2_4/troubleshooting-4-2-4.adoc
5+
6+
[id="migration-debugging-velero-resources_{context}"]
7+
= Using the Velero CLI to debug Backup and Restore CRs
8+
9+
You can debug the `Backup` and `Restore` custom resources (CRs) and partial migration failures with the Velero command line interface (CLI). The Velero CLI runs in the `velero` pod.
10+
11+
[id="velero-command-syntax_{context}"]
12+
== Velero command syntax
13+
14+
Velero CLI commands use the following syntax:
15+
[source,terminal]
16+
----
17+
$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero <resource> <command> <resource_id>
18+
----
19+
20+
You can specify `velero-<pod> -n openshift-migration` in place of `$(oc get pods -n openshift-migration -o name | grep velero)`.
21+
22+
[id="help-command_{context}"]
23+
== Help command
24+
25+
The Velero `help` command lists all the Velero CLI commands:
26+
[source,terminal]
27+
----
28+
$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero --help
29+
----
30+
31+
[id="describe-command_{context}"]
32+
== Describe command
33+
34+
The Velero `describe` command provides a summary of warnings and errors associated with a Velero resource:
35+
[source,terminal]
36+
----
37+
$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero <resource> describe <resource_id>
38+
----
39+
40+
.Example
41+
[source,terminal]
42+
----
43+
$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
44+
----
45+
46+
[id="logs-command_{context}"]
47+
== Logs command
48+
49+
The Velero `logs` command provides the logs associated with a Velero resource:
50+
[source,terminal]
51+
----
52+
velero <resource> logs <resource_id>
53+
----
54+
55+
.Example
56+
[source,terminal]
57+
----
58+
$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
59+
----

‎modules/migration-partial-failure-velero.adoc‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
// Module included in the following assemblies:
2+
// * migration/migrating_3_4/troubleshooting-3-4.adoc
3+
// * migration/migrating_4_1_4/troubleshooting-4-1-4.adoc
4+
// * migration/migrating_4_2_4/troubleshooting-4-2-4.adoc
5+
6+
[id="migration-partial-failure-velero_{context}"]
7+
= Debugging a partial migration failure
8+
9+
You can debug a partial migration failure warning message by using the Velero CLI to examine the `Restore` custom resource (CR) logs.
10+
11+
A partial failure occurs when Velero encounters an issue that does not cause a migration to fail. For example, if a custom resource definition (CRD) is missing or if there is a discrepancy between CRD versions on the source and target clusters, the migration completes but the CR is not created on the target cluster.
12+
13+
Velero logs the issue as a partial failure and then processes the rest of the objects in the `Backup` CR.
14+
15+
.Procedure
16+
17+
. Check the status of a `MigMigration` CR:
18+
+
19+
[source,terminal]
20+
----
21+
$ oc get migmigration <migmigration> -o yaml
22+
----
23+
+
24+
.Example output
25+
+
26+
[source,yaml]
27+
----
28+
status:
29+
conditions:
30+
- category: Warn
31+
durable: true
32+
lastTransitionTime: "2021-01-26T20:48:40Z"
33+
message: 'Final Restore openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf: partially failed on destination cluster'
34+
status: "True"
35+
type: VeleroFinalRestorePartiallyFailed
36+
- category: Advisory
37+
durable: true
38+
lastTransitionTime: "2021-01-26T20:48:42Z"
39+
message: The migration has completed with warnings, please look at `Warn` conditions.
40+
reason: Completed
41+
status: "True"
42+
type: SucceededWithWarnings
43+
----
44+
45+
. Check the status of the `Restore` CR by using the Velero `describe` command:
46+
+
47+
[source,yaml]
48+
----
49+
$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -n openshift-migration -- ./velero restore describe <restore>
50+
----
51+
+
52+
.Example output
53+
+
54+
[source,yaml]
55+
----
56+
Phase: PartiallyFailed (run 'velero restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf' for more information)
57+
58+
Errors:
59+
Velero: <none>
60+
Cluster: <none>
61+
Namespaces:
62+
migration-example: error restoring example.com/migration-example/migration-example: the server could not find the requested resource
63+
----
64+
65+
. Check the `Restore` CR logs by using the Velero `logs` command:
66+
+
67+
[source,yaml]
68+
----
69+
$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -n openshift-migration -- ./velero restore logs <restore>
70+
----
71+
+
72+
.Example output
73+
+
74+
[source,yaml]
75+
----
76+
time="2021-01-26T20:48:37Z" level=info msg="Attempting to restore migration-example: migration-example" logSource="pkg/restore/restore.go:1107" restore=openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
77+
time="2021-01-26T20:48:37Z" level=info msg="error restoring migration-example: the server could not find the requested resource" logSource="pkg/restore/restore.go:1170" restore=openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
78+
----
79+
+
80+
The `Restore` CR log error message, `the server could not find the requested resource`, indicates the cause of the partially failed migration.

‎modules/migration-rolling-back-migration-cli.adoc‎

Lines changed: 7 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -6,20 +6,19 @@
66
[id='migration-rolling-back-migration-cli_{context}']
77
== Rolling back a migration from the CLI
88

9-
You can roll back a migration by using the CLI.
9+
You can roll back a migration by creating a `MigMigration` custom resource (CR) from the CLI.
1010

1111
If your application was stopped during a failed migration, you must roll back the migration in order to prevent data corruption in the persistent volume.
1212

1313
Rollback is not required if the application was not stopped during migration because the original application is still running on the source cluster.
1414

1515
.Procedure
1616

17-
. Create a `MigMigration` CR object based on the following example:
17+
. Create a `MigMigration` CR based on the following example:
1818
+
1919
[source,yaml]
2020
----
2121
$ cat << EOF | oc apply -f -
22-
---
2322
apiVersion: migration.openshift.io/v1alpha1
2423
kind: MigMigration
2524
metadata:
@@ -28,23 +27,15 @@ metadata:
2827
name: migration-rollback
2928
namespace: openshift-migration
3029
spec:
31-
# 'canceled: true' cancels the migration
32-
canceled: false
33-
# 'rollback: true' rolls back the migration
30+
...
3431
rollback: true
35-
# 'stage: true' runs a stage migration without quiescing the application on the source cluster.
36-
stage: false
37-
# 'quiescePods: true' scales the pods on the source cluster to '0' after the 'Backup' stage of a migration has finished
38-
quiescePods: false
39-
# 'keepAnnotations: true' retains the labels and annotations applied by the migration
40-
keepAnnotations: false
41-
32+
...
4233
migPlanRef:
43-
name: <migplan-name> <1>
34+
name: <migplan_name> <1>
4435
namespace: openshift-migration
4536
EOF
4637
----
47-
<1> Specify the name of the migration plan that you want to roll back.
38+
<1> Specify the name of the associated `MigPlan` CR.
4839

49-
. In the {mtc-short} console, verify that the migrated project resources have been removed from the target cluster.
40+
. In the {mtc-short} web console, verify that the migrated project resources have been removed from the target cluster.
5041
. Verify that the migrated project resources are present in the source cluster and that the application is running.

0 commit comments

Comments
 (0)