Skip to content

Commit a054ca4

Browse files
RichardHochRichardHoch
committed
PodVolumeBackup opt-in, opt-out
1 parent 7e5ecae commit a054ca4

File tree

6 files changed

+135
-45
lines changed

6 files changed

+135
-45
lines changed

‎backup_and_restore/application_backup_and_restore/oadp-advanced-topics.adoc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,10 @@ include::modules/oadp-using-enable-api-group-versions.adoc[leveloffset=+2]
1919
== Backing up data from one cluster and restoring it to another cluster
2020

2121
include::modules/oadp-about-backing-and-restoring-from-cluster-to-cluster.adoc[leveloffset=+2]
22+
include::modules/oadp-pod-volume-backup.adoc[leveloffset=+2]
23+
include::modules/oadp-backing-up-opt-in.adoc[leveloffset=+3]
24+
include::modules/oadp-backing-up-opt-out.adoc[leveloffset=+3]
25+
include::modules/oadp-cluster-to-cluster-uid-and-gid-ranges.adoc[leveloffset=+2]
2226
include::modules/oadp-backing-and-restoring-from-cluster-to-cluster.adoc[leveloffset=+2]
2327

2428
[role="_additional-resources"]

‎modules/oadp-about-backing-and-restoring-from-cluster-to-cluster.adoc

Lines changed: 1 addition & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -24,58 +24,14 @@ You must exclude Operators from the backup of an application for backup and rest
2424
[id="oadp-cluster-to-cluster-velero_{context}"]
2525
== Use of Velero
2626

27-
Velero, which OADP is built upon, does not natively support migrating persistent volume snapshots across cloud providers. To migrate volume snapshot data between cloud platforms, you must _either_ enable the Velero Restic file system backup option, which backs up volume contents at the filesystem level, _or_ use the OADP Data Mover for CSI snapshots.
27+
Velero, which OADP is built upon, does not natively support migrating persistent volume snapshots across cloud providers. To migrate volume snapshot data between cloud platforms, you must _either_ enable the Velero Restic file system backup option, which backs up volume contents at the file system level, _or_ use the OADP Data Mover for CSI snapshots.
2828

2929
[NOTE]
3030
====
3131
In OADP 1.1 and earlier, the Velero Restic file system backup option is called `restic`.
3232
In OADP 1.2 and later, the Velero Restic file system backup option is called `file-system-backup`.
3333
====
3434

35-
[NOTE]
36-
====
37-
Velero's file system backup feature supports both Kopia and Restic, but currently OADP supports only Restic.
38-
====
39-
4035
* You must also use Velero's link:https://velero.io/docs/main/file-system-backup/[File System Backup] to migrate data between AWS regions or between Microsoft Azure regions.
4136
* Velero does not support restoring data to a cluster with an _earlier_ Kubernetes version than the source cluster.
4237
* It is theoretically possible to migrate workloads to a destination with a _later_ Kubernetes version than the source, but you must consider the compatibility of API groups between clusters for each custom resource. If a Kubernetes version upgrade breaks the compatibility of core or native API groups, you must first update the impacted custom resources.
43-
44-
[id="oadp-cluster-to-cluster-uid-and-gid-ranges_{context}"]
45-
== UID and GID ranges
46-
47-
When you back up data from one cluster and restore it to another cluster, there are potential issues that might arise with UID (User ID) and GID (Group ID) ranges. The following section explains these potential issues and mitigations:
48-
49-
Summary of issues::
50-
The UID and GID ranges of the namespace might change on the destination cluster. OADP does not back up and restore OpenShift UID range metadata. If the backed application requires a specific UID, ensure the range is available when restored. For more information about OpenShift's UID and GID ranges, see link:https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids[A Guide to OpenShift and UIDs].
51-
52-
Detailed description of issues::
53-
When you create a namespace in {product-title} by using the shell command `oc create namespace`, {product-title} assigns the namespace a unique User ID (UID) range from its available pool of UIDs, a Supplemental Group (GID) range, and unique SELinux MCS labels. This information is stored in the `metadata.annotations` field of the cluster. This information is part of the Security Context Constraints (SCC) annotations, which comprise the following components:
54-
55-
* `openshift.io/sa.scc.mcs`
56-
* `openshift.io/sa.scc.supplemental-groups`
57-
* `openshift.io/sa.scc.uid-range`
58-
59-
+
60-
When you use OADP to restore the namespace, it automatically uses the information in `metadata.annotations` without resetting it for the destination cluster. As a result, the workload might not have access to the backed up data if one of the following is true:
61-
62-
* There is a pre-existing namespace with different SCC annotations, for example, on a different cluster. In this case, at backup time, OADP reuses the pre-existing namespace instead of the namespace you are trying to restore.
63-
* The backup used a label selector, but the namespace where workloads run on does not have the label on it. In this case, OADP does not back up the namespace, but instead creates a new namespace during restore that does not include the annotations of the namespace you backed up. This causes a new UID range to be assigned to the namespace.
64-
+
65-
This might be an issue for customer workloads if {product-title} assigns a pod a `securityContext` UID based on namespace annotations that have changed from the time the persistent volume data was backed up.
66-
* The container UID no longer matches the UID of the file owner.
67-
* An error occurs because {product-title} did not modify the UID range of the destination cluster to match the data of the backup cluster. As a result, the backup cluster has a different UID than the destination cluster, which means the application cannot read or write data to the destination cluster.
68-
69-
Mitigations::
70-
71-
You can use one or more of the following mitigations to resolve the UID and GID range issues:
72-
73-
* Simple mitigations:
74-
75-
** If you use a label selector in the `Backup` CR to filter the objects to include in the backup, be sure to add this label selector to the namespace that contains the workspace.
76-
** Remove any pre-existing version of a namespace on the destination cluster before attempting to restore a namespace with the same name.
77-
78-
* Advanced mitigations:
79-
** Fix UID ranges after migration by performing steps 1-4 of link:https://access.redhat.com/articles/6844071[Fixing UID ranges after migration]. Step 1 is optional.
80-
81-
For an in-depth discussion of UID and GID ranges in {product-title} with an emphasis on overcoming issues in backing up data on one cluster and restoring it on another, see link:https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids[A Guide to OpenShift and UIDs].

‎modules/oadp-backing-up-opt-in.adoc

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * backup_and_restore/application_backup_and_restore/advanced-topics.adoc
4+
5+
[id="oadp-backing-up-opt-in_{context}"]
6+
:_mod-docs-content-type: PROCEDURE
7+
= Backing up pod volumes by using the opt-in method
8+
9+
You can use the opt-in method to specify which volumes need to be backed up by File System Backup (FSB). You can do this by using the `backup.velero.io/backup-volumes` command.
10+
11+
.Procedure
12+
13+
* On each pod that contains one or more volumes that you want to back up, enter the following command:
14+
+
15+
[source,terminal]
16+
----
17+
$ oc -n <your_pod_namespace> annotate pod/<your_pod_name> \
18+
backup.velero.io/backup-volumes=<your_volume_name_1>, \ <your_volume_name_2>>,...,<your_volume_name_n>
19+
----
20+
+
21+
where:
22+
23+
`<your_volume_name_x>`:: specifies the name of the xth volume in the pod specification.

‎modules/oadp-backing-up-opt-out.adoc

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * backup_and_restore/application_backup_and_restore/advanced-topics.adoc
4+
5+
[id="oadp-backing-up-opt-out_{context}"]
6+
:_mod-docs-content-type: PROCEDURE
7+
= Backing up pod volumes by using the opt-out method
8+
9+
When using the opt-out approach, all pod volumes are backed up by using File System Backup (FSB), although there are some exceptions:
10+
11+
* Volumes that mount the default service account token, secrets, and configuration maps.
12+
13+
* `hostPath` volumes
14+
15+
You can use the opt-out method to specify which volumes *not* to back up. You can do this by using the `backup.velero.io/backup-volumes-excludes` command.
16+
17+
.Procedure
18+
19+
* On each pod that contains one or more volumes that you do not want to back up, run the following command:
20+
+
21+
[source,terminal]
22+
----
23+
$ oc -n <your_pod_namespace> annotate pod/<your_pod_name> \
24+
backup.velero.io/backup-volumes-excludes=<your_volume_name_1>, \ <your_volume_name_2>>,...,<your_volume_name_n>
25+
----
26+
+
27+
where:
28+
29+
`<your_volume_name_x>`:: specifies the name of the xth volume in the pod specification.
30+
31+
[NOTE]
32+
====
33+
You can enable this behavior for all Velero backups by running the `velero install` command with the `--default-volumes-to-fs-backup` flag.
34+
====

‎modules/oadp-cluster-to-cluster-uid-and-gid-ranges.adoc

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * backup_and_restore/application_backup_and_restore/advanced-topics.adoc
4+
5+
6+
:_mod-docs-content-type: CONCEPT
7+
[id="oadp-cluster-to-cluster-uid-and-gid-ranges_{context}"]
8+
= UID and GID ranges
9+
10+
If you back up data from one cluster and restore it to another cluster, problems might occur with UID (User ID) and GID (Group ID) ranges. The following section explains these potential issues and mitigations:
11+
12+
Summary of the issues::
13+
The namespace UID and GID ranges might change depending on the destination cluster. OADP does not back up and restore OpenShift UID range metadata. If the backed up application requires a specific UID, ensure the range is availableupon restore. For more information about OpenShift's UID and GID ranges, see link:https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids[A Guide to OpenShift and UIDs].
14+
15+
Detailed description of the issues::
16+
When you create a namespace in {product-title} by using the shell command `oc create namespace`, {product-title} assigns the namespace a unique User ID (UID) range from its available pool of UIDs, a Supplemental Group (GID) range, and unique SELinux MCS labels. This information is stored in the `metadata.annotations` field of the cluster. This information is part of the Security Context Constraints (SCC) annotations, which comprise of the following components:
17+
18+
* `openshift.io/sa.scc.mcs`
19+
* `openshift.io/sa.scc.supplemental-groups`
20+
* `openshift.io/sa.scc.uid-range`
21+
22+
When you use OADP to restore the namespace, it automatically uses the information in `metadata.annotations` without resetting it for the destination cluster. As a result, the workload might not have access to the backed up data if any of the following is true:
23+
24+
* There is an existing namespace with other SCC annotations, for example, on another cluster. In this case, OADP uses the existing namespace during the backup instead of the namespace you want to restore.
25+
* A label selector was used during the backup, but the namespace in which the workloads are executed does not have the label. In this case, OADP does not back up the namespace, but creates a new namespace during the restore that does not contain the annotations of the backed up namespace. This results in a new UID range being assigned to the namespace.
26+
+
27+
This can be an issue for customer workloads if {product-title} assigns a pod a `securityContext` UID to a pod based on namespace annotations that have changed since the persistent volume data was backed up.
28+
* The UID of the container no longer matches the UID of the file owner.
29+
* An error occurs because {product-title} has not changed the UID range of the destination cluster to match the backup cluster data. As a result, the backup cluster has a different UID than the destination cluster, which means that the application cannot read or write data on the destination cluster.
30+
31+
Mitigations::
32+
33+
You can use one or more of the following mitigations to resolve the UID and GID range issues:
34+
35+
* Simple mitigations:
36+
37+
** If you use a label selector in the `Backup` CR to filter the objects to include in the backup, be sure to add this label selector to the namespace that contains the workspace.
38+
** Remove any pre-existing version of a namespace on the destination cluster before attempting to restore a namespace with the same name.
39+
40+
* Advanced mitigations:
41+
** Fix UID ranges after migration by link:https://access.redhat.com/articles/6844071[Resolving overlapping UID ranges in OpenShift namespaces after migration]. Step 1 is optional.
42+
43+
For an in-depth discussion of UID and GID ranges in {product-title} with an emphasis on overcoming issues in backing up data on one cluster and restoring it on another, see link:https://cloud.redhat.com/blog/a-guide-to-openshift-and-uids[A Guide to OpenShift and UIDs].

‎modules/oadp-pod-volume-backup.adoc

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * backup_and_restore/application_backup_and_restore/advanced-topics.adoc
4+
5+
[id="oadp-pod-volume-backup_{context}"]
6+
:_mod-docs-content-type: CONCEPT
7+
= About determining which pod volumes to back up
8+
9+
Before you start a backup operation by using File System Backup (FSB), you must specify which pods contain a volume that you want to back up. Velero refers to this process as "discovering" the appropriate pod volumes.
10+
11+
Velero supports two approaches for determining pod volumes:
12+
13+
* *Opt-in approach*: The opt-in approach requires that you actively indicate that you want to include - _opt-in_ - a volume in a backup. You do this by labelling each pod that contains a volume to be backed up with the name of the volume. When Velero finds a persistent volume (PV), it checks the pod that mounted the volume. If the pod is labelled with the name of the volume, Velero backs up the pod.
14+
* *Opt-out approach*: With the opt-out approach, you must actively specify that you want to exclude a volume from a backup. You do this by labelling each pod that contains a volume you do not want to back up with the name of the volume. When Velero finds a PV, it checks the pod that mounted the volume. If the pod is labelled with the volume's name, Velero does not back up the pod.
15+
16+
[id=pod-volume-limitations_{context}]
17+
== Limitations
18+
19+
* FSB does not support backing up and restoring `hostpath` volumes. However, FSB does support backing up and restoring local volumes.
20+
* Velero uses a static, common encryption key for all backup repositories it creates. *This static key means that anyone who can access your backup storage can also decrypt your backup data*. It is essential that you limit access to backup storage.
21+
* For PVCs, every incremental backup chain is maintained across pod reschedules.
22+
+
23+
For pod volumes that are _not_ PVCs, such as `emptyDir` volumes, if
24+
a pod is deleted or recreated, for example, by a `ReplicaSet` or a deployment, the next backup of those volumes will be a full backup and not an incremental backup. It is assumed that the lifecycle of a pod volume is defined by its pod.
25+
* Even though backup data can be kept incrementally, backing up large files, such as a database, can take a long time. This is because FSB uses deduplication to find the difference that needs to be backed up.
26+
* FSB reads and writes data from volumes by accessing the file system of the node on which the pod is running. For this reason, FSB can only back up volumes that are mounted from a pod and not directly from a PVC. Some Velero users have overcome this limitation by running a staging pod, such as a BusyBox or Alpine container with an infinite sleep, to mount these PVC and PV pairs before performing a Velero backup..
27+
* FSB expects volumes to be mounted under `<hostPath>/<pod UID>`, with
28+
`<hostPath>` being configurable. Some Kubernetes systems, for example,
29+
vCluster, do not mount volumes under the `<pod UID>` subdirectory, and
30+
VFSB does not work with them as expected.

0 commit comments

Comments
 (0)