Skip to content

Commit 3d57bee

Browse files
jeana-redhatjeana-redhat
authored
Merge pull request #66678 from kowen-rh/osdocs-6496
OSDOCS#6496: Document automated etcd backups
2 parents 5c6f482 + 195ed68 commit 3d57bee

File tree

2 files changed

+359
-0
lines changed

2 files changed

+359
-0
lines changed

‎backup_and_restore/control_plane_backup_and_restore/backing-up-etcd.adoc

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,3 +26,6 @@ include::modules/backup-etcd.adoc[leveloffset=+1]
2626
[id="additional-resources_backup-etcd"]
2727
== Additional resources
2828
* xref:../../hosted_control_planes/hcp-backup-restore-dr.adoc#hcp-backup-restore[Backing up and restoring etcd on a hosted cluster]
29+
30+
// Creating automated etcd backups
31+
include::modules/etcd-creating-automated-backups.adoc[leveloffset=+1]

‎modules/etcd-creating-automated-backups.adoc

Lines changed: 356 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,356 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * backup_and_restore/control_plane_backup_and_restore/backing-up-etcd.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="creating-automated-etcd-backups_{context}"]
7+
= Creating automated etcd backups
8+
9+
The automated backup feature for etcd supports both recurring and single backups. Recurring backups create a cron job that starts a single backup each time the job triggers.
10+
11+
:FeatureName: Automating etcd backups
12+
include::snippets/technology-preview.adoc[]
13+
14+
[id="enabling-automated-etcd-backups_{context}"]
15+
== Enabling automated etcd backups
16+
17+
Follow these steps to enable automated backups for etcd.
18+
19+
[WARNING]
20+
====
21+
Enabling the `TechPreviewNoUpgrade` feature set on your cluster prevents minor version updates. The `TechPreviewNoUpgrade` feature set cannot be disabled. Do not enable this feature set on production clusters.
22+
====
23+
24+
.Prerequisites
25+
26+
* You have access to the cluster as a user with the `cluster-admin` role.
27+
* You have access to the OpenShift CLI (`oc`).
28+
29+
.Procedure
30+
31+
. Create a `FeatureGate` custom resource (CR) file named `enable-tech-preview-no-upgrade.yaml` with the following contents:
32+
+
33+
[source,yaml]
34+
----
35+
apiVersion: config.openshift.io/v1
36+
kind: FeatureGate
37+
metadata:
38+
name: cluster
39+
spec:
40+
featureSet: TechPreviewNoUpgrade
41+
----
42+
43+
. Apply the CR and enable automated backups:
44+
+
45+
[source,terminal]
46+
----
47+
$ oc apply -f enable-tech-preview-no-upgrade.yaml
48+
----
49+
50+
. It takes time to enable the related APIs. Verify the creation of the custom resource definition (CRD) by running the following command:
51+
+
52+
[source,terminal]
53+
----
54+
$ oc get crd | grep backup
55+
----
56+
+
57+
.Example output
58+
[source,terminal]
59+
----
60+
backups.config.openshift.io 2023-10-25T13:32:43Z
61+
etcdbackups.operator.openshift.io 2023-10-25T13:32:04Z
62+
----
63+
64+
[id="creating-single-etcd-backup_{context}"]
65+
== Creating a single etcd backup
66+
67+
Follow these steps to create a single etcd backup by creating and applying a custom resource (CR).
68+
69+
.Prerequisites
70+
71+
* You have access to the cluster as a user with the `cluster-admin` role.
72+
* You have access to the OpenShift CLI (`oc`).
73+
* You have a PVC to save backup data to.
74+
75+
.Procedure
76+
77+
. Create a CR file named `etcd-single-backup.yaml` with contents such as the following example:
78+
+
79+
[source,yaml]
80+
----
81+
apiVersion: operator.openshift.io/v1alpha1
82+
kind: EtcdBackup
83+
metadata:
84+
name: etcd-single-backup
85+
namespace: openshift-etcd
86+
spec:
87+
pvcName: etcd-backup-pvc <.>
88+
----
89+
<.> The name of the persistent volume claim (PVC) to save the backup to. Adjust this value according to your environment.
90+
91+
. Apply the CR to start a single backup:
92+
+
93+
[source,terminal]
94+
----
95+
$ oc apply -f etcd-single-backup.yaml
96+
----
97+
98+
[id="creating-recurring-etcd-backups_{context}"]
99+
== Creating recurring etcd backups
100+
101+
Follow these steps to create automated recurring backups of etcd.
102+
103+
Use dynamically-provisioned storage to keep the created etcd backup data in a safe, external location if possible. If dynamically-provisioned storage is not available, consider storing the backup data on an NFS share to make backup recovery more accessible.
104+
105+
.Prerequisites
106+
107+
* You have access to the cluster as a user with the `cluster-admin` role.
108+
* You have access to the OpenShift CLI (`oc`).
109+
110+
.Procedure
111+
112+
. If dynamically-provisioned storage is available, complete the following steps to create automated recurring backups:
113+
114+
.. Create a persistent volume claim (PVC) named `etcd-backup-pvc.yaml` with contents such as the following example:
115+
+
116+
[source,yaml]
117+
----
118+
kind: PersistentVolumeClaim
119+
apiVersion: v1
120+
metadata:
121+
name: etcd-backup-pvc
122+
namespace: openshift-etcd
123+
spec:
124+
accessModes:
125+
- ReadWriteOnce
126+
resources:
127+
requests:
128+
storage: 200Gi <.>
129+
storageClassName: standard-csi <.>
130+
volumeMode: Filesystem
131+
----
132+
<.> The amount of storage available to the PVC. Adjust this value for your requirements.
133+
<.> The name of the `StorageClass` required by the claim. Adjust this value according to your environment.
134+
+
135+
[NOTE]
136+
====
137+
Each of the following providers require changes to the `accessModes` and `storageClassName` keys:
138+
139+
[cols="1,1,1"]
140+
|===
141+
|Provider|`accessModes` value|`storageClassName` value
142+
143+
|AWS with the `versioned-installer-efc_operator-ci` profile
144+
|`- ReadWriteMany`
145+
|`efs-sc`
146+
147+
|Google Cloud Platform
148+
|`- ReadWriteMany`
149+
|`filestore-csi`
150+
151+
|Microsoft Azure
152+
|`- ReadWriteMany`
153+
|`azurefile-csi`
154+
|===
155+
====
156+
157+
.. Apply the PVC by running the following command:
158+
+
159+
[source,terminal]
160+
----
161+
$ oc apply -f etcd-backup-pvc.yaml
162+
----
163+
164+
.. Verify the creation of the PVC by running the following command:
165+
+
166+
[source,terminal]
167+
----
168+
$ oc get pvc
169+
----
170+
+
171+
.Example output
172+
[source,terminal]
173+
----
174+
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
175+
etcd-backup-pvc Pending standard-csi 51s
176+
----
177+
+
178+
[NOTE]
179+
====
180+
Dynamic PVCs stay in the `Pending` state until they are mounted.
181+
====
182+
183+
. If dynamically-provisioned storage is unavailable, create a local storage PVC by completing the following steps:
184+
+
185+
[WARNING]
186+
====
187+
If you delete or otherwise lose access to the node that contains the stored backup data, you can lose data.
188+
====
189+
190+
.. Create a `StorageClass` CR file named `etcd-backup-local-storage.yaml` with the following contents:
191+
+
192+
[source,yaml]
193+
----
194+
apiVersion: storage.k8s.io/v1
195+
kind: StorageClass
196+
metadata:
197+
name: etcd-backup-local-storage
198+
provisioner: kubernetes.io/no-provisioner
199+
volumeBindingMode: WaitForFirstConsumer
200+
----
201+
202+
.. Apply the `StorageClass` CR by running the following command:
203+
+
204+
[source,terminal]
205+
----
206+
$ oc apply -f etcd-backup-local-storage.yaml
207+
----
208+
209+
.. Create a PV named `etcd-backup-pv-fs.yaml` from the applied `StorageClass` with content such as the following example:
210+
+
211+
[source,yaml]
212+
----
213+
apiVersion: v1
214+
kind: PersistentVolume
215+
metadata:
216+
name: etcd-backup-pv-fs
217+
spec:
218+
capacity:
219+
storage: 100Gi <.>
220+
volumeMode: Filesystem
221+
accessModes:
222+
- ReadWriteMany
223+
persistentVolumeReclaimPolicy: Delete
224+
storageClassName: local-storage
225+
local:
226+
path: /mnt/
227+
nodeAffinity:
228+
required:
229+
nodeSelectorTerms:
230+
- matchExpressions:
231+
- key: kubernetes.io/hostname
232+
operator: In
233+
values:
234+
- <example-master-node> <.>
235+
----
236+
<.> The amount of storage available to the PV. Adjust this value for your requirements.
237+
<.> Replace this value with the node to attach this PV to.
238+
+
239+
[TIP]
240+
====
241+
Run the following command to list the available nodes:
242+
243+
[source,terminal]
244+
----
245+
$ oc get nodes
246+
----
247+
====
248+
249+
.. Verify the creation of the PV by running the following command:
250+
+
251+
[source,terminal]
252+
----
253+
$ oc get pv
254+
----
255+
+
256+
.Example output
257+
[source,terminal]
258+
----
259+
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
260+
etcd-backup-pv-fs 100Gi RWX Delete Available local-storage 10s
261+
----
262+
263+
.. Create a PVC named `etcd-backup-pvc.yaml` with contents such as the following example:
264+
+
265+
[source,yaml]
266+
----
267+
kind: PersistentVolumeClaim
268+
apiVersion: v1
269+
metadata:
270+
name: etcd-backup-pvc
271+
spec:
272+
accessModes:
273+
- ReadWriteMany
274+
volumeMode: Filesystem
275+
resources:
276+
requests:
277+
storage: 10Gi <.>
278+
storageClassName: local-storage
279+
----
280+
<.> The amount of storage available to the PVC. Adjust this value for your requirements.
281+
282+
.. Apply the PVC by running the following command:
283+
+
284+
[source,terminal]
285+
----
286+
$ oc apply -f etcd-backup-pvc.yaml
287+
----
288+
289+
. Create a custom resource definition (CRD) file named `etcd-recurring-backups.yaml`. The contents of the created CRD define the schedule and retention type of automated backups.
290+
+
291+
For the default retention type of `RetentionNumber` with 15 retained backups, use contents such as the following example:
292+
+
293+
[source,yaml]
294+
----
295+
apiVersion: config.openshift.io/v1alpha1
296+
kind: Backup
297+
metadata:
298+
name: etcd-recurring-backup
299+
spec:
300+
etcd:
301+
schedule: "20 4 * * *" <.>
302+
timeZone: "UTC"
303+
pvcName: etcd-backup-pvc
304+
----
305+
<.> The `CronTab` schedule for recurring backups. Adjust this value for your needs.
306+
+
307+
To use retention based on the maximum number of backups, add the following key-value pairs to the `etcd` key:
308+
+
309+
[source,yaml]
310+
----
311+
spec:
312+
etcd:
313+
retentionPolicy:
314+
retentionType: RetentionNumber <.>
315+
retentionNumber:
316+
maxNumberOfBackups: 5 <.>
317+
----
318+
<.> The retention type. Defaults to `RetentionNumber` if unspecified.
319+
<.> The maximum number of backups to retain. Adjust this value for your needs. Defaults to 15 backups if unspecified.
320+
+
321+
[WARNING]
322+
====
323+
A known issue causes the number of retained backups to be one greater than the configured value.
324+
====
325+
+
326+
For retention based on the file size of backups, use the following:
327+
+
328+
[source,yaml]
329+
----
330+
spec:
331+
etcd:
332+
retentionPolicy:
333+
retentionType: RetentionSize
334+
retentionSize:
335+
maxSizeOfBackupsGb: 20 <.>
336+
----
337+
<.> The maximum file size of the retained backups in gigabytes. Adjust this value for your needs. Defaults to 10 GB if unspecified.
338+
+
339+
[WARNING]
340+
====
341+
A known issue causes the maximum size of retained backups to be up to 10 GB greater than the configured value.
342+
====
343+
344+
. Create the cron job defined by the CRD by running the following command:
345+
+
346+
[source,terminal]
347+
----
348+
$ oc create -f etcd-recurring-backup.yaml
349+
----
350+
351+
. To find the created cron job, run the following command:
352+
+
353+
[source,terminal]
354+
----
355+
$ oc get cronjob -n openshift-etcd
356+
----

0 commit comments

Comments
 (0)