Skip to content

Commit 9cec9db

Browse files
RidRisRRidRisR
authored
br compact backup user document (pingcap#2676)
1 parent 82fc2e3 commit 9cec9db

8 files changed

Lines changed: 907 additions & 5 deletions

‎en/backup-restore-cr.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,33 @@ This section introduces the fields in the `Backup` CR.
258258
* `.spec.local.volume`: the persistent volume configuration.
259259
* `.spec.local.volumeMount`: the persistent volume mount configuration.
260260
261+
## CompactBackup CR fields
262+
263+
For TiDB v9.0.0 and later versions, you can use `CompactBackup` to accelerate PITR (Point-in-time recovery). To compact log backup data into structured SST files, you can create a custom `CompactBackup` CR object to define a backup task. The following introduces the fields in the `CompactBackup` CR:
264+
265+
* `.spec.startTs`: the start timestamp for log compaction backup.
266+
* `.spec.endTs`: the end timestamp for log compaction backup.
267+
* `.spec.concurrency`: the maximum number of concurrent log compaction tasks. The default value is `4`.
268+
* `.spec.maxRetryTimes`: the maximum number of retries for failed compaction tasks. The default value is `6`.
269+
* `.spec.toolImage`：the tool image used by `CompactBackup`. BR is the only tool image used in `CompactBackup`. When using BR for backup, you can specify the BR version with this field:
270+
271+
- If not specified or left empty, the `pingcap/br:${tikv_version}` image is used for backup by default.
272+
- If a BR version is specified, such as `.spec.toolImage: pingcap/br:v9.0.0`, the image of the specified version is used for backup.
273+
- If an image is specified without a version, such as `.spec.toolImage: private/registry/br`, the `private/registry/br:${tikv_version}` image is used for backup.
274+
275+
* `.spec.env`: the environment variables for the Pod that runs the compaction task.
276+
* `.spec.affinity`: the affinity configuration for the Pod that runs the compaction task. For details on affinity, refer to [Affinity and anti-affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity).
277+
* `.spec.tolerations`: specifies that the Pod that runs the compaction task can schedule onto nodes with matching [taints](https://kubernetes.io/docs/reference/glossary/?all=true#term-taint). For details on taints and tolerations, refer to [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/).
278+
* `.spec.podSecurityContext`: the security context configuration for the Pod that runs the compaction task, which allows the Pod to run as a non-root user. For details on `podSecurityContext`, refer to [Run Containers as a Non-root User](containers-run-as-non-root-user.md).
279+
* `.spec.priorityClassName`: the name of the priority class for the Pod that runs the compaction task, which sets priority for the Pod. For details on priority classes, refer to [Pod Priority and Preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/).
280+
* `.spec.imagePullSecrets`: the [imagePullSecrets](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod) for the Pod that runs the compaction task.
281+
* `.spec.serviceAccount`: the name of the ServiceAccount used for compact.
282+
* `.spec.useKMS`: whether to use AWS-KMS to decrypt the S3 storage key used for the backup.
283+
* `.spec.br`: BR-related configuration. For more information, refer to [BR fields](#br-fields).
284+
* `.spec.s3`: S3-related configuration. For more information, refer to [S3 storage fields](#s3-storage-fields).
285+
* `.spec.gcs`: GCS-related configuration. For more information, refer to [GCS fields](#gcs-fields).
286+
* `.spec.azblob`：Azure Blob Storage-related configuration. For more information, refer to [Azure Blob Storage fields](#azure-blob-storage-fields).
287+
261288
## Restore CR fields
262289
263290
To restore data to a TiDB cluster on Kubernetes, you can create a `Restore` CR object. For detailed restore process, refer to documents listed in [Restore data](backup-restore-overview.md#restore-data).
@@ -352,6 +379,7 @@ The `backupSchedule` configuration consists of three parts: the configuration of
352379
353380
* `backupTemplate`: the configuration of the snapshot backup. Specifies the configuration related to the cluster and remote storage of the snapshot backup, which is the same as the `spec` configuration of [the `Backup` CR](#backup-cr-fields).
354381
* `logBackupTemplate`: the configuration of the log backup. Specifies the configuration related to the cluster and remote storage of the log backup, which is the same as the `spec` configuration of [the `Backup` CR](#backup-cr-fields). The log backup is created and deleted along with `backupSchedule` and recycled according to `.spec.maxReservedTime`. The log backup name is saved in `status.logBackup`.
382+
* `compactBackupTemplate`: the configuration template of the log compaction backup. The fields are the same as those in the `spec` configuration of [the `CompactBackup` CR](#compactbackup-cr-fields). The compaction backup is created and deleted along with `backupSchedule`. The log backup names are stored in `status.logBackup`. The storage settings of the compaction backup should be the same as that of `logBackupTemplate` in the same `backupSchedule`.
355383
356384
> **Note:**
357385
>
@@ -362,4 +390,5 @@ The `backupSchedule` configuration consists of three parts: the configuration of
362390
* `.spec.maxBackups`: a backup retention policy, which determines the maximum number of backup files to be retained. When the number of backup files exceeds this value, the outdated backup file will be deleted. If you set this field to `0`, all backup items are retained.
363391
* `.spec.maxReservedTime`: a backup retention policy based on time. For example, if you set the value of this field to `24h`, only backup files within the recent 24 hours are retained. All backup files older than this value are deleted. For the time format, refer to [`func ParseDuration`](https://golang.org/pkg/time/#ParseDuration). If you have set `.spec.maxBackups` and `.spec.maxReservedTime` at the same time, the latter takes effect.
364392
* `.spec.schedule`: the time scheduling format of Cron. Refer to [Cron](https://en.wikipedia.org/wiki/Cron) for details.
393+
* `.spec.compactInterval`: the time interval used to trigger a new compaction task.
365394
* `.spec.pause`: `false` by default. If this field is set to `true`, the scheduled scheduling is paused. In this situation, the backup operation will not be performed even if the scheduling time point is reached. During this pause, the backup garbage collection runs normally. If you change `true` to `false`, the scheduled snapshot backup process is restarted. Because currently, log backup does not support pause, this configuration does not take effect for log backup.

‎en/backup-to-aws-s3-using-br.md‎

Lines changed: 147 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -491,8 +491,7 @@ In TiDB Operator v1.5.4, v1.6.0, and earlier versions, you can use the `logStop:
491491
backupMode: log
492492
logSubcommand: log-start/log-pause/log-stop
493493
br:
494-
cluster: demo1
495-
clusterNamespace: test1
494+
mespace: test1
496495
sendCredToTikv: true
497496
s3:
498497
provider: aws
@@ -538,6 +537,65 @@ In TiDB Operator v1.5.4, v1.6.0, and earlier versions, you can use the `logStop:
538537
demo1-log-backup-s3 log Stopped ... 2022-10-10T15:21:00+08:00
539538
```
540539

540+
### Compact log backup
541+
542+
For TiDB v9.0.0 and later versions, you can use a `CompactBackup` CR to compact log backup data into SST format, accelerating downstream PITR (Point-in-time recovery).
543+
544+
This section explains how to compact log backup based on the log backup example from previous sections.
545+
546+
1. In the `backup-test` namespace, create a `CompactBackup` CR named `demo1-compact-backup`.
547+
548+
```shell
549+
kubectl apply -f compact-backup-demo1.yaml
550+
```
551+
552+
The content of `compact-backup-demo1.yaml` is as follows:
553+
554+
```yaml
555+
---
556+
apiVersion: pingcap.com/v1alpha1
557+
kind: CompactBackup
558+
metadata:
559+
name: demo1-compact-backup
560+
namespace: backup-test
561+
spec:
562+
startTs: "***"
563+
endTs: "***"
564+
concurrency: 8
565+
maxRetryTimes: 2
566+
br:
567+
cluster: demo1
568+
clusterNamespace: test1
569+
sendCredToTikv: true
570+
s3:
571+
provider: aws
572+
secretName: s3-secret
573+
region: us-west-1
574+
bucket: my-bucket
575+
prefix: my-log-backup-folder
576+
```
577+
578+
The `startTs` and `endTs` fields specify the time range for the logs to be compacted by `demo1-compact-backup`. Any log that contains at least one write within this time range will be included in the compaction process. As a result, the final compacted data might include data written outside this range.
579+
580+
The `s3` settings should be the same as the storage settings of the log backup to be compacted. `CompactBackup` reads log files from the corresponding location and compact them.
581+
582+
#### View the status of log backup compaction
583+
584+
After creating the `CompactBackup` CR, TiDB Operator automatically starts compacting the log backup. You can check the backup status using the following command:
585+
586+
```shell
587+
kubectl get cpbk -n backup-test
588+
```
589+
590+
From the output, you can find the status of the `CompactBackup` CR named `demo1-compact-backup`. An example output is as follows:
591+
592+
```
593+
NAME STATUS PROGRESS MESSAGE
594+
demo1-compact-backup Complete [READ_META(17/17),COMPACT_WORK(1291/1291)]
595+
```
596+
597+
If the `STATUS` field displays `Complete`, the compact log backup process has finished successfully.
598+
541599
### Backup CR examples
542600

543601
<details>
@@ -917,6 +975,93 @@ The steps to prepare for a scheduled snapshot backup are the same as those of [P
917975
log-integrated-backup-schedule-s3 log Running ....
918976
```
919977

978+
## Integrated management of scheduled snapshot backup, log backup, and compact log backup
979+
980+
To accelerate downstream recovery, you can enable `CompactBackup` CR in the `BackupSchedule` CR. This feature periodically compacts log backup files in remote storage. You must enable log backup before using log backup compaction. This section extends the configuration from the previous section.
981+
982+
### Prerequisites: Prepare for a scheduled snapshot backup
983+
984+
The steps to prepare for a scheduled snapshot backup are the same as that of [Prepare for an ad-hoc backup](#prerequisites-prepare-for-an-ad-hoc-backup).
985+
986+
### Create `BackupSchedule`
987+
988+
1. Create a `BackupSchedule` CR named `integrated-backup-schedule-s3` in the `backup-test` namespace.
989+
990+
```shell
991+
kubectl apply -f integrated-backup-schedule-s3.yaml
992+
```
993+
994+
The content of `integrated-backup-schedule-s3.yaml` is as follows:
995+
996+
```yaml
997+
---
998+
apiVersion: pingcap.com/v1alpha1
999+
kind: BackupSchedule
1000+
metadata:
1001+
name: integrated-backup-schedule-s3
1002+
namespace: backup-test
1003+
spec:
1004+
maxReservedTime: "3h"
1005+
schedule: "* */2 * * *"
1006+
compactInterval: "1h"
1007+
backupTemplate:
1008+
backupType: full
1009+
cleanPolicy: Delete
1010+
br:
1011+
cluster: demo1
1012+
clusterNamespace: test1
1013+
sendCredToTikv: true
1014+
s3:
1015+
provider: aws
1016+
secretName: s3-secret
1017+
region: us-west-1
1018+
bucket: my-bucket
1019+
prefix: my-folder-snapshot
1020+
logBackupTemplate:
1021+
backupMode: log
1022+
br:
1023+
cluster: demo1
1024+
clusterNamespace: test1
1025+
sendCredToTikv: true
1026+
s3:
1027+
provider: aws
1028+
secretName: s3-secret
1029+
region: us-west-1
1030+
bucket: my-bucket
1031+
prefix: my-folder-log
1032+
compactBackupTemplate:
1033+
br:
1034+
cluster: demo1
1035+
clusterNamespace: test1
1036+
sendCredToTikv: true
1037+
s3:
1038+
provider: aws
1039+
secretName: s3-secret
1040+
region: us-west-1
1041+
bucket: my-bucket
1042+
prefix: my-folder-log
1043+
```
1044+
1045+
In the preceding example of `integrated-backup-schedule-s3.yaml`, the `backupSchedule` configuration is based on the previous section, with the following additions for `compactBackup`:
1046+
1047+
* Added the `BackupSchedule.spec.compactInterval` field to specify the interval for log backup compaction. It is recommended not to exceed the interval of scheduled snapshot backups and to keep it between one-half to one-third of the scheduled snapshot backup interval.
1048+
1049+
* Added the `BackupSchedule.spec.compactBackupTemplate` field. Ensure that the `BackupSchedule.spec.compactBackupTemplate.s3` configuration matches the `BackupSchedule.spec.logBackupTemplate.s3` configuration.
1050+
1051+
For the field description of `backupSchedule`, refer to [BackupSchedule CR fields](backup-restore-cr.md#backupschedule-cr-fields).
1052+
1053+
2. After creating `backupSchedule`, use the following command to check the backup status:
1054+
1055+
```shell
1056+
kubectl get bks -n backup-test -o wide
1057+
```
1058+
1059+
A compact log backup task is created together with `backupSchedule`. You can check the `CompactBackup` CR using the following command:
1060+
1061+
```shell
1062+
kubectl get cpbk -n backup-test
1063+
```
1064+
9201065
## Delete the backup CR
9211066

9221067
If you no longer need the backup CR, refer to [Delete the Backup CR](backup-restore-overview.md#delete-the-backup-cr).

‎en/backup-to-azblob-using-br.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -449,6 +449,63 @@ In TiDB Operator v1.5.4, v1.6.0, and earlier versions, you can use the `logStop:
449449
demo1-log-backup log Complete ... 2022-10-10T15:21:00+08:00
450450
```
451451

452+
### Compact log backup
453+
454+
For TiDB v9.0.0 and later versions, you can use a `CompactBackup` CR to compact log backup data into SST format, accelerating downstream PITR (Point-in-time recovery).
455+
456+
This section explains how to compact log backup based on the log backup example from previous sections.
457+
458+
1. In the `backup-test` namespace, create a `CompactBackup` CR named `demo1-compact-backup`.
459+
460+
```shell
461+
kubectl apply -f compact-backup-demo1.yaml
462+
```
463+
464+
The content of `compact-backup-demo1.yaml` is as follows:
465+
466+
```yaml
467+
---
468+
apiVersion: pingcap.com/v1alpha1
469+
kind: CompactBackup
470+
metadata:
471+
name: demo1-compact-backup
472+
namespace: backup-test
473+
spec:
474+
startTs: "***"
475+
endTs: "***"
476+
concurrency: 8
477+
maxRetryTimes: 2
478+
br:
479+
cluster: demo1
480+
clusterNamespace: test1
481+
sendCredToTikv: true
482+
azblob:
483+
secretName: azblob-secret
484+
container: my-container
485+
prefix: my-log-backup-folder
486+
```
487+
488+
The `startTs` and `endTs` fields specify the time range for the logs to be compacted by `demo1-compact-backup`. Any log that contains at least one write within this time range will be included in the compaction process. As a result, the final compacted data might include data written outside this range.
489+
490+
The `azblob` settings should be the same as the storage settings of the log backup to be compacted. `CompactBackup` reads log files from the corresponding location and compact them.
491+
492+
#### View the status of log backup compaction
493+
494+
After creating the `CompactBackup` CR, TiDB Operator automatically starts compacting the log backup. You can check the backup status using the following command:
495+
496+
```shell
497+
kubectl get cpbk -n backup-test
498+
```
499+
500+
From the output, you can find the status of the `CompactBackup` CR named `demo1-compact-backup`. An example output is as follows:
501+
502+
```
503+
NAME STATUS PROGRESS MESSAGE
504+
demo1-compact-backup Complete [READ_META(17/17),COMPACT_WORK(1291/1291)]
505+
```
506+
507+
If the `STATUS` field displays `Complete`, the compact log backup process has finished successfully.
508+
452509
### Backup CR examples
453510

454511
<details>
@@ -768,6 +825,89 @@ The steps to prepare for a scheduled snapshot backup are the same as those of [P
768825
log-integrated-backup-schedule-azblob log Running ....
769826
```
770827

828+
## Integrated management of scheduled snapshot backup, log backup, and compact log backup
829+
830+
To accelerate downstream recovery, you can enable `CompactBackup` CR in the `BackupSchedule` CR. This feature periodically compacts log backup files in remote storage. You must enable log backup before using log backup compaction. This section extends the configuration from the previous section.
831+
832+
### Prerequisites: Prepare for a scheduled snapshot backup
833+
834+
The steps to prepare for a scheduled snapshot backup are the same as that of [Prepare for an ad-hoc backup](#prerequisites-prepare-an-ad-hoc-backup-environment).
835+
836+
### Create `BackupSchedule`
837+
838+
1. Create a `BackupSchedule` CR named `integrated-backup-schedule-azblob` in the `backup-test` namespace.
839+
840+
```shell
841+
kubectl apply -f integrated-backup-schedule-azblob.yaml
842+
```
843+
844+
The content of `integrated-backup-schedule-azblob.yaml` is as follows:
845+
846+
```yaml
847+
---
848+
apiVersion: pingcap.com/v1alpha1
849+
kind: BackupSchedule
850+
metadata:
851+
name: integrated-backup-schedule-azblob
852+
namespace: backup-test
853+
spec:
854+
maxReservedTime: "3h"
855+
schedule: "* */2 * * *"
856+
backupTemplate:
857+
backupType: full
858+
cleanPolicy: Delete
859+
br:
860+
cluster: demo1
861+
clusterNamespace: test1
862+
sendCredToTikv: true
863+
azblob:
864+
secretName: azblob-secret
865+
container: my-container
866+
prefix: schedule-backup-folder-snapshot
867+
#accessTier: Hot
868+
logBackupTemplate:
869+
backupMode: log
870+
br:
871+
cluster: demo1
872+
clusterNamespace: test1
873+
sendCredToTikv: true
874+
azblob:
875+
secretName: azblob-secret
876+
container: my-container
877+
prefix: schedule-backup-folder-log
878+
#accessTier: Hot
879+
compactBackupTemplate:
880+
br:
881+
cluster: demo1
882+
clusterNamespace: test1
883+
sendCredToTikv: true
884+
azblob:
885+
secretName: azblob-secret
886+
container: my-container
887+
prefix: schedule-backup-folder-log
888+
#accessTier: Hot
889+
```
890+
891+
In the preceding example of `integrated-backup-schedule-azblob.yaml`, the `backupSchedule` configuration is based on the previous section, with the following additions for `compactBackup`:
892+
893+
* Added the `BackupSchedule.spec.compactInterval` field to specify the time interval for log backup compaction. It is recommended not to exceed the interval of scheduled snapshot backups and to keep it between one-half to one-third of the scheduled snapshot backup interval.
894+
895+
* Added the `BackupSchedule.spec.compactBackupTemplate` field. Ensure that the `BackupSchedule.spec.compactBackupTemplate.azblob` configuration matches the `BackupSchedule.spec.logBackupTemplate.azblob` configuration.
896+
897+
For the field description of `backupSchedule`, refer to [BackupSchedule CR fields](backup-restore-cr.md#backupschedule-cr-fields).
898+
899+
2. After creating `backupSchedule`, use the following command to check the backup status:
900+
901+
```shell
902+
kubectl get bks -n backup-test -o wide
903+
```
904+
905+
A compact log backup task is created together with `backupSchedule`. You can check the `CompactBackup` CR using the following command:
906+
907+
```shell
908+
kubectl get cpbk -n backup-test
909+
```
910+
771911
## Delete the backup CR
772912

773913
If you no longer need the backup CR, you can delete it by referring to [Delete the Backup CR](backup-restore-overview.md#delete-the-backup-cr).

0 commit comments

Comments
 (0)