K8SPSMDB-1237: add incremental backups #1836
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pooknull
requested review from
egegunes,
eleo007,
gkech,
hors,
jvpasinatto and
nmarukovich
as code owners
hors
requested changes
Feb 25, 2025
Collaborator
hors
left a comment
hors
left a comment
There was a problem hiding this comment.
@pooknull please add example into https://github.com/percona/percona-server-mongodb-operator/blob/main/deploy/cr.yaml
gkech
reviewed
Feb 26, 2025
gkech
reviewed
Feb 26, 2025
pkg/psmdb/backup/pbm.go
Outdated
Comment on lines
52
to
55
| const ( | ||
| BackupTypeIncrementalBase defs.BackupType = defs.IncrementalBackup + "-base" | ||
| ) | ||
|
|
Contributor
There was a problem hiding this comment.
We can also move this type to pkg/apis/psmdb/v1/perconaservermongodbbackup_types.go since it is more related to
type PerconaServerMongoDBBackupSpec struct {
...
...
....
// +kubebuilder:validation:Enum={logical,physical,incremental}
Type defs.BackupType `json:"type,omitempty"`
}
egegunes
requested changes
Feb 27, 2025
gkech
previously approved these changes
Mar 4, 2025
nmarukovich
previously approved these changes
Mar 4, 2025
egegunes
previously approved these changes
Mar 4, 2025
12 tasks
hors
added a commit
that referenced
this pull request
Mar 4, 2025
* K8SPSMDB-1219: PBM multi storage support
Operator always supported multiple storages but didn't have native
support for having multiple backup storages until v2.6.0. In operator we
were reconfiguring PBM every time user selects a storage for their
backups/restores different than the previous storage. This was causing
long wait periods esp. in storages with lots of backups due to resync
operation.
Another limitation was forcing users to have only 1 backup storage if
they want to enable point-in-time-recovery. In case of multiple
storages, PBM would upload oplog chunks to whatever storage is last used
by a backup/restore and this would make consistent recovery impossible.
PBM v2.6.0 added native support for multiple storages and these changes
introduce it to our operator:
* User can have one main storage in PBM configuration. Any other
storages can be added as profiles.
Main storage can be found in:
```
kubectl exec cluster1-rs0-0 -c backup-agent -- pbm config
```
This commit introduces a new field `main` in storage spec:
```
storages:
s3-us-west:
main: true
type: s3
s3:
bucket: operator-testing
credentialsSecret: cluster1-s3-secrets
region: us-west-2
````
If user only has 1 storage configured in `cr.yaml`, operator will
automatically use it as main storage. If more than 1 storage is
configured, one of them must have `main: true`. User can't have more
than 1 storage with `main: true`.
If user changes main storage in cr.yaml, operator will configure PBM
with the new storage and start resync.
Any other storage in `cr.yaml` will be added to PBM as a profile.
User can see profiles using cli:
```
kubectl exec cluster1-rs0-0 -c backup-agent -- pbm profile list
```
When user adds a new profile to `cr.yaml`, operator will add it to PBM
but won't start resync.
**`pbm config --force-resync` only start resync for the main storage.**
To manually resync a profile:
```
kubectl exec cluster1-rs0-0 -c backup-agent -- pbm profile sync <storage-name>
```
If user starts a restore using a backup in a storage configured as
profile, operator will start resync operation for profile and block
restore until resync finishes.
Note: Profiles are also called external storages in PBM documentation.
If user has multiple storages in `cr.yaml` and changes main storage
between them, operator:
1. configures PBM with the new main storage.
2. adds the old main as a profile.
2. deletes profile for the new main storage.
If user configures `backupSource` in backups/restores:
* if `cr.yaml` has no storages configured, operator configures PBM with
storage data in `backupSource` field. This storage will effectively be
the main storage until user adds a storage to `cr.yaml`. After a
storage is configured PBM configuration will be overwritten and
`backupSource` storage will be gone.
* if `cr.yaml` has a storage configured, operator adds `backupSource`
storage as a profile.
* Oplog chunks will be only be uploaded to main storage.
User can use any backup as base backup for point-in-time-recovery.
* Incremental backup chains all need to be stored in the same storage.
TBD after #1836 merged.
---
Other significant changes in operator behavior:
* Operator now configures automatically configures PBM on a fresh
cluster.
Before this changes, PBM was not configured until user starts a
backup/restore after deploying a fresh cluster. Now, PBM will directly
configured with main storage in `cr.yaml` and resync will be started
in the background.
There's a new field in `PerconaServerMongoDB` status:
`backupConfigHash`. Operator will maintain hash of the current PBM
configuration and reconfigures PBM if hash is changed. Fields in
`spec.backup.pitr` are excluded from hash calculation, they're handled
separately.
* If `PerconaServerMongoDB` is annotated with `percona.com/resync-pbm=true`,
operator will start resync operation both for main storage and
profiles. Resync for profiles are started with the equivalent of `pbm
profile sync --all`. These resync operations will be run in the
background and will not block the reconciliation.
* If a backup that has `percona.com/delete-backup` finalizer is deleted,
operator will only delete oplogs chunks if it's in main storage.
* fix users
* fix repeatedly sending backup commands
* fix users
* address review comments
* assert err in main storage unit test
* don't use break labels
---------
Co-authored-by: Viacheslav Sarzhan <[email protected]>
hors
requested changes
Mar 5, 2025
| # - name: weekly-s3-us-west-incremental | ||
| # enabled: false | ||
| # schedule: "0 1 * * *" | ||
| # keep: 5 |
Collaborator
There was a problem hiding this comment.
@pooknull please check how the retention works for incremental backups. We can implement it in next release but now we need to remove "keep" option from incremental.
hors
requested changes
Mar 5, 2025
| # enabled: false | ||
| # schedule: "0 1 * * *" | ||
| # keep: 5 | ||
| # type: incremental |
Collaborator
There was a problem hiding this comment.
@pooknull I have error when I try to use it
# perconaservermongodbs.psmdb.percona.com "some-name" was not valid:
# * spec.backup.tasks[0].type: Unsupported value: "incremental": supported values: "logical", "physical"
Collaborator
There was a problem hiding this comment.
we need to skip percona.com/delete-backup finalizer for incremental backups
If we delete the base backup, all incremental backups should be deleted as well
hors
requested changes
Mar 5, 2025
| # enabled: false | ||
| # schedule: "0 1 * * *" | ||
| # keep: 5 | ||
| # type: incremental |
Collaborator
There was a problem hiding this comment.
we need to skip percona.com/delete-backup finalizer for incremental backups
If we delete the base backup, all incremental backups should be deleted as well
hors
requested changes
Mar 5, 2025
pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go
Outdated
Show resolved
Hide resolved
github-actions
bot
added
the
dependencies
hors
approved these changes
Mar 6, 2025
egegunes
approved these changes
Mar 7, 2025
gkech
approved these changes
Mar 7, 2025
Collaborator
commit: ed8e6bc |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://perconadev.atlassian.net/browse/K8SPSMDB-1237
DESCRIPTION
This PR introduces support for incremental physical backups.
A new backup types for the field
.spec.typehave been added to thePerconaServerMongoDBBackupresource:incremental: makes an incremental backup. Important note:percona.com/delete-backupfinalizer and.spec.backup.tasks.[].keepoption are not available for this type.incremental-base: makes a full incremental backupCHECKLIST
Jira
Needs Doc) and QA (Needs QA)?Tests
compare/*-oc.yml)?Config/Logging/Testability