Skip to content

PBM-1490-Documented the ability to set file size limit for large backups #298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nastena1606 merged 3 commits into from
Sep 25, 2025
Merged

PBM-1490-Documented the ability to set file size limit for large backups #298

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4b0824a
PBM-1490-Documented the ability to set file size limit for large backups
nastena1606 Sep 9, 2025
35c00dc
Updated after the review
nastena1606 Sep 15, 2025
b55ee7f
Extended the warning with performance issues
nastena1606 Sep 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/styles/config/vocabularies/Percona/accept.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ mongosh
[Oo]plog
[Ss]hard
PBM
PLM
PMM
Percona Monitoring and Management
Percona Backup for MongoDB
Expand All @@ -20,3 +21,5 @@ SSE
UTC
MinIO
[Rr]esync
(?i)HMAC
retryer
2 changes: 1 addition & 1 deletion docs/details/storage-configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Percona Backup for MongoDB (PBM) saves backup data to a designated directory on

The oplog entries ensure backup consistency, and the end time of the oplog slice(s) is the data-consistent point in time of a backup snapshot.

Using the [`pbm list`](../reference/pbm-commands.md#pbm-list) or [`pbm status`](../reference/pbm-commands.md#pbm-status) commands, you can scan the backup directory to find existing backups, even if never used PBM on your computer before.
Using the [`pbm list`](../reference/pbm-commands.md#pbm-list) or [`pbm status`](../reference/pbm-commands.md#pbm-status) commands, you can scan the backup directory to find existing backups, even if you never used PBM on your computer before.

## Supported storage types

Expand Down
45 changes: 45 additions & 0 deletions docs/features/split-merge-backup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Manage large backup files upload

!!! admonition "Version added: [2.11.0](../release-notes/2.11.0.md)

As your database grows, so do your backups. Eventually, a collection or index may become so large that its backup file exceeds the maximum object size limit of your cloud or local storage. When this happens, Percona Backup for MongoDB (PBM) can't upload the file, which can disrupt your backup strategy.

The following table provides default maximum size limits for the supported backup storages:

Check failure on line 7 in docs/features/split-merge-backup.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/features/split-merge-backup.md#L7 

[Vale.Spelling] Did you really mean 'storages'?
Raw output 
{"message": "[Vale.Spelling] Did you really mean 'storages'?", "location": {"path": "docs/features/split-merge-backup.md", "range": {"start": {"line": 7, "column": 83}}}, "severity": "ERROR"}

| Storage | Default size limit|
| :--- | :--- |
| **AWS S3** | 4.9 TB |
| **GCS** | 4.9 TB |

Check notice on line 12 in docs/features/split-merge-backup.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/features/split-merge-backup.md#L12 

[Google.Acronyms] Spell out 'GCS', if it's unfamiliar to the audience.
Raw output 
{"message": "[Google.Acronyms] Spell out 'GCS', if it's unfamiliar to the audience.", "location": {"path": "docs/features/split-merge-backup.md", "range": {"start": {"line": 12, "column": 5}}}, "severity": "INFO"}
| **Azure Blob Storage** | 190 TB |
| **Filesystem storage** | 4.9 TB |

These defaults are sufficient to satisfy the majority of use cases. However, you can configure a new maximum size for backup files for the storage you use. To do this, define the file size in GB for the `maxObjSizeGB` configuration parameter.

Define new limits with caution, only when you are absolutely sure in your actions. Setting a too low value may cause a single backup file to be split into a too large number of smaller files and this may affect performance.

Check notice on line 18 in docs/features/split-merge-backup.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/features/split-merge-backup.md#L18 

[Google.Passive] In general, use active voice instead of passive voice ('be split').
Raw output 
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('be split').", "location": {"path": "docs/features/split-merge-backup.md", "range": {"start": {"line": 18, "column": 142}}}, "severity": "INFO"}

This is the configuration example for the AWS S3 storage:

```yaml
storage:
type: s3
s3:
region: us-west-2
bucket: pbm-test-bucket
prefix: data/pbm/backup
credentials:
access-key-id: <your-access-key-id-here>
secret-access-key: <your-secret-key-here>
maxObjSizeGB: 5018
```

## How it works

When a backup file exceeds the configured size limit, PBM does the following:

* Splits the file into pieces, each of which doesn't exceed the defined size limit. PBM respects your compression settings, so compressed backups stay compressed after the split
* Names each piece by adding the `pbmpart{number}` token to the filename. This lets PBM identify and manage all the pieces of a single backup.
* Uploads the pieces to the storage

When reading data for an operation like a restore, PBM automatically reverses this process: it retrieves all the pieces from storage, merges them back into a single file, and proceeds with the command. This makes the entire process transparent to your application.

Check warning on line 43 in docs/features/split-merge-backup.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/features/split-merge-backup.md#L43 

[Google.WordList] Use 'app' instead of 'application'.
Raw output 
{"message": "[Google.WordList] Use 'app' instead of 'application'.", "location": {"path": "docs/features/split-merge-backup.md", "range": {"start": {"line": 43, "column": 253}}}, "severity": "WARNING"}

With this ability to set the maximum file size, you can future-proof your backup strategy and gain peace of mind knowing your data is always safe and recoverable.
35 changes: 35 additions & 0 deletions docs/reference/configuration-options.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
numMaxRetries: 3
minRetryDelay: 30ms
maxRetryDelay: 5m
maxObjSizeGB: 5018
Copy link

Copilot AI Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The maxObjSizeGB parameter appears under the filesystem storage section but should be at the same indentation level as filesystem to be a sibling of the storage type configuration, not nested within it.

Copilot uses AI. Check for mistakes.
```

### storage.s3.provider
Expand Down Expand Up @@ -246,6 +247,14 @@

The maximum time to wait before the next retry, specified as a *time.Duration*. Units like ms, s, etc., are supported. Defaults to nanoseconds if no unit is provided.

### storage.s3.maxObjSizeGB

Check warning on line 250 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L250 

[Google.Headings] 'storage.s3.maxObjSizeGB' should use sentence-style capitalization.
Raw output 
{"message": "[Google.Headings] 'storage.s3.maxObjSizeGB' should use sentence-style capitalization.", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 250, "column": 5}}}, "severity": "WARNING"}

*Type*: float64 <br>
*Required*: NO <br>

Check warning on line 253 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L253 

[Google.Colons] ': N' should be in lowercase.
Raw output 
{"message": "[Google.Colons] ': N' should be in lowercase.", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 253, "column": 11}}}, "severity": "WARNING"}
*Default*: 5018

The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the limit. Read more about [Managing large backup files](../features/split-merge-backup.md).

Check notice on line 256 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L256 

[Google.Passive] In general, use active voice instead of passive voice ('be stored').
Raw output 
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('be stored').", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 256, "column": 26}}}, "severity": "INFO"}

## GCS type storage options

```yaml
Expand All @@ -260,6 +269,7 @@
privateKey: <your-private-key-here>
hmacAccessKey: <your-HMAC-key-here>
hmacSecret: <your-HMAC-secret-here>
maxObjSizeGB: 5018
```

### storage.gcs.bucket
Expand Down Expand Up @@ -332,13 +342,22 @@

Each time PBM fails and tries again, it increases the wait time by multiplying it by this number (usually 2). For example, if the first wait time is 1 second, the next will be 2 seconds, then 4 seconds, and so on, until it reaches the maximum. Default value is 2 sec.

### storage.gcs.maxObjSizeGB

Check warning on line 345 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L345 

[Google.Headings] 'storage.gcs.maxObjSizeGB' should use sentence-style capitalization.
Raw output 
{"message": "[Google.Headings] 'storage.gcs.maxObjSizeGB' should use sentence-style capitalization.", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 345, "column": 5}}}, "severity": "WARNING"}

*Type*: float64 <br>
*Required*: NO <br>

Check warning on line 348 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L348 

[Google.Colons] ': N' should be in lowercase.
Raw output 
{"message": "[Google.Colons] ': N' should be in lowercase.", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 348, "column": 11}}}, "severity": "WARNING"}
*Default*: 5018

The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the defined limit. Read more about [Managing large backup files](../features/split-merge-backup.md).

Check notice on line 351 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L351 

[Google.Passive] In general, use active voice instead of passive voice ('be stored').
Raw output 
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('be stored').", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 351, "column": 26}}}, "severity": "INFO"}

## Filesystem storage options

```yaml
storage:
type: filesystem
filesystem:
path: <string>
maxObjSizeGB: 5018
```

### storage.filesystem.path
Expand All @@ -348,6 +367,14 @@

The path to the backup directory

### storage.filesystem.maxObjSizeGB

Check warning on line 370 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L370 

[Google.Headings] 'storage.filesystem.maxObjSizeGB' should use sentence-style capitalization.
Raw output 
{"message": "[Google.Headings] 'storage.filesystem.maxObjSizeGB' should use sentence-style capitalization.", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 370, "column": 5}}}, "severity": "WARNING"}

*Type*: float64 <br>
*Required*: NO <br>

Check warning on line 373 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L373 

[Google.Colons] ': N' should be in lowercase.
Raw output 
{"message": "[Google.Colons] ': N' should be in lowercase.", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 373, "column": 11}}}, "severity": "WARNING"}
*Default*: 5018

The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the defined limit. Read more about [Managing large backup files](../features/split-merge-backup.md).

Check notice on line 376 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L376 

[Google.Passive] In general, use active voice instead of passive voice ('be stored').
Raw output 
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('be stored').", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 376, "column": 26}}}, "severity": "INFO"}

## Microsoft Azure Blob storage options

```yaml
Expand All @@ -360,6 +387,7 @@
prefix: <string>
credentials:
key: <your-access-key>
maxObjSizeGB: 194560
```

### storage.azure.account
Expand Down Expand Up @@ -397,3 +425,10 @@

Your access key to authorize access to data in your storage account.

### storage.azure.maxObjSizeGB

Check warning on line 428 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L428 

[Google.Headings] 'storage.azure.maxObjSizeGB' should use sentence-style capitalization.
Raw output 
{"message": "[Google.Headings] 'storage.azure.maxObjSizeGB' should use sentence-style capitalization.", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 428, "column": 5}}}, "severity": "WARNING"}

*Type*: float64 <br>
*Required*: NO <br>

Check warning on line 431 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L431 

[Google.Colons] ': N' should be in lowercase.
Raw output 
{"message": "[Google.Colons] ': N' should be in lowercase.", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 431, "column": 11}}}, "severity": "WARNING"}
*Default*: 194560

The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the defined limit. Read more about [Managing large backup files](../features/split-merge-backup.md).

Check notice on line 434 in docs/reference/configuration-options.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/reference/configuration-options.md#L434 

[Google.Passive] In general, use active voice instead of passive voice ('be stored').
Raw output 
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('be stored').", "location": {"path": "docs/reference/configuration-options.md", "range": {"start": {"line": 434, "column": 26}}}, "severity": "INFO"}
1 change: 1 addition & 0 deletions mkdocs-base.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,7 @@ nav:
- details/filesystem-storage.md
- details/azure.md
- features/multi-storage.md
- features/split-merge-backup.md
- PBM commands: reference/pbm-commands.md
- Configuration file:
- Percona Backup for MongoDB configuration in a cluster: reference/config.md
Expand Down