Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
6dd6b2c
PBM-1631 Documented the minio storage type
nastena1606 Oct 17, 2025
4adfcab
Update docs/reference/configuration-options.md
nastena1606 Oct 20, 2025
397971d
Update docs/reference/configuration-options.md
nastena1606 Oct 20, 2025
ea765d6
PBM-1631 Documented the minio storage type
nastena1606 Oct 17, 2025
3901844
Merge remote-tracking branch 'upstream/PBM-1631-Doc-minio-storage' in…
nastena1606 Oct 20, 2025
a59eb1e
Update docs/details/endpoint-map.md
nastena1606 Oct 20, 2025
721b14c
Update docs/details/minio.md
nastena1606 Oct 20, 2025
66fd87b
Update docs/details/minio.md
nastena1606 Oct 20, 2025
46b8065
Update docs/reference/configuration-options.md
nastena1606 Oct 20, 2025
b79710d
Update docs/reference/glossary.md
nastena1606 Oct 20, 2025
43f5d9d
Update docs/features/split-merge-backup.md
nastena1606 Oct 20, 2025
2024c4a
Update docs/details/minio.md
nastena1606 Oct 20, 2025
ed13d75
Update docs/reference/configuration-options.md
nastena1606 Oct 20, 2025
96da008
Apply suggestion from @nastena1606
nastena1606 Oct 21, 2025
4ae7c66
Updated linter dictionary
nastena1606 Oct 29, 2025
b4388c3
Updated after the review
nastena1606 Oct 30, 2025
a38a585
Update docs/reference/configuration-options.md
nastena1606 Nov 3, 2025
ec1c6fa
Update docs/reference/configuration-options.md
nastena1606 Nov 3, 2025
1658c54
Update docs/details/minio.md
nastena1606 Nov 3, 2025
f9151c2
Update docs/details/endpoint-map.md
nastena1606 Nov 3, 2025
416f7e0
fixed line breaks for options
nastena1606 Nov 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions docs/details/endpoint-map.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Multiple endpoints to the same storage

!!! admonition "Version added: [2.8.0](../release-notes/2.8.0.md)"

In environments where `pbm-agents` run on servers that are distributed across several data centers, accessing the same remote backup storage can become challenging. This can be due to complex network configurations or strict policies that prevent direct connections to the outside world. As a result, `pbm-agents` can't use the same URL to reach the storage, which is necessary for Percona Backup for MongoDB to work properly.

Check notice on line 5 in docs/details/endpoint-map.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/endpoint-map.md#L5

[Google.Passive] In general, use active voice instead of passive voice ('are distributed').
Raw output
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('are distributed').", "location": {"path": "docs/details/endpoint-map.md", "range": {"start": {"line": 5, "column": 56}}}, "severity": "INFO"}

To address these challenges, you can configure custom endpoint URLs for specific nodes in the PBM configuration. This allows all `pbm-agents` to access the same storage while respecting the network settings of their data centers.

The supported storage types are:

* AWS S3
* MinIO and S3-compatible storage services
* Microsoft Azure Blob storage

Here's the example of the configuration file with the endpoint map:

=== ":fontawesome-brands-amazon: AWS S3"

```yaml
storage:
type: s3
s3:
endpointUrl: http://S3:9000
endpointUrlMap:
"node01:27017": "did.socf.s3.com"
"node03:27017": "https://example.aws.s3.com"
...
```

=== ":simple-minio: MinIO and S3-compatible storage"

```yaml
storage:
type: minio
minio:
endpoint: localhost:9100
endpointMap:
"node01:27017": "did.socf.s3.com"
"node03:27017": "https://example.min.io"
...
```

=== ":material-microsoft-azure: Microsoft Azure Blob storage"

```yaml
storage:
type: azure
azure:
endpointUrl: https://myaccount.blob.core.windows.net
endpointUrlMap:
"node01:27017": "did.socf.blob.core.windows.net"
"node03:27017": "https://example.azure.blob.core.windows.net"
...
```


You can define the specific nodes for the `endpointUrlMap` parameter for AWS S3 and Azure or for the `endpointMap` for MinIO and S3-compatible storage. Not listed nodes use the endpoint defined for the `endpointUrl` / `endpoint` parameter.

For the solution to work, you should also have the mapping mechanism in place. This mechanism should be able to map the custom endpoints to the main endpoint URL of the storage, routing the requests from `pbm-agents` to the storage and back seamlessly.

With this ability to control the endpoints for `pbm-agents` to reach the same storage, you reduce the administrative overhead on PBM configuration and ensure its proper functioning.
133 changes: 133 additions & 0 deletions docs/details/minio.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# MinIO and S3-compatible storage

Check warning on line 1 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L1

[Google.Headings] 'MinIO and S3-compatible storage' should use sentence-style capitalization.
Raw output
{"message": "[Google.Headings] 'MinIO and S3-compatible storage' should use sentence-style capitalization.", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 1, "column": 3}}}, "severity": "WARNING"}

Percona Backup for MongoDB (PBM) works with both AWS S3 and other S3-compatible storage services. We test S3-compatible storages with PBM using [MinIO :octicons-link-external-16:](https://min.io/)

Check warning on line 3 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L3

[Google.We] Try to avoid using first-person plural like 'We'.
Raw output
{"message": "[Google.We] Try to avoid using first-person plural like 'We'.", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 3, "column": 99}}}, "severity": "WARNING"}

Check failure on line 3 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L3

[Vale.Spelling] Did you really mean 'storages'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'storages'?", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 3, "column": 121}}}, "severity": "ERROR"}

This document provides an overview of MinIO as the closest S3-compatible storage. To use the native AWS S3 service, see [AWS S3 storage](s3-storage.md).

[Configuration example :material-arrow-down:](#configuration-example){.md-button}

## Bucket creation

1. Install a [MinIO client :octicons-link-external-16:](https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc). After the installation, the `mc` is available for you.

2. Configure the `mc` command line tool with a MinIO Server

```{.bash data-prompt="$"}
$ mc alias set myminio http://127.0.0.1:9000 MINIO_ACCESS_KEY MINIO_SECRET_KEY
```

3. Create a bucket

```{.bash data-prompt="$"}
$ mc mb myminio/my-minio-bucket
```

4. Verify the bucket creation

```{.bash data-prompt="$"}
$ mc ls myminio
```

After the bucket is created, apply the proper [permissions for PBM to use the bucket](storage-configuration.md#permissions-setup).

Check notice on line 31 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L31

[Google.Passive] In general, use active voice instead of passive voice ('is created').
Raw output
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('is created').", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 31, "column": 18}}}, "severity": "INFO"}

## Configuration example

!!! important

Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#storage-bucket-creation) and managed solely by PBM.

This is the example for the basic configuration of MinIO and other S3-compatible storages in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields.

Check failure on line 39 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L39

[Vale.Spelling] Did you really mean 'storages'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'storages'?", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 39, "column": 82}}}, "severity": "ERROR"}

Check failure on line 39 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L39

[Vale.Spelling] Did you really mean 'uncomment'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'uncomment'?", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 39, "column": 308}}}, "severity": "ERROR"}

```yaml
storage:
type: minio
minio:
endpoint: localhost:9100
region: us-east-1
bucket: pbm-example
prefix: data/pbm/test
credentials:
access-key-id: <your-access-key-id-here>
secret-access-key: <your-secret-key-here>
```

For the description of configuration options, see [Configuration file options](../reference/configuration-options.md).

## Fine-tune storage configuration

The following sections describe how you can fine-tune your storage configuration:

* [debug logging](#debug-logging)
* [upload retries](#upload-retries)
* [data upload to storage with self-signed TLS certificates](#data-upload-to-storage-with-self-signed-tls-certificates)
* [multiple endpoints to the same S3 storage](endpoint-map.md)

### Debug logging

You can enable debug logging for different types of S3 requests in Percona Backup for MongoDB. Percona Backup for MongoDB prints S3 log messages in the `pbm logs` output so that you can debug and diagnose S3 request issues or failures.

To enable debug logging, set the `storage.minio.debugTrace` option in Percona Backup for MongoDB configuration. This instructs PBM to also print HTTP trace from the S3 storage in the logs.

## Upload retries

You can set up the number of attempts for Percona Backup for MongoDB to upload data to S3 storage as well as the min and max time to wait for the next retry. Set the `storage.minio.retryer.numMaxRetries` option in Percona Backup for MongoDB configuration.

```yaml
retryer:
numMaxRetries: 3
```

This upload retry increases the chances of data upload completion in cases of unstable connection.

## Data upload to storage with self-signed TLS certificates

Percona Backup for MongoDB supports data upload to S3-compatible storage service over HTTPS with a self-signed or a private CA certificate. This feature is especially important when you use services like MinIO, Ceph, or internal S3 gateways that don't use certificates signed by public Certificate Authorities (CAs).

Check failure on line 84 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L84

[Vale.Spelling] Did you really mean 'Ceph'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'Ceph'?", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 84, "column": 212}}}, "severity": "ERROR"}

Check notice on line 84 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L84

[Google.Parens] Use parentheses judiciously.
Raw output
{"message": "[Google.Parens] Use parentheses judiciously.", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 84, "column": 311}}}, "severity": "INFO"}

Check failure on line 84 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L84

[Vale.Spelling] Did you really mean 'CAs'?
Raw output
{"message": "[Vale.Spelling] Did you really mean 'CAs'?", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 84, "column": 312}}}, "severity": "ERROR"}

Providing a whole chain of certificates is recommended to ensure the connection is legit. The `SSL_CERT_FILE` environment variable specifies the path to a custom certificate chain file in PEM-format that PBM uses to validate TLS/SSL connection.

Check notice on line 86 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L86

[Google.Passive] In general, use active voice instead of passive voice ('is recommended').
Raw output
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('is recommended').", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 86, "column": 41}}}, "severity": "INFO"}

Check notice on line 86 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L86

[Google.Acronyms] Spell out 'PEM', if it's unfamiliar to the audience.
Raw output
{"message": "[Google.Acronyms] Spell out 'PEM', if it's unfamiliar to the audience.", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 86, "column": 189}}}, "severity": "INFO"}

### Usage example

Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path and your S3 endpoint is `https://minio.internal.local:9000`. To use self-issued TLS certificates, do the following:

Check warning on line 90 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L90

[Google.We] Try to avoid using first-person plural like 'Let's'.
Raw output
{"message": "[Google.We] Try to avoid using first-person plural like 'Let's'.", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 90, "column": 1}}}, "severity": "WARNING"}

1. Ensure the cert file is in PEM format. Use the following command to check it:

Check notice on line 92 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L92

[Google.Acronyms] Spell out 'PEM', if it's unfamiliar to the audience.
Raw output
{"message": "[Google.Acronyms] Spell out 'PEM', if it's unfamiliar to the audience.", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 92, "column": 31}}}, "severity": "INFO"}

```bash
cat /etc/ssl/minio-ca.crt
```

??? example "Sample output"


```{text .no-copy}
-----BEGIN CERTIFICATE-----
MIIC+TCCAeGgAwIBAgIJANH3WljB...
-----END CERTIFICATE-----
```

2. Set the `SSL_CERT_FILE` environment variable to that file's path on each host where `pbm-agent` and PBM CLI are running:

Check warning on line 107 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L107

[Google.WordList] Use 'command-line tool' instead of 'CLI'.
Raw output
{"message": "[Google.WordList] Use 'command-line tool' instead of 'CLI'.", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 107, "column": 108}}}, "severity": "WARNING"}

```{.bash data-prompt="$"}
$ export SSL_CERT_FILE=/etc/ssl/minio-ca.crt
```

If this variable isn't set, PBM uses the system root certificates.

3. Restart `pbm-agent`:

```{.bash data-prompt="$"}
$ sudo systemctl start pbm-agent
```

4. Verify that your custom certificate is recognized. Check PBM logs for successful S3 access.

Check notice on line 121 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L121

[Google.Passive] In general, use active voice instead of passive voice ('is recognized').
Raw output
{"message": "[Google.Passive] In general, use active voice instead of passive voice ('is recognized').", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 121, "column": 40}}}, "severity": "INFO"}


Alternatively, you can disable the TLS verification of the S3 storage in Percona Backup for MongoDB configuration:

Check warning on line 124 in docs/details/minio.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/details/minio.md#L124

[Google.WordList] Use 'turn off' or 'off' instead of 'disable'.
Raw output
{"message": "[Google.WordList] Use 'turn off' or 'off' instead of 'disable'.", "location": {"path": "docs/details/minio.md", "range": {"start": {"line": 124, "column": 24}}}, "severity": "WARNING"}

```{.bash data-prompt="$"}
$ pbm config --set storage.minio.insecureSkipTLSVerify=True
```

!!! warning

Use this option with caution as it might leave a hole for man-in-the-middle attacks.

Loading