From 6dd6b2cc0360c3865eab4ee1920f730819e45226 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Fri, 17 Oct 2025 19:30:41 +0200 Subject: [PATCH 01/20] PBM-1631 Documented the minio storage type new file: docs/details/endpoint-map.md new file: docs/details/minio.md modified: docs/details/s3-storage.md modified: docs/details/storage-configuration.md modified: docs/features/split-merge-backup.md modified: docs/install/backup-storage.md modified: docs/reference/configuration-options.md modified: docs/reference/glossary.md modified: docs/troubleshoot/index.md modified: mkdocs-base.yml --- docs/details/endpoint-map.md | 61 ++++++++ docs/details/minio.md | 147 +++++++++++++++++++ docs/details/s3-storage.md | 183 +++++------------------- docs/details/storage-configuration.md | 30 ++-- docs/features/split-merge-backup.md | 1 + docs/install/backup-storage.md | 14 ++ docs/reference/configuration-options.md | 158 +++++++++++++++++++- docs/reference/glossary.md | 11 ++ docs/troubleshoot/index.md | 1 + mkdocs-base.yml | 6 +- 10 files changed, 443 insertions(+), 169 deletions(-) create mode 100644 docs/details/endpoint-map.md create mode 100644 docs/details/minio.md diff --git a/docs/details/endpoint-map.md b/docs/details/endpoint-map.md new file mode 100644 index 00000000..b3212f14 --- /dev/null +++ b/docs/details/endpoint-map.md @@ -0,0 +1,61 @@ +# Multiple endpoints to the same storage + +!!! admonition "Version added: [2.8.0](../release-notes/2.8.0.md)" + +In environments where `pbm-agents` run on servers that are distributed across several data centers, accessing the same remote backup storage can become challenging. This can be due to complex network configurations or strict policies that prevent direct connections to the outside world. As a result, `pbm-agents` can't use the same URL to reach the storage, which is necessary for Percona Backup for MongoDB to work properly. + +To address these challenges, you can configure custom endpoint URLs for specific nodes in the PBM configuration. This allows all `pbm-agents` to access the same storage while respecting the network settings of their data centers. + +The supported storage types are: + +* AWS S3, +* MinIO and S3-compatible storage services +* Microsoft Azure Blob storage. + +Here's the example of the configuration file with the endpoint map: + +=== ":fontawesome-brands-amazon: AWS S3" + + ```yaml + storage: + type: s3 + s3: + endpointUrl: http://S3:9000 + endpointUrlMap: + "node01:27017": "did.socf.s3.com" + "node03:27017": "https://example.aws.s3.com" + ... + ``` + +=== ":simple-minio: MinIO and S3-compatible storage" + + ```yaml + storage: + type: minio + minio: + endpoint: localhost:9100 + endpointMap: + "node01:27017": "did.socf.s3.com" + "node03:27017": "https://example.min.io" + ... + ``` + +=== ":material-microsoft-azure: Microsoft Azure Blob storage" + + ```yaml + storage: + type: azure + azure: + endpointUrl: https://myaccount.blob.core.windows.net + endpointUrlMap: + "node01:27017": "did.socf.blob.core.windows.net" + "node03:27017": "https://example.azure.blob.core.windows.net" + ... + ``` + + +You can define the specific nodes for the `endpointUrlMap` parameter for AWS S3 and Azure or for the `endpointMap` for MinIO and S3-compatible storage. Not listed nodes use the endpoint defined for the `endpointUrl` / `endpoint` parameter. + +For the solution to work, you should also have the mapping mechanism in place. This mechanism should be able to map the custom endpoints to the main endpoint URL of the storage, routing the requests from `pbm-agents` to the storage and back seamlessly. + +With this ability to control the endpoints for `pbm-agents` to reach the same storage, you reduce the administrative overhead on PBM configuration and ensure its proper functioning. \ No newline at end of file diff --git a/docs/details/minio.md b/docs/details/minio.md new file mode 100644 index 00000000..46b745de --- /dev/null +++ b/docs/details/minio.md @@ -0,0 +1,147 @@ +# MinIO and S3-compatible storage + +Percona Backup for MongoDB (PBM) works with both AWS S3 and other S3-compatible storage services. We test S3-compatible storages with PBM using [MinIO :octicons-link-external-16:](https://min.io/) + +This document provides overview for MinIO as the closest S3-compatible storage. To use a native AWS S3 services, see [AWS S3 storage](s3-storage.md). + +[Configuration example :material-arrow-down:](#configuration-example){.md-button} + +## Bucket creation + +1. Install a [MinIO client :octicons-link-external-16:](https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc). After the installation, the `mc` is available for you. + +2. Configure the `mc` command line tool with a MinIO Server + + ```{.bash data-prompt="$"} + $ mc alias set myminio http://127.0.0.1:9000 MINIO_ACCESS_KEY MINIO_SECRET_KEY + ``` + +3. Create a bucket + + ```{.bash data-prompt="$"} + $ mc mb myminio/my-minio-bucket + ``` + +4. Verify the bucket creation + + ```{.bash data-prompt="$"} + $ mc ls myminio + ``` + +After the bucket is created, apply the proper [permissions for PBM to use the bucket](storage-configuration.md#permissions-setup). + +## Configuration example + +!!! important + + Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#storage-bucket-creation) and managed solely by PBM. + +This is the examples for the basic configuration of MinIO and other S3-compatible storages in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. + +```yaml +storage: + type: minio + minio: + endpoint: localhost:9100 + bucket: pbm-example + prefix: data/pbm/test + credentials: + access-key-id: + secret-access-key: +``` + +For the description of configuration options, see [Configuration file options](../reference/configuration-options.md). + +## Fine-tune storage configuration + +The following sections describe how you can fine-tune your storage configuration: + +* [debug logging](#debug-logging) +* [upload retries](#upload-retries) +* [data upload to storage with self-signed TLS certificates](#data-upload-to-storage-with-self-signed-tls-certificates) +* [multiple endpoints to the same S3 storage](endpoint-map.md) + +### Debug logging + +You can enable debug logging for different types of S3 requests in Percona Backup for MongoDB. Percona Backup for MongoDB prints S3 log messages in the `pbm logs` output so that you can debug and diagnose S3 request issues or failures. + +To enable S3 debug logging, set the `storage.minio.DebugLogLevel` option in Percona Backup for MongoDB configuration. The supported values are: `LogDebug`, `Signing`, `HTTPBody`, `RequestRetries`, `RequestErrors`, `EventStreamBody`. + +## Storage classes + +Percona Backup for MongoDB supports [Amazon S3 storage classes :octicons-link-external-16:](https://aws.amazon.com/s3/storage-classes/). Knowing your data access patterns, you can set the S3 storage class in Percona Backup for MongoDB configuration. When Percona Backup for MongoDB uploads data to S3, the data is distributed to the corresponding storage class. The support of S3 bucket storage types allows you to effectively manage S3 storage space and costs. + +To set the storage class, specify the `storage.s3.storageClass` option in Percona Backup for MongoDB configuration file: + +```yaml +storage: + type: minio + minio: + storageClass: REDUCED_REDUNDANCY +``` + +When the option is undefined, the MinIO Standard (`STANDARD`) storage class is used. + +## Upload retries + +You can set up the number of attempts for Percona Backup for MongoDB to upload data to S3 storage as well as the min and max time to wait for the next retry. Set the `storage.minio.retryer.numMaxRetries` option in Percona Backup for MongoDB configuration. + +```yaml +retryer: + numMaxRetries: 3 +``` + +This upload retry increases the chances of data upload completion in cases of unstable connection. + +## Data upload to storage with self-signed TLS certificates + +Percona Backup for MongoDB supports data upload to S3-compatible storage service over HTTPS with a self-signed or a private CA certificate. This feature is especially important when you use services like MinIO, Ceph, or internal S3 gateways that don't use certificates signed by public Certificate Authorities (CAs). + +Providing a whole chain of certificates is recommended to ensure the connection is legit. The `SSL_CERT_FILE` environment variable specifies the path to a custom certificate chain file in PEM-format that PBM uses to validate TLS/SSL connection. + +### Usage example + +Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path and your S3 endpoint is `https://minio.internal.local:9000`. To use self-issued TLS certificates, do the following: + +1. Ensure the cert file is in PEM format. Use the following command to check it: + + ```bash + cat /etc/ssl/minio-ca.crt + ``` + + ??? example "Sample output" + + + ```{text .no-copy} + -----BEGIN CERTIFICATE----- + MIIC+TCCAeGgAwIBAgIJANH3WljB... + -----END CERTIFICATE----- + ``` + +2. Set the `SSL_CERT_FILE` environment variable to that file's path on each host where `pbm-agent` and PBM CLI are running: + + ```{.bash data-prompt="$"} + $ export SSL_CERT_FILE=/etc/ssl/minio-ca.crt + ``` + + If this variable isn't set, PBM uses the system root certificates. + +3. Restart `pbm-agent`: + + ```{.bash data-prompt="$"} + $ sudo systemctl start pbm-agent + ``` + +4. Verify that your custom certificate is recognized. Check PBM logs for successful S3 access. + + +Alternatively, you can disable the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: + +```{.bash data-prompt="$"} +$ pbm config --set storage.s3.insecureSkipTLSVerify=True +``` + +!!! warning + + Use this option with caution as it might leave a hole for man-in-the-middle attacks. + diff --git a/docs/details/s3-storage.md b/docs/details/s3-storage.md index 5616f076..90808b4b 100644 --- a/docs/details/s3-storage.md +++ b/docs/details/s3-storage.md @@ -1,76 +1,43 @@ -# AWS or S3-compatible storage +# AWS S3 storage Percona Backup for MongoDB (PBM) works with AWS S3 and other S3-compatible storage services. We test PBM with the following services: * [Amazon Simple Storage Service :octicons-link-external-16:](https://docs.aws.amazon.com/s3/index.html) * [MinIO :octicons-link-external-16:](https://min.io/) -[Configuration examples :material-arrow-down:](#configuration-examples){.md-button} +This document provides overview for the native AWS S3 services. To use MinIO and other S3-compatible storage services, see [S3-compatible storage](minio.md). -## Storage bucket creation - -Here are some examples of the steps required to create a bucket. - -=== ":material-aws: Amazon S3" - - 1. Install and configure [AWS CLI :octicons-link-external-16:](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) - - 2. Create an S3 bucket +[Configuration example :material-arrow-down:](#configuration-example){.md-button} - ```{.bash data-prompt="$"} - $ aws s3api create-bucket --bucket my-s3-bucket --region us-east-1 - ``` - - 3. Verify the bucket creation - - ```{.bash data-prompt="$"} - $ aws s3 ls - ``` - -=== ":simple-minio: MinIo" +## Storage bucket creation - 1. Install a [MinIO client :octicons-link-external-16:](https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc). After the installation, the `mc` is available for you. +To create a bucket, do the following. - 2. Configure the `mc` command line tool with a MinIO Server +1. Install and configure [AWS CLI :octicons-link-external-16:](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) - ```{.bash data-prompt="$"} - $ mc alias set myminio http://127.0.0.1:9000 MINIO_ACCESS_KEY MINIO_SECRET_KEY - ``` - - 3. Create a bucket +2. Create an S3 bucket - ```{.bash data-prompt="$"} - $ mc mb myminio/my-minio-bucket - ``` - - 4. Verify the bucket creation + ```{.bash data-prompt="$"} + $ aws s3api create-bucket --bucket my-s3-bucket --region us-east-1 + ``` + +3. Verify the bucket creation - ```{.bash data-prompt="$"} - $ mc ls myminio - ``` + ```{.bash data-prompt="$"} + $ aws s3 ls + ``` After the bucket is created, apply the proper [permissions for PBM to use the bucket](storage-configuration.md#permissions-setup). -## Configuration examples +## Configuration example !!! important Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#storage-bucket-creation) and managed solely by PBM. -These are the examples for the basic configuration of S3-compatible storage in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. - -The following sections describe how you can fine-tune your storage configuration: - -* [server-side encryption](#server-side-encryption), -* [multiple endpoints to the same S3 storage](#multiple-endpoints-to-the-same-s3-storage), -* [debug logging](#debug-logging), -* [storage classes](#storage-classes), -* [upload retries](#upload-retries), -* [data upload to storage with self-signed TLS certificates](#data-upload-to-storage-with-self-signed-tls-certificates). +This is the example for the basic configuration of AWS S3 storage in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. -### Amazon Simple Storage Service - ```yaml storage: type: s3 @@ -86,25 +53,19 @@ storage: kmsKeyID: ``` -### MinIO +For the description of configuration options, see [Configuration file options](../reference/configuration-options.md). -```yaml -storage: - type: s3 - s3: - endpointUrl: "http://localhost:9000" - region: my-region - bucket: pbm-example - prefix: data/pbm/test - credentials: - access-key-id: - secret-access-key: -``` +## Fine-tune storage configuration -For the description of configuration options, see [Configuration file options](../reference/configuration-options.md). +The following sections describe how you can fine-tune your storage configuration: +* [server-side encryption](#server-side-encryption) +* [debug logging](#debug-logging) +* [storage classes](#storage-classes) +* [upload retries](#upload-retries) +* [multiple endpoints to the same S3 storage](endpoint-map.md) -## Server-side encryption +### Server-side encryption Percona Backup for MongoDB supports [server-side encryption](../reference/glossary.md#server-side-encryption) for [S3 buckets](../reference/glossary.md#bucket) with the following encryption types: @@ -112,7 +73,7 @@ Percona Backup for MongoDB supports [server-side encryption](../reference/glossa * [customer-provided keys stored on the client side (SSE-C)](#using-customer-provided-keys-sse-c) * [Amazon S3 managed encryption keys (SSE-S3)](#using-amazon-s3-managed-keys-sse-s3) -### Using AWS KMS keys (SSE-KMS) +#### Using AWS KMS keys (SSE-KMS) To use the SSE-KMS encryption, specify the following parameters in the Percona Backup for MongoDB configuration file: @@ -122,7 +83,7 @@ serverSideEncryption: sseAlgorithm: aws:kms ``` -### Using customer-provided keys (SSE-C) +#### Using customer-provided keys (SSE-C) !!! admonition "Version added: [2.0.1](../release-notes/2.0.1.md)" @@ -141,7 +102,7 @@ serverSideEncryption: sseCustomerKey: ``` -### Using Amazon S3 managed keys (SSE-S3) +#### Using Amazon S3 managed keys (SSE-S3) !!! admonition "Version added: [2.6.0](../release-notes/2.6.0.md)" @@ -154,42 +115,13 @@ serverSideEncryption: sseAlgorithm: AES256 ``` -## Multiple endpoints to the same S3 storage - -!!! admonition "Version added: [2.8.0](../release-notes/2.8.0.md)" - -In environments where `pbm-agents` run on servers that are distributed across several data centers, accessing the same remote backup storage can become challenging. This can be due to complex network configurations or strict policies that prevent direct connections to the outside world. As a result, `pbm-agents` can't use the same URL to reach the storage, which is necessary for Percona Backup for MongoDB to work properly. - -To address these challenges, you can configure custom endpoint URLs for specific nodes in the PBM configuration. This allows all `pbm-agents` to access the same storage while respecting the network settings of their data centers. - -The supported storage types are Amazon S3 and Microsoft Azure Blob storage. - -Here's the example of the configuration file with the endpoint map: - -```yaml -storage: - type: s3 - s3: - endpointUrl: http://S3:9000 - endpointUrlMap: - "node01:27017": "did.socf.s3.com" - "node03:27017": "https://example.aws.s3.com" - ... -``` - -You can define the specific nodes for the `endpointUrlMap` parameter. Not listed nodes use the URL defined for the `endpointUrl` parameter. - -For the solution to work, you should also have the mapping mechanism in place. This mechanism should be able to map the custom endpoint URLs to the main endpoint URL of the storage, routing the requests from `pbm-agents` to the storage and back seamlessly. - -With this ability to control the endpoints for `pbm-agents` to reach the same storage, you reduce the administrative overhead on PBM configuration and ensure its proper functioning. - -## Debug logging +### Debug logging You can enable debug logging for different types of S3 requests in Percona Backup for MongoDB. Percona Backup for MongoDB prints S3 log messages in the `pbm logs` output so that you can debug and diagnose S3 request issues or failures. To enable S3 debug logging, set the `storage.s3.DebugLogLevel` option in Percona Backup for MongoDB configuration. The supported values are: `LogDebug`, `Signing`, `HTTPBody`, `RequestRetries`, `RequestErrors`, `EventStreamBody`. -## Storage classes +### Storage classes Percona Backup for MongoDB supports [Amazon S3 storage classes :octicons-link-external-16:](https://aws.amazon.com/s3/storage-classes/). Knowing your data access patterns, you can set the S3 storage class in Percona Backup for MongoDB configuration. When Percona Backup for MongoDB uploads data to S3, the data is distributed to the corresponding storage class. The support of S3 bucket storage types allows you to effectively manage S3 storage space and costs. @@ -204,7 +136,7 @@ storage: When the option is undefined, the S3 Standard (`STANDARD`) storage type is used. -## Upload retries +### Upload retries You can set up the number of attempts for Percona Backup for MongoDB to upload data to S3 storage as well as the min and max time to wait for the next retry. Set the options `storage.s3.retryer.numMaxRetries`, `storage.s3.retryer.minRetryDelay` and `storage.s3.retryer.maxRetryDelay` in Percona Backup for MongoDB configuration. @@ -217,56 +149,5 @@ retryer: This upload retry increases the chances of data upload completion in cases of unstable connection. -## Data upload to storage with self-signed TLS certificates - -Percona Backup for MongoDB supports data upload to S3-compatible storage service over HTTPS with a self-signed or a private CA certificate. This feature is especially important when you use services like MinIO, Ceph, or internal S3 gateways that don't use certificates signed by public Certificate Authorities (CAs). - -Providing a whole chain of certificates is recommended to ensure the connection is legit. The `SSL_CERT_FILE` environment variable specifies the path to a custom certificate chain file in PEM-format that PBM uses to validate TLS/SSL connection. - -### Usage example - -Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path and your S3 endpoint is `https://minio.internal.local:9000`. To use self-issued TLS certificates, do the following: - -1. Ensure the cert file is in PEM format. Use the following command to check it: - - ```bash - cat /etc/ssl/minio-ca.crt - ``` - - ??? example "Sample output" - - - ```{text .no-copy} - -----BEGIN CERTIFICATE----- - MIIC+TCCAeGgAwIBAgIJANH3WljB... - -----END CERTIFICATE----- - ``` - -2. Set the `SSL_CERT_FILE` environment variable to that file's path on each host where `pbm-agent` and PBM CLI are running: - - ```{.bash data-prompt="$"} - $ export SSL_CERT_FILE=/etc/ssl/minio-ca.crt - ``` - - If this variable isn't set, PBM uses the system root certificates. - -3. Restart `pbm-agent`: - - ```{.bash data-prompt="$"} - $ sudo systemctl start pbm-agent - ``` - -4. Verify that your custom certificate is recognized. Check PBM logs for successful S3 access. - - -Alternatively, you can disable the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: - -```{.bash data-prompt="$"} -$ pbm config --set storage.s3.insecureSkipTLSVerify=True -``` - -!!! warning - - Use this option with caution as it might leave a hole for man-in-the-middle attacks. diff --git a/docs/details/storage-configuration.md b/docs/details/storage-configuration.md index 2c970e0d..779e86ca 100644 --- a/docs/details/storage-configuration.md +++ b/docs/details/storage-configuration.md @@ -8,26 +8,30 @@ The backup storage serves several purposes: * Ensures data durability and availability * Allows for backup data portability across different environments -Percona Backup for MongoDB (PBM) saves backup data to a designated directory on the backup storage. It can be a specific directory you define for the storage or the root folder. Each backup is prefixed with the UTC starting time for easy identification and consists of: +## Supported storage types -* A metadata file containing backup information -* For each replica set: +Percona Backup for MongoDB supports the following storage types: - - A compressed mongodump archive of all collections - - A compressed BSON file containing the oplog entries for the backup period +* [Amazon S3](s3-storage.md) +* [MinIO and S3-compatible storage](minio.md) +* [Filesystem server storage](filesystem-storage.md) +* [Microsoft Azure Blob storage](azure.md) -The oplog entries ensure backup consistency, and the end time of the oplog slice(s) is the data-consistent point in time of a backup snapshot. +## How PBM organizes backups on the storage -Using the [`pbm list`](../reference/pbm-commands.md#pbm-list) or [`pbm status`](../reference/pbm-commands.md#pbm-status) commands, you can scan the backup directory to find existing backups, even if you never used PBM on your computer before. +Percona Backup for MongoDB (PBM) saves backup data to a designated directory on the backup storage. It can be a specific directory you define for the storage or the root folder. -## Supported storage types +Each backup is prefixed with the UTC starting time for easy identification and consists of: -Percona Backup for MongoDB supports the following storage types: +* A metadata file containing backup information +* For each replica set: -* [S3-compatible storage](s3-storage.md) -* [Filesystem server storage](filesystem-storage.md) -* [Microsoft Azure Blob storage](azure.md) + * A compressed mongodump archive of all collections + * A compressed BSON file containing the oplog entries for the backup period +The oplog entries ensure backup consistency, and the end time of the oplog slice(s) is the data-consistent point in time of a backup snapshot. + +Using the [`pbm list`](../reference/pbm-commands.md#pbm-list) or [`pbm status`](../reference/pbm-commands.md#pbm-status) commands, you can scan the backup directory to find existing backups, even if you never used PBM on your computer before. ## Permissions setup @@ -61,6 +65,8 @@ The following example shows the permissions configuration to the `pbm-testing` b } ``` +### Storage-specific documentation + Please refer to the documentation of your selected storage for the data access management. !!! admonition "See also" diff --git a/docs/features/split-merge-backup.md b/docs/features/split-merge-backup.md index b5a96372..8a1d9e11 100644 --- a/docs/features/split-merge-backup.md +++ b/docs/features/split-merge-backup.md @@ -9,6 +9,7 @@ The following table provides default maximum size limits for the supported backu | Storage | Default size limit| | :--- | :--- | | **AWS S3** | 4.9 TB | +| **MinIO and S3 compatible storage** | 4.9 TB | | **GCS** | 4.9 TB | | **Azure Blob Storage** | 190 TB | | **Filesystem storage** | 4.9 TB | diff --git a/docs/install/backup-storage.md b/docs/install/backup-storage.md index b8974150..56dbc640 100644 --- a/docs/install/backup-storage.md +++ b/docs/install/backup-storage.md @@ -26,6 +26,20 @@ Percona Backup for MongoDB needs its own dedicated S3 bucket exclusively for bac secret-access-key: ``` + === ":simple-minio: S3-compatible (MinIO)" + + ```yaml + storage: + type: minio + minio: + endpoint: http://minio.example.com:9000 + bucket: pbm-test-bucket + prefix: data/pbm/backup + credentials: + access-key-id: + secret-access-key: + ``` + === ":material-google-cloud: GCS (SA)" ```yaml diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index 4d6aedf8..004dbeaf 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -7,9 +7,9 @@ *Type*: string
*Required*: YES -Remote backup storage type. Supported values: `s3`, `filesystem`, `azure`. +Remote backup storage type. Supported values: `s3`, `minio`, `gcs`, `filesystem`, `azure`. -## S3 type storage options +## AWS S3 storage options ```yaml storage: @@ -64,7 +64,7 @@ The name of the storage bucket. See the [AWS Bucket naming rules](https://docs.a *Required*: YES (for AWS) The location of the storage bucket. -Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) and [GCS region list](https://cloud.google.com/storage/docs/locations) to define the bucket region +Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region ### storage.s3.prefix @@ -85,7 +85,7 @@ The URL to access the bucket. *Type*: array of strings
*Required*: NO -The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/s3-storage.md#multiple-endpoints-to-the-same-s3-storage) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. +The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. ### storage.s3.forcePathStyle @@ -255,6 +255,156 @@ The maximum time to wait before the next retry, specified as a *time.Duration*. The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the limit. Read more about [Managing large backup files](../features/split-merge-backup.md). +## MinIO type storage options + +You can use this storage type for other S3-compatible storages + +```yaml +storage: + type: minio + minio: + region: + bucket: + prefix: + endpoint: + endpointMap: + "node01:2017": + "node02:2017": + secure: false + insecureSkipTLSVerify: false + forcePathStyle: false + credentials: + access-key-id: + secret-access-key: + session-token: + signature-ver: V4 + partSize: 10485760 (10 MB) + retryer: + numMaxRetries: 10 + maxObjSizeGB: 5018 + debugTrace: false +``` + +### storage.minio.region + +*Type*: string
+*Required*: YES + +The location of the storage bucket. Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region. If undefined, the default `us-east-1` region is used. + +### storage.minio.bucket + + +*Type*: string
+*Required*: YES + +The name of the storage bucket. See the [AWS Bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules) for bucket name requirements. + +### storage.minio.prefix + + +*Type*: string
+*Required*: NO + +The path to the data directory in the bucket. If undefined, backups are stored in the bucket’s root directory. + +### storage.minio.endpoint + +*Type*: string
+*Required*: YES + +The network address (URL or IP) where your MinIO server is accessible. + +### storage.minio.endpointMap + +*Type*: array of strings +*Required*: NO + +The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. + +### storage.minio.secure + +*Type*: boolean +*Required*: NO +*Default*: false + +Defines whether to use HTTP or HTTPS protocol when accessing the server. Default: `false`. + +### storage.minio.insecureSkipTLSVerify + +*Type*: boolean +*Required*: NO +*Default*: false + +Disables the TLS verification of the S3 storage. This allows Percona Backup for MongoDB to upload data to S3-like storages that use self-issued TLS certificates. Use it with caution as it might leave a hole for man-in-the-middle attacks. + +### storage.minio.forcePathStyle + +*Type*: boolean +*Required*: NO +*Default*: false + +Enforces the use of [path style access](../reference/glossary.md#path-style-access-to-the-storage) to the storage. Default is `false` which means PBM uses the [virtual-hosted-style](../reference/glossary.md#virtual-hosted-style-access) access to the storage + +### storage.minio.credentials.access-key-id + +*Type*: string +*Required*: YES + +Your access key to the storage bucket. This option can be omitted when you run Percona Backup for MongoDB using an EC2 instance profile. To learn more, refer to [Automate access to S3 buckets for Percona Backup for MongoDB](../manage/automate-s3-access.md). + +### storage.minio.credentials.secret-access-key-id + +*Type*: string +*Required*: YES + +The key to sign your programmatic requests to the storage bucket. This option can be omitted when you run Percona Backup for MongoDB using an EC2 instance profile. To learn more, refer to [Automate access to S3 buckets for Percona Backup for MongoDB](../manage/automate-s3-access.md). + +### storage.minio.credentials.session-token + +*Type*: string +*Required*: NO + +The MinIO session token used to validate the temporary security credentials for accessing the storage. + +### storage.minio.credentials.signature-version + +*Type*: string +*Required*: NO +*Default*: V4 + +Allows using the deprecated AWS Signature version 2. May be used for backward compatibility with the storages that don't support Signature version 4. Default: `V4`. + +### storage.minio.partSize + +*Type*: int +*Required*: NO + +The size of data chunks in bytes to be uploaded to the storage bucket. Default: 10MB. + +### storage.minio.retryer.numMaxRetries + +*Type*: int +*Required*: NO +*Deafult*: 10 + +The maximum number of retries to upload data to S3 storage. A zero value means no retries will be performed. + +### storage.minio.maxObjSizeGB + +*Type*: float64 +*Required*: NO +*Default*: 5018 + +The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the limit. Read more about [Managing large backup files](../features/split-merge-backup.md). + +### storage.minio.debugTrace + +*Type*: boolean +*Required*: NO + +If set to `true`, outputs all http communication trace in PBM log. Default: false. + ## GCS type storage options ```yaml diff --git a/docs/reference/glossary.md b/docs/reference/glossary.md index ecfa3b36..0e1c9dbd 100644 --- a/docs/reference/glossary.md +++ b/docs/reference/glossary.md @@ -50,6 +50,10 @@ Once a transaction is committed, it will remain so. An EBS (Amazon Elastic Block Storage) snapshot is the point-in-time copy of your data, and can be used to enable disaster recovery, migrate data across regions and accounts, and improve backup compliance. +## Endpoint + +The network address (URL or IP) where an S3-compatible storage service (like MinIO) is accessible. + ## GCP GCP (Google Cloud Platform) is the set of services, including storage service, that runs on Google Cloud infrastructure. @@ -82,6 +86,10 @@ A compressed bundle of [oplog](#oplog) entries stored in the Oplog Store databas A unique identifier of an operation such as backup, restore, resync. When a pbm-agent starts processing an operation, it acquires a lock and an opID. This prevents processing the same operation twice (for example, if there are network issues in distributed systems). Using opID as a log filter allows viewing logs for an operation in progress. +## Path style access to the storage + +A method of constructing S3 URLs where the bucket name appears in the path portion of the URL. The URL format is ``. Preferred for S3-compatible storage systems like MinIO, especially in environments without wildcard DNS or custom SSL certificates. + ## `pbm-agent` A `pbm-agent` is a PBM process running on the mongod node for backup and restore operations. A pbm-agent instance is required for every mongod node (including replica set secondary members and config server replica set nodes). @@ -122,3 +130,6 @@ Server-side encryption is the encryption of data by the remote storage server as Technical preview features are not yet ready for enterprise use and are not included in support via SLA. They are included in this release so that users can provide feedback prior to the full release of the feature in a future GA release (or removal of the feature if it is deemed not useful). This functionality can change (APIs, CLIs, etc.) from tech preview to GA. +## Virtual-hosted-style access + +A method of constructing S3 URLs where the bucket name is part of the domain name. The URL format is ``. Required by AWS S3 in newer regions; enables better routing and performance in large-scale deployments. diff --git a/docs/troubleshoot/index.md b/docs/troubleshoot/index.md index 4efb2f37..1143f53e 100644 --- a/docs/troubleshoot/index.md +++ b/docs/troubleshoot/index.md @@ -4,6 +4,7 @@ If you encounter issues when working with Percona Backup for MongoDB, consult th - [Diagnose PBM](troubleshooting.md) - [Check PBM status](status.md) +- [Generate diagnostics report](pbm-report.md) - [Partially done physical restores](restore-partial.md) - [Troubleshoot backup management via Percona Monitoring and Management](pbm-pmm.md) - [FAQ](faq.md) diff --git a/mkdocs-base.yml b/mkdocs-base.yml index 49d7758b..9f46eae7 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -228,10 +228,12 @@ nav: - Backup management via PMM: pmm.md - Storage: - Remote backup storage overview: details/storage-configuration.md - - AWS or s3-compatible storage: details/s3-storage.md + - AWS storage: details/s3-storage.md + - details/minio.md - details/gcs.md - - details/filesystem-storage.md - details/azure.md + - details/filesystem-storage.md + - details/endpoint-map.md - features/multi-storage.md - features/split-merge-backup.md - PBM commands: reference/pbm-commands.md From 4adfcab751cc4d124265a02086d28635c7decc81 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 20 Oct 2025 11:11:09 +0200 Subject: [PATCH 02/20] Update docs/reference/configuration-options.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/reference/configuration-options.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index 004dbeaf..245b79cc 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -317,10 +317,10 @@ The network address (URL or IP) where your MinIO server is accessible. ### storage.minio.endpointMap -*Type*: array of strings +*Type*: map[string]string *Required*: NO -The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. +A mapping of node names to custom endpoint URLs for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3, MinIO, and Microsoft Azure Blob storages. Available with version 2.8.0. ### storage.minio.secure From 397971d0cead423b3daf1b088f71f03da123672f Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 20 Oct 2025 11:11:22 +0200 Subject: [PATCH 03/20] Update docs/reference/configuration-options.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/reference/configuration-options.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index 245b79cc..a8ae95f8 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -386,7 +386,7 @@ The size of data chunks in bytes to be uploaded to the storage bucket. Default: *Type*: int *Required*: NO -*Deafult*: 10 +*Default*: 10 The maximum number of retries to upload data to S3 storage. A zero value means no retries will be performed. From ea765d689e9acc8ca6e5c3becd836b33f107aac5 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Fri, 17 Oct 2025 19:30:41 +0200 Subject: [PATCH 04/20] PBM-1631 Documented the minio storage type new file: docs/details/endpoint-map.md new file: docs/details/minio.md modified: docs/details/s3-storage.md modified: docs/details/storage-configuration.md modified: docs/features/split-merge-backup.md modified: docs/install/backup-storage.md modified: docs/reference/configuration-options.md modified: docs/reference/glossary.md modified: docs/troubleshoot/index.md modified: mkdocs-base.yml --- docs/details/endpoint-map.md | 61 ++++++++ docs/details/minio.md | 132 +++++++++++++++++ docs/details/s3-storage.md | 183 +++++------------------- docs/details/storage-configuration.md | 30 ++-- docs/features/split-merge-backup.md | 1 + docs/install/backup-storage.md | 14 ++ docs/reference/configuration-options.md | 166 ++++++++++++++++++++- docs/reference/glossary.md | 11 ++ docs/troubleshoot/index.md | 1 + mkdocs-base.yml | 6 +- 10 files changed, 436 insertions(+), 169 deletions(-) create mode 100644 docs/details/endpoint-map.md create mode 100644 docs/details/minio.md diff --git a/docs/details/endpoint-map.md b/docs/details/endpoint-map.md new file mode 100644 index 00000000..b3212f14 --- /dev/null +++ b/docs/details/endpoint-map.md @@ -0,0 +1,61 @@ +# Multiple endpoints to the same storage + +!!! admonition "Version added: [2.8.0](../release-notes/2.8.0.md)" + +In environments where `pbm-agents` run on servers that are distributed across several data centers, accessing the same remote backup storage can become challenging. This can be due to complex network configurations or strict policies that prevent direct connections to the outside world. As a result, `pbm-agents` can't use the same URL to reach the storage, which is necessary for Percona Backup for MongoDB to work properly. + +To address these challenges, you can configure custom endpoint URLs for specific nodes in the PBM configuration. This allows all `pbm-agents` to access the same storage while respecting the network settings of their data centers. + +The supported storage types are: + +* AWS S3, +* MinIO and S3-compatible storage services +* Microsoft Azure Blob storage. + +Here's the example of the configuration file with the endpoint map: + +=== ":fontawesome-brands-amazon: AWS S3" + + ```yaml + storage: + type: s3 + s3: + endpointUrl: http://S3:9000 + endpointUrlMap: + "node01:27017": "did.socf.s3.com" + "node03:27017": "https://example.aws.s3.com" + ... + ``` + +=== ":simple-minio: MinIO and S3-compatible storage" + + ```yaml + storage: + type: minio + minio: + endpoint: localhost:9100 + endpointMap: + "node01:27017": "did.socf.s3.com" + "node03:27017": "https://example.min.io" + ... + ``` + +=== ":material-microsoft-azure: Microsoft Azure Blob storage" + + ```yaml + storage: + type: azure + azure: + endpointUrl: https://myaccount.blob.core.windows.net + endpointUrlMap: + "node01:27017": "did.socf.blob.core.windows.net" + "node03:27017": "https://example.azure.blob.core.windows.net" + ... + ``` + + +You can define the specific nodes for the `endpointUrlMap` parameter for AWS S3 and Azure or for the `endpointMap` for MinIO and S3-compatible storage. Not listed nodes use the endpoint defined for the `endpointUrl` / `endpoint` parameter. + +For the solution to work, you should also have the mapping mechanism in place. This mechanism should be able to map the custom endpoints to the main endpoint URL of the storage, routing the requests from `pbm-agents` to the storage and back seamlessly. + +With this ability to control the endpoints for `pbm-agents` to reach the same storage, you reduce the administrative overhead on PBM configuration and ensure its proper functioning. \ No newline at end of file diff --git a/docs/details/minio.md b/docs/details/minio.md new file mode 100644 index 00000000..5fd94a77 --- /dev/null +++ b/docs/details/minio.md @@ -0,0 +1,132 @@ +# MinIO and S3-compatible storage + +Percona Backup for MongoDB (PBM) works with both AWS S3 and other S3-compatible storage services. We test S3-compatible storages with PBM using [MinIO :octicons-link-external-16:](https://min.io/) + +This document provides overview for MinIO as the closest S3-compatible storage. To use a native AWS S3 services, see [AWS S3 storage](s3-storage.md). + +[Configuration example :material-arrow-down:](#configuration-example){.md-button} + +## Bucket creation + +1. Install a [MinIO client :octicons-link-external-16:](https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc). After the installation, the `mc` is available for you. + +2. Configure the `mc` command line tool with a MinIO Server + + ```{.bash data-prompt="$"} + $ mc alias set myminio http://127.0.0.1:9000 MINIO_ACCESS_KEY MINIO_SECRET_KEY + ``` + +3. Create a bucket + + ```{.bash data-prompt="$"} + $ mc mb myminio/my-minio-bucket + ``` + +4. Verify the bucket creation + + ```{.bash data-prompt="$"} + $ mc ls myminio + ``` + +After the bucket is created, apply the proper [permissions for PBM to use the bucket](storage-configuration.md#permissions-setup). + +## Configuration example + +!!! important + + Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#storage-bucket-creation) and managed solely by PBM. + +This is the examples for the basic configuration of MinIO and other S3-compatible storages in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. + +```yaml +storage: + type: minio + minio: + endpoint: localhost:9100 + bucket: pbm-example + prefix: data/pbm/test + credentials: + access-key-id: + secret-access-key: +``` + +For the description of configuration options, see [Configuration file options](../reference/configuration-options.md). + +## Fine-tune storage configuration + +The following sections describe how you can fine-tune your storage configuration: + +* [debug logging](#debug-logging) +* [upload retries](#upload-retries) +* [data upload to storage with self-signed TLS certificates](#data-upload-to-storage-with-self-signed-tls-certificates) +* [multiple endpoints to the same S3 storage](endpoint-map.md) + +### Debug logging + +You can enable debug logging for different types of S3 requests in Percona Backup for MongoDB. Percona Backup for MongoDB prints S3 log messages in the `pbm logs` output so that you can debug and diagnose S3 request issues or failures. + +To enable debug logging, set the `storage.minio.debugTrace` option in Percona Backup for MongoDB configuration. This instructs PBM to also print HTTP trace from the S3 storage in the logs. + +## Upload retries + +You can set up the number of attempts for Percona Backup for MongoDB to upload data to S3 storage as well as the min and max time to wait for the next retry. Set the `storage.minio.retryer.numMaxRetries` option in Percona Backup for MongoDB configuration. + +```yaml +retryer: + numMaxRetries: 3 +``` + +This upload retry increases the chances of data upload completion in cases of unstable connection. + +## Data upload to storage with self-signed TLS certificates + +Percona Backup for MongoDB supports data upload to S3-compatible storage service over HTTPS with a self-signed or a private CA certificate. This feature is especially important when you use services like MinIO, Ceph, or internal S3 gateways that don't use certificates signed by public Certificate Authorities (CAs). + +Providing a whole chain of certificates is recommended to ensure the connection is legit. The `SSL_CERT_FILE` environment variable specifies the path to a custom certificate chain file in PEM-format that PBM uses to validate TLS/SSL connection. + +### Usage example + +Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path and your S3 endpoint is `https://minio.internal.local:9000`. To use self-issued TLS certificates, do the following: + +1. Ensure the cert file is in PEM format. Use the following command to check it: + + ```bash + cat /etc/ssl/minio-ca.crt + ``` + + ??? example "Sample output" + + + ```{text .no-copy} + -----BEGIN CERTIFICATE----- + MIIC+TCCAeGgAwIBAgIJANH3WljB... + -----END CERTIFICATE----- + ``` + +2. Set the `SSL_CERT_FILE` environment variable to that file's path on each host where `pbm-agent` and PBM CLI are running: + + ```{.bash data-prompt="$"} + $ export SSL_CERT_FILE=/etc/ssl/minio-ca.crt + ``` + + If this variable isn't set, PBM uses the system root certificates. + +3. Restart `pbm-agent`: + + ```{.bash data-prompt="$"} + $ sudo systemctl start pbm-agent + ``` + +4. Verify that your custom certificate is recognized. Check PBM logs for successful S3 access. + + +Alternatively, you can disable the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: + +```{.bash data-prompt="$"} +$ pbm config --set storage.s3.insecureSkipTLSVerify=True +``` + +!!! warning + + Use this option with caution as it might leave a hole for man-in-the-middle attacks. + diff --git a/docs/details/s3-storage.md b/docs/details/s3-storage.md index 5616f076..90808b4b 100644 --- a/docs/details/s3-storage.md +++ b/docs/details/s3-storage.md @@ -1,76 +1,43 @@ -# AWS or S3-compatible storage +# AWS S3 storage Percona Backup for MongoDB (PBM) works with AWS S3 and other S3-compatible storage services. We test PBM with the following services: * [Amazon Simple Storage Service :octicons-link-external-16:](https://docs.aws.amazon.com/s3/index.html) * [MinIO :octicons-link-external-16:](https://min.io/) -[Configuration examples :material-arrow-down:](#configuration-examples){.md-button} +This document provides overview for the native AWS S3 services. To use MinIO and other S3-compatible storage services, see [S3-compatible storage](minio.md). -## Storage bucket creation - -Here are some examples of the steps required to create a bucket. - -=== ":material-aws: Amazon S3" - - 1. Install and configure [AWS CLI :octicons-link-external-16:](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) - - 2. Create an S3 bucket +[Configuration example :material-arrow-down:](#configuration-example){.md-button} - ```{.bash data-prompt="$"} - $ aws s3api create-bucket --bucket my-s3-bucket --region us-east-1 - ``` - - 3. Verify the bucket creation - - ```{.bash data-prompt="$"} - $ aws s3 ls - ``` - -=== ":simple-minio: MinIo" +## Storage bucket creation - 1. Install a [MinIO client :octicons-link-external-16:](https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc). After the installation, the `mc` is available for you. +To create a bucket, do the following. - 2. Configure the `mc` command line tool with a MinIO Server +1. Install and configure [AWS CLI :octicons-link-external-16:](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) - ```{.bash data-prompt="$"} - $ mc alias set myminio http://127.0.0.1:9000 MINIO_ACCESS_KEY MINIO_SECRET_KEY - ``` - - 3. Create a bucket +2. Create an S3 bucket - ```{.bash data-prompt="$"} - $ mc mb myminio/my-minio-bucket - ``` - - 4. Verify the bucket creation + ```{.bash data-prompt="$"} + $ aws s3api create-bucket --bucket my-s3-bucket --region us-east-1 + ``` + +3. Verify the bucket creation - ```{.bash data-prompt="$"} - $ mc ls myminio - ``` + ```{.bash data-prompt="$"} + $ aws s3 ls + ``` After the bucket is created, apply the proper [permissions for PBM to use the bucket](storage-configuration.md#permissions-setup). -## Configuration examples +## Configuration example !!! important Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#storage-bucket-creation) and managed solely by PBM. -These are the examples for the basic configuration of S3-compatible storage in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. - -The following sections describe how you can fine-tune your storage configuration: - -* [server-side encryption](#server-side-encryption), -* [multiple endpoints to the same S3 storage](#multiple-endpoints-to-the-same-s3-storage), -* [debug logging](#debug-logging), -* [storage classes](#storage-classes), -* [upload retries](#upload-retries), -* [data upload to storage with self-signed TLS certificates](#data-upload-to-storage-with-self-signed-tls-certificates). +This is the example for the basic configuration of AWS S3 storage in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. -### Amazon Simple Storage Service - ```yaml storage: type: s3 @@ -86,25 +53,19 @@ storage: kmsKeyID: ``` -### MinIO +For the description of configuration options, see [Configuration file options](../reference/configuration-options.md). -```yaml -storage: - type: s3 - s3: - endpointUrl: "http://localhost:9000" - region: my-region - bucket: pbm-example - prefix: data/pbm/test - credentials: - access-key-id: - secret-access-key: -``` +## Fine-tune storage configuration -For the description of configuration options, see [Configuration file options](../reference/configuration-options.md). +The following sections describe how you can fine-tune your storage configuration: +* [server-side encryption](#server-side-encryption) +* [debug logging](#debug-logging) +* [storage classes](#storage-classes) +* [upload retries](#upload-retries) +* [multiple endpoints to the same S3 storage](endpoint-map.md) -## Server-side encryption +### Server-side encryption Percona Backup for MongoDB supports [server-side encryption](../reference/glossary.md#server-side-encryption) for [S3 buckets](../reference/glossary.md#bucket) with the following encryption types: @@ -112,7 +73,7 @@ Percona Backup for MongoDB supports [server-side encryption](../reference/glossa * [customer-provided keys stored on the client side (SSE-C)](#using-customer-provided-keys-sse-c) * [Amazon S3 managed encryption keys (SSE-S3)](#using-amazon-s3-managed-keys-sse-s3) -### Using AWS KMS keys (SSE-KMS) +#### Using AWS KMS keys (SSE-KMS) To use the SSE-KMS encryption, specify the following parameters in the Percona Backup for MongoDB configuration file: @@ -122,7 +83,7 @@ serverSideEncryption: sseAlgorithm: aws:kms ``` -### Using customer-provided keys (SSE-C) +#### Using customer-provided keys (SSE-C) !!! admonition "Version added: [2.0.1](../release-notes/2.0.1.md)" @@ -141,7 +102,7 @@ serverSideEncryption: sseCustomerKey: ``` -### Using Amazon S3 managed keys (SSE-S3) +#### Using Amazon S3 managed keys (SSE-S3) !!! admonition "Version added: [2.6.0](../release-notes/2.6.0.md)" @@ -154,42 +115,13 @@ serverSideEncryption: sseAlgorithm: AES256 ``` -## Multiple endpoints to the same S3 storage - -!!! admonition "Version added: [2.8.0](../release-notes/2.8.0.md)" - -In environments where `pbm-agents` run on servers that are distributed across several data centers, accessing the same remote backup storage can become challenging. This can be due to complex network configurations or strict policies that prevent direct connections to the outside world. As a result, `pbm-agents` can't use the same URL to reach the storage, which is necessary for Percona Backup for MongoDB to work properly. - -To address these challenges, you can configure custom endpoint URLs for specific nodes in the PBM configuration. This allows all `pbm-agents` to access the same storage while respecting the network settings of their data centers. - -The supported storage types are Amazon S3 and Microsoft Azure Blob storage. - -Here's the example of the configuration file with the endpoint map: - -```yaml -storage: - type: s3 - s3: - endpointUrl: http://S3:9000 - endpointUrlMap: - "node01:27017": "did.socf.s3.com" - "node03:27017": "https://example.aws.s3.com" - ... -``` - -You can define the specific nodes for the `endpointUrlMap` parameter. Not listed nodes use the URL defined for the `endpointUrl` parameter. - -For the solution to work, you should also have the mapping mechanism in place. This mechanism should be able to map the custom endpoint URLs to the main endpoint URL of the storage, routing the requests from `pbm-agents` to the storage and back seamlessly. - -With this ability to control the endpoints for `pbm-agents` to reach the same storage, you reduce the administrative overhead on PBM configuration and ensure its proper functioning. - -## Debug logging +### Debug logging You can enable debug logging for different types of S3 requests in Percona Backup for MongoDB. Percona Backup for MongoDB prints S3 log messages in the `pbm logs` output so that you can debug and diagnose S3 request issues or failures. To enable S3 debug logging, set the `storage.s3.DebugLogLevel` option in Percona Backup for MongoDB configuration. The supported values are: `LogDebug`, `Signing`, `HTTPBody`, `RequestRetries`, `RequestErrors`, `EventStreamBody`. -## Storage classes +### Storage classes Percona Backup for MongoDB supports [Amazon S3 storage classes :octicons-link-external-16:](https://aws.amazon.com/s3/storage-classes/). Knowing your data access patterns, you can set the S3 storage class in Percona Backup for MongoDB configuration. When Percona Backup for MongoDB uploads data to S3, the data is distributed to the corresponding storage class. The support of S3 bucket storage types allows you to effectively manage S3 storage space and costs. @@ -204,7 +136,7 @@ storage: When the option is undefined, the S3 Standard (`STANDARD`) storage type is used. -## Upload retries +### Upload retries You can set up the number of attempts for Percona Backup for MongoDB to upload data to S3 storage as well as the min and max time to wait for the next retry. Set the options `storage.s3.retryer.numMaxRetries`, `storage.s3.retryer.minRetryDelay` and `storage.s3.retryer.maxRetryDelay` in Percona Backup for MongoDB configuration. @@ -217,56 +149,5 @@ retryer: This upload retry increases the chances of data upload completion in cases of unstable connection. -## Data upload to storage with self-signed TLS certificates - -Percona Backup for MongoDB supports data upload to S3-compatible storage service over HTTPS with a self-signed or a private CA certificate. This feature is especially important when you use services like MinIO, Ceph, or internal S3 gateways that don't use certificates signed by public Certificate Authorities (CAs). - -Providing a whole chain of certificates is recommended to ensure the connection is legit. The `SSL_CERT_FILE` environment variable specifies the path to a custom certificate chain file in PEM-format that PBM uses to validate TLS/SSL connection. - -### Usage example - -Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path and your S3 endpoint is `https://minio.internal.local:9000`. To use self-issued TLS certificates, do the following: - -1. Ensure the cert file is in PEM format. Use the following command to check it: - - ```bash - cat /etc/ssl/minio-ca.crt - ``` - - ??? example "Sample output" - - - ```{text .no-copy} - -----BEGIN CERTIFICATE----- - MIIC+TCCAeGgAwIBAgIJANH3WljB... - -----END CERTIFICATE----- - ``` - -2. Set the `SSL_CERT_FILE` environment variable to that file's path on each host where `pbm-agent` and PBM CLI are running: - - ```{.bash data-prompt="$"} - $ export SSL_CERT_FILE=/etc/ssl/minio-ca.crt - ``` - - If this variable isn't set, PBM uses the system root certificates. - -3. Restart `pbm-agent`: - - ```{.bash data-prompt="$"} - $ sudo systemctl start pbm-agent - ``` - -4. Verify that your custom certificate is recognized. Check PBM logs for successful S3 access. - - -Alternatively, you can disable the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: - -```{.bash data-prompt="$"} -$ pbm config --set storage.s3.insecureSkipTLSVerify=True -``` - -!!! warning - - Use this option with caution as it might leave a hole for man-in-the-middle attacks. diff --git a/docs/details/storage-configuration.md b/docs/details/storage-configuration.md index 2c970e0d..779e86ca 100644 --- a/docs/details/storage-configuration.md +++ b/docs/details/storage-configuration.md @@ -8,26 +8,30 @@ The backup storage serves several purposes: * Ensures data durability and availability * Allows for backup data portability across different environments -Percona Backup for MongoDB (PBM) saves backup data to a designated directory on the backup storage. It can be a specific directory you define for the storage or the root folder. Each backup is prefixed with the UTC starting time for easy identification and consists of: +## Supported storage types -* A metadata file containing backup information -* For each replica set: +Percona Backup for MongoDB supports the following storage types: - - A compressed mongodump archive of all collections - - A compressed BSON file containing the oplog entries for the backup period +* [Amazon S3](s3-storage.md) +* [MinIO and S3-compatible storage](minio.md) +* [Filesystem server storage](filesystem-storage.md) +* [Microsoft Azure Blob storage](azure.md) -The oplog entries ensure backup consistency, and the end time of the oplog slice(s) is the data-consistent point in time of a backup snapshot. +## How PBM organizes backups on the storage -Using the [`pbm list`](../reference/pbm-commands.md#pbm-list) or [`pbm status`](../reference/pbm-commands.md#pbm-status) commands, you can scan the backup directory to find existing backups, even if you never used PBM on your computer before. +Percona Backup for MongoDB (PBM) saves backup data to a designated directory on the backup storage. It can be a specific directory you define for the storage or the root folder. -## Supported storage types +Each backup is prefixed with the UTC starting time for easy identification and consists of: -Percona Backup for MongoDB supports the following storage types: +* A metadata file containing backup information +* For each replica set: -* [S3-compatible storage](s3-storage.md) -* [Filesystem server storage](filesystem-storage.md) -* [Microsoft Azure Blob storage](azure.md) + * A compressed mongodump archive of all collections + * A compressed BSON file containing the oplog entries for the backup period +The oplog entries ensure backup consistency, and the end time of the oplog slice(s) is the data-consistent point in time of a backup snapshot. + +Using the [`pbm list`](../reference/pbm-commands.md#pbm-list) or [`pbm status`](../reference/pbm-commands.md#pbm-status) commands, you can scan the backup directory to find existing backups, even if you never used PBM on your computer before. ## Permissions setup @@ -61,6 +65,8 @@ The following example shows the permissions configuration to the `pbm-testing` b } ``` +### Storage-specific documentation + Please refer to the documentation of your selected storage for the data access management. !!! admonition "See also" diff --git a/docs/features/split-merge-backup.md b/docs/features/split-merge-backup.md index b5a96372..8a1d9e11 100644 --- a/docs/features/split-merge-backup.md +++ b/docs/features/split-merge-backup.md @@ -9,6 +9,7 @@ The following table provides default maximum size limits for the supported backu | Storage | Default size limit| | :--- | :--- | | **AWS S3** | 4.9 TB | +| **MinIO and S3 compatible storage** | 4.9 TB | | **GCS** | 4.9 TB | | **Azure Blob Storage** | 190 TB | | **Filesystem storage** | 4.9 TB | diff --git a/docs/install/backup-storage.md b/docs/install/backup-storage.md index b8974150..56dbc640 100644 --- a/docs/install/backup-storage.md +++ b/docs/install/backup-storage.md @@ -26,6 +26,20 @@ Percona Backup for MongoDB needs its own dedicated S3 bucket exclusively for bac secret-access-key: ``` + === ":simple-minio: S3-compatible (MinIO)" + + ```yaml + storage: + type: minio + minio: + endpoint: http://minio.example.com:9000 + bucket: pbm-test-bucket + prefix: data/pbm/backup + credentials: + access-key-id: + secret-access-key: + ``` + === ":material-google-cloud: GCS (SA)" ```yaml diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index 4d6aedf8..5de12492 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -7,9 +7,9 @@ *Type*: string
*Required*: YES -Remote backup storage type. Supported values: `s3`, `filesystem`, `azure`. +Remote backup storage type. Supported values: `s3`, `minio`, `gcs`, `filesystem`, `azure`. -## S3 type storage options +## AWS S3 storage options ```yaml storage: @@ -64,7 +64,7 @@ The name of the storage bucket. See the [AWS Bucket naming rules](https://docs.a *Required*: YES (for AWS) The location of the storage bucket. -Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) and [GCS region list](https://cloud.google.com/storage/docs/locations) to define the bucket region +Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region ### storage.s3.prefix @@ -85,7 +85,7 @@ The URL to access the bucket. *Type*: array of strings
*Required*: NO -The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/s3-storage.md#multiple-endpoints-to-the-same-s3-storage) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. +The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. ### storage.s3.forcePathStyle @@ -255,6 +255,156 @@ The maximum time to wait before the next retry, specified as a *time.Duration*. The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the limit. Read more about [Managing large backup files](../features/split-merge-backup.md). +## MinIO type storage options + +You can use this storage type for other S3-compatible storages + +```yaml +storage: + type: minio + minio: + region: + bucket: + prefix: + endpoint: + endpointMap: + "node01:2017": + "node02:2017": + secure: false + insecureSkipTLSVerify: false + forcePathStyle: false + credentials: + access-key-id: + secret-access-key: + session-token: + signature-ver: V4 + partSize: 10485760 (10 MB) + retryer: + numMaxRetries: 10 + maxObjSizeGB: 5018 + debugTrace: false +``` + +### storage.minio.region + +*Type*: string
+*Required*: YES + +The location of the storage bucket. Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region. If undefined, the default `us-east-1` region is used. + +### storage.minio.bucket + + +*Type*: string
+*Required*: YES + +The name of the storage bucket. See the [AWS Bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules) for bucket name requirements. + +### storage.minio.prefix + + +*Type*: string
+*Required*: NO + +The path to the data directory in the bucket. If undefined, backups are stored in the bucket’s root directory. + +### storage.minio.endpoint + +*Type*: string
+*Required*: YES + +The network address (URL or IP) where your MinIO server is accessible. + +### storage.minio.endpointMap + +*Type*: array of strings +*Required*: NO + +The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. + +### storage.minio.secure + +*Type*: boolean +*Required*: NO +*Default*: false + +Defines whether to use HTTP or HTTPS protocol for communication between PBM and S3 storage. Default: `false`. + +### storage.minio.insecureSkipTLSVerify + +*Type*: boolean +*Required*: NO +*Default*: false + +Disables the TLS verification of the S3 storage. This allows Percona Backup for MongoDB to upload data to S3-like storages that use self-issued TLS certificates. Use it with caution as it might leave a hole for man-in-the-middle attacks. + +### storage.minio.forcePathStyle + +*Type*: boolean +*Required*: NO +*Default*: false + +Enforces the use of [path style access](../reference/glossary.md#path-style-access-to-the-storage) to the storage. Default is `false` which means PBM uses the [virtual-hosted-style](../reference/glossary.md#virtual-hosted-style-access) access to the storage + +### storage.minio.credentials.access-key-id + +*Type*: string +*Required*: YES + +Your access key to the storage bucket. This option can be omitted when you run Percona Backup for MongoDB using an EC2 instance profile. To learn more, refer to [Automate access to S3 buckets for Percona Backup for MongoDB](../manage/automate-s3-access.md). + +### storage.minio.credentials.secret-access-key + +*Type*: string +*Required*: YES + +The key to sign your programmatic requests to the storage bucket. This option can be omitted when you run Percona Backup for MongoDB using an EC2 instance profile. To learn more, refer to [Automate access to S3 buckets for Percona Backup for MongoDB](../manage/automate-s3-access.md). + +### storage.minio.credentials.session-token + +*Type*: string +*Required*: NO + +The MinIO session token used to validate the temporary security credentials for accessing the storage. + +### storage.minio.credentials.signature-ver + +*Type*: string +*Required*: NO +*Default*: V4 + +Allows using the deprecated AWS Signature version 2. May be used for backward compatibility with the storages that don't support Signature version 4. Default: `V4`. + +### storage.minio.partSize + +*Type*: int +*Required*: NO + +The size of data chunks in bytes to be uploaded to the storage bucket. Default: 10MB. + +### storage.minio.retryer.numMaxRetries + +*Type*: int +*Required*: NO +*Deafult*: 10 + +The maximum number of retries to upload data to S3 storage. A zero value means no retries will be performed. + +### storage.minio.maxObjSizeGB + +*Type*: float64 +*Required*: NO +*Default*: 5018 + +The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the limit. Read more about [Managing large backup files](../features/split-merge-backup.md). + +### storage.minio.debugTrace + +*Type*: boolean +*Required*: NO + +If set to `true`, outputs all http communication trace in PBM log. Default: false. + ## GCS type storage options ```yaml @@ -411,6 +561,14 @@ The name of the storage container. See the [Container names](https://docs.micro The URL to access the data in Microsoft Azure Blob Storage. The default value is `https://.blob.core.windows.net`. +### storage.azure.endpointUrlMap + +*Type*: array of strings
+*Required*: NO + +The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Available with version 2.8.0. + + ### storage.azure.prefix *Type*: string
diff --git a/docs/reference/glossary.md b/docs/reference/glossary.md index ecfa3b36..0e1c9dbd 100644 --- a/docs/reference/glossary.md +++ b/docs/reference/glossary.md @@ -50,6 +50,10 @@ Once a transaction is committed, it will remain so. An EBS (Amazon Elastic Block Storage) snapshot is the point-in-time copy of your data, and can be used to enable disaster recovery, migrate data across regions and accounts, and improve backup compliance. +## Endpoint + +The network address (URL or IP) where an S3-compatible storage service (like MinIO) is accessible. + ## GCP GCP (Google Cloud Platform) is the set of services, including storage service, that runs on Google Cloud infrastructure. @@ -82,6 +86,10 @@ A compressed bundle of [oplog](#oplog) entries stored in the Oplog Store databas A unique identifier of an operation such as backup, restore, resync. When a pbm-agent starts processing an operation, it acquires a lock and an opID. This prevents processing the same operation twice (for example, if there are network issues in distributed systems). Using opID as a log filter allows viewing logs for an operation in progress. +## Path style access to the storage + +A method of constructing S3 URLs where the bucket name appears in the path portion of the URL. The URL format is ``. Preferred for S3-compatible storage systems like MinIO, especially in environments without wildcard DNS or custom SSL certificates. + ## `pbm-agent` A `pbm-agent` is a PBM process running on the mongod node for backup and restore operations. A pbm-agent instance is required for every mongod node (including replica set secondary members and config server replica set nodes). @@ -122,3 +130,6 @@ Server-side encryption is the encryption of data by the remote storage server as Technical preview features are not yet ready for enterprise use and are not included in support via SLA. They are included in this release so that users can provide feedback prior to the full release of the feature in a future GA release (or removal of the feature if it is deemed not useful). This functionality can change (APIs, CLIs, etc.) from tech preview to GA. +## Virtual-hosted-style access + +A method of constructing S3 URLs where the bucket name is part of the domain name. The URL format is ``. Required by AWS S3 in newer regions; enables better routing and performance in large-scale deployments. diff --git a/docs/troubleshoot/index.md b/docs/troubleshoot/index.md index 4efb2f37..1143f53e 100644 --- a/docs/troubleshoot/index.md +++ b/docs/troubleshoot/index.md @@ -4,6 +4,7 @@ If you encounter issues when working with Percona Backup for MongoDB, consult th - [Diagnose PBM](troubleshooting.md) - [Check PBM status](status.md) +- [Generate diagnostics report](pbm-report.md) - [Partially done physical restores](restore-partial.md) - [Troubleshoot backup management via Percona Monitoring and Management](pbm-pmm.md) - [FAQ](faq.md) diff --git a/mkdocs-base.yml b/mkdocs-base.yml index 49d7758b..9f46eae7 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -228,10 +228,12 @@ nav: - Backup management via PMM: pmm.md - Storage: - Remote backup storage overview: details/storage-configuration.md - - AWS or s3-compatible storage: details/s3-storage.md + - AWS storage: details/s3-storage.md + - details/minio.md - details/gcs.md - - details/filesystem-storage.md - details/azure.md + - details/filesystem-storage.md + - details/endpoint-map.md - features/multi-storage.md - features/split-merge-backup.md - PBM commands: reference/pbm-commands.md From a59eb1e06cb098f4429709c4ca3538bf784ddd9d Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 20 Oct 2025 12:27:02 +0200 Subject: [PATCH 05/20] Update docs/details/endpoint-map.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/details/endpoint-map.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/details/endpoint-map.md b/docs/details/endpoint-map.md index b3212f14..10eddaec 100644 --- a/docs/details/endpoint-map.md +++ b/docs/details/endpoint-map.md @@ -1,4 +1,4 @@ -# Multiple endpoints to the same storage +# Multiple endpoints to the same storage !!! admonition "Version added: [2.8.0](../release-notes/2.8.0.md)" From 721b14c9780ead8b2c300b25c2d91dd755dac2f0 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 20 Oct 2025 12:27:10 +0200 Subject: [PATCH 06/20] Update docs/details/minio.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/details/minio.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/details/minio.md b/docs/details/minio.md index 5fd94a77..496b2ca4 100644 --- a/docs/details/minio.md +++ b/docs/details/minio.md @@ -123,7 +123,7 @@ Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path Alternatively, you can disable the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: ```{.bash data-prompt="$"} -$ pbm config --set storage.s3.insecureSkipTLSVerify=True +$ pbm config --set storage.minio.insecureSkipTLSVerify=True ``` !!! warning From 66fd87b2ecfb0ee7e2ae23470a0668ee873daec3 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 20 Oct 2025 12:27:40 +0200 Subject: [PATCH 07/20] Update docs/details/minio.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/details/minio.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/details/minio.md b/docs/details/minio.md index 496b2ca4..10b3ef09 100644 --- a/docs/details/minio.md +++ b/docs/details/minio.md @@ -2,7 +2,7 @@ Percona Backup for MongoDB (PBM) works with both AWS S3 and other S3-compatible storage services. We test S3-compatible storages with PBM using [MinIO :octicons-link-external-16:](https://min.io/) -This document provides overview for MinIO as the closest S3-compatible storage. To use a native AWS S3 services, see [AWS S3 storage](s3-storage.md). +This document provides an overview of MinIO as the closest S3-compatible storage. To use the native AWS S3 service, see [AWS S3 storage](s3-storage.md). [Configuration example :material-arrow-down:](#configuration-example){.md-button} From 46b80652eed7da16a4ef05a0457bf44b64c65812 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 20 Oct 2025 12:28:13 +0200 Subject: [PATCH 08/20] Update docs/reference/configuration-options.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/details/endpoint-map.md | 4 ++-- docs/details/minio.md | 15 ++++++++------- docs/reference/configuration-options.md | 22 ++++++++++------------ 3 files changed, 20 insertions(+), 21 deletions(-) diff --git a/docs/details/endpoint-map.md b/docs/details/endpoint-map.md index 10eddaec..a22feb98 100644 --- a/docs/details/endpoint-map.md +++ b/docs/details/endpoint-map.md @@ -8,9 +8,9 @@ To address these challenges, you can configure custom endpoint URLs for specific The supported storage types are: -* AWS S3, +* AWS S3 * MinIO and S3-compatible storage services -* Microsoft Azure Blob storage. +* Microsoft Azure Blob storage Here's the example of the configuration file with the endpoint map: diff --git a/docs/details/minio.md b/docs/details/minio.md index 10b3ef09..5e0f6487 100644 --- a/docs/details/minio.md +++ b/docs/details/minio.md @@ -36,18 +36,19 @@ After the bucket is created, apply the proper [permissions for PBM to use the bu Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#storage-bucket-creation) and managed solely by PBM. -This is the examples for the basic configuration of MinIO and other S3-compatible storages in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. +This is the example for the basic configuration of MinIO and other S3-compatible storages in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. ```yaml storage: type: minio minio: - endpoint: localhost:9100 - bucket: pbm-example - prefix: data/pbm/test - credentials: - access-key-id: - secret-access-key: + endpoint: localhost:9100 + region: us-east-1 + bucket: pbm-example + prefix: data/pbm/test + credentials: + access-key-id: + secret-access-key: ``` For the description of configuration options, see [Configuration file options](../reference/configuration-options.md). diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index 800602eb..a45b6ce7 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -7,10 +7,8 @@ *Type*: string
*Required*: YES -Remote backup storage type. Supported values: `s3`, `minio`, `gcs`, `filesystem`, `azure`. Remote backup storage type. Supported values: `s3`, `minio`, `gcs`, `filesystem`, `azure`. -## AWS S3 storage options ## AWS S3 storage options ```yaml @@ -67,7 +65,6 @@ The name of the storage bucket. See the [AWS Bucket naming rules](https://docs.a The location of the storage bucket. Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region -Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region ### storage.s3.prefix @@ -88,7 +85,6 @@ The URL to access the bucket. *Type*: array of strings
*Required*: NO -The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. ### storage.s3.forcePathStyle @@ -294,7 +290,7 @@ storage: *Type*: string
*Required*: YES -The location of the storage bucket. Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region. If undefined, the default `us-east-1` region is used. +The location of the storage bucket. Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region. If not specified, the default `us-east-1` region is used. ### storage.minio.bucket @@ -324,7 +320,7 @@ The network address (URL or IP) where your MinIO server is accessible. *Type*: array of strings *Required*: NO -The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. +A mapping of custom endpoints for `pbm-agents` on different servers to the same MinIO storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3, MinIO, and Microsoft Azure Blob storages. Available with version 2.8.0. ### storage.minio.secure @@ -340,7 +336,7 @@ Defines whether to use HTTP or HTTPS protocol for communication between PBM and *Required*: NO *Default*: false -Disables the TLS verification of the S3 storage. This allows Percona Backup for MongoDB to upload data to S3-like storages that use self-issued TLS certificates. Use it with caution as it might leave a hole for man-in-the-middle attacks. +Disables the TLS verification of the MinIO / S3-compatible storage. This allows Percona Backup for MongoDB to upload data to MinIO / S3-compatible storages that use self-issued TLS certificates. Use it with caution as it might leave a hole for man-in-the-middle attacks. ### storage.minio.forcePathStyle @@ -377,7 +373,9 @@ The MinIO session token used to validate the temporary security credentials for *Required*: NO *Default*: V4 -Allows using the deprecated AWS Signature version 2. May be used for backward compatibility with the storages that don't support Signature version 4. Default: `V4`. +Specifies the AWS Signature version to use for authentication. Accepted values: `V2`, `V4`. + +Allows using the deprecated AWS Signature version 2 for backward compatibility with storages that don't support Signature version 4. Default: `V4`. ### storage.minio.partSize @@ -390,9 +388,9 @@ The size of data chunks in bytes to be uploaded to the storage bucket. Default: *Type*: int *Required*: NO -*Deafult*: 10 +*Default*: 10 -The maximum number of retries to upload data to S3 storage. A zero value means no retries will be performed. +The maximum number of retries to upload data to MinIO / S3-compatible storage. A zero value means no retries will be performed. ### storage.minio.maxObjSizeGB @@ -567,10 +565,10 @@ The URL to access the data in Microsoft Azure Blob Storage. The default value is ### storage.azure.endpointUrlMap -*Type*: array of strings
+*Type*: object (host:port -> endpoint URL)
*Required*: NO -The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Available with version 2.8.0. +A mapping of custom endpoint URLs for `pbm-agents` on different servers to the same remote storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same remote storage](../details/endpoint-map.md) section. Available with version 2.8.0. ### storage.azure.prefix From b79710de6d74b749842ac0b20c8a1683c3ff21ad Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 20 Oct 2025 14:59:02 +0200 Subject: [PATCH 09/20] Update docs/reference/glossary.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/reference/glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/glossary.md b/docs/reference/glossary.md index 0e1c9dbd..a272731f 100644 --- a/docs/reference/glossary.md +++ b/docs/reference/glossary.md @@ -86,7 +86,7 @@ A compressed bundle of [oplog](#oplog) entries stored in the Oplog Store databas A unique identifier of an operation such as backup, restore, resync. When a pbm-agent starts processing an operation, it acquires a lock and an opID. This prevents processing the same operation twice (for example, if there are network issues in distributed systems). Using opID as a log filter allows viewing logs for an operation in progress. -## Path style access to the storage +## Path-style access to the storage A method of constructing S3 URLs where the bucket name appears in the path portion of the URL. The URL format is ``. Preferred for S3-compatible storage systems like MinIO, especially in environments without wildcard DNS or custom SSL certificates. From 43f5d9d423231f7d79029406372cee7fb696bcc4 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 20 Oct 2025 14:59:14 +0200 Subject: [PATCH 10/20] Update docs/features/split-merge-backup.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/features/split-merge-backup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features/split-merge-backup.md b/docs/features/split-merge-backup.md index 8a1d9e11..d0cb023b 100644 --- a/docs/features/split-merge-backup.md +++ b/docs/features/split-merge-backup.md @@ -9,7 +9,7 @@ The following table provides default maximum size limits for the supported backu | Storage | Default size limit| | :--- | :--- | | **AWS S3** | 4.9 TB | -| **MinIO and S3 compatible storage** | 4.9 TB | +| **MinIO and S3-compatible storage** | 4.9 TB | | **GCS** | 4.9 TB | | **Azure Blob Storage** | 190 TB | | **Filesystem storage** | 4.9 TB | From 2024c4aab58d511b9d32e4d7e0a12de57a9f4a87 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 20 Oct 2025 14:59:42 +0200 Subject: [PATCH 11/20] Update docs/details/minio.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/details/minio.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/details/minio.md b/docs/details/minio.md index 5e0f6487..a70175de 100644 --- a/docs/details/minio.md +++ b/docs/details/minio.md @@ -34,7 +34,7 @@ After the bucket is created, apply the proper [permissions for PBM to use the bu !!! important - Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#storage-bucket-creation) and managed solely by PBM. + Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#bucket-creation) and managed solely by PBM. This is the example for the basic configuration of MinIO and other S3-compatible storages in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. From ed13d751f3d26eaedae9495cbd2b5a8347730273 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 20 Oct 2025 15:00:00 +0200 Subject: [PATCH 12/20] Update docs/reference/configuration-options.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/details/endpoint-map.md | 2 +- docs/reference/configuration-options.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/details/endpoint-map.md b/docs/details/endpoint-map.md index a22feb98..b8b00980 100644 --- a/docs/details/endpoint-map.md +++ b/docs/details/endpoint-map.md @@ -54,7 +54,7 @@ Here's the example of the configuration file with the endpoint map: ``` -You can define the specific nodes for the `endpointUrlMap` parameter for AWS S3 and Azure or for the `endpointMap` for MinIO and S3-compatible storage. Not listed nodes use the endpoint defined for the `endpointUrl` / `endpoint` parameter. +You can define specific nodes using the `endpointUrlMap` (for AWS S3 and Azure) or `endpointMap` (for MinIO and S3-compatible storage) parameters. Both parameters are maps of `'host:port'` to endpoint URL, as shown in the examples above. Nodes not listed in the map will use the endpoint defined by the `endpointUrl` or `endpoint` parameter, respectively. For the solution to work, you should also have the mapping mechanism in place. This mechanism should be able to map the custom endpoints to the main endpoint URL of the storage, routing the requests from `pbm-agents` to the storage and back seamlessly. diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index a45b6ce7..892c91d1 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -257,7 +257,7 @@ The maximum file size to be stored on the backup storage. If the file to upload ## MinIO type storage options -You can use this storage type for other S3-compatible storages +You can use this storage type for other S3-compatible storage services ```yaml storage: From 96da008af160bd1782ff841e572828c4f9640bd2 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Tue, 21 Oct 2025 15:43:16 +0200 Subject: [PATCH 13/20] Apply suggestion from @nastena1606 --- docs/details/storage-configuration.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/details/storage-configuration.md b/docs/details/storage-configuration.md index 779e86ca..b839ce1c 100644 --- a/docs/details/storage-configuration.md +++ b/docs/details/storage-configuration.md @@ -13,6 +13,7 @@ The backup storage serves several purposes: Percona Backup for MongoDB supports the following storage types: * [Amazon S3](s3-storage.md) +* [Google Cloud storage](gcs.md) * [MinIO and S3-compatible storage](minio.md) * [Filesystem server storage](filesystem-storage.md) * [Microsoft Azure Blob storage](azure.md) From 4ae7c668f4f9bd0d5e895695d22285bd3f68ced0 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Wed, 29 Oct 2025 09:28:20 +0100 Subject: [PATCH 14/20] Updated linter dictionary --- .github/styles/config/vocabularies/Percona/accept.txt | 5 +++++ .vale.ini | 4 +++- docs/details/endpoint-map.md | 2 +- docs/details/minio.md | 6 +++--- 4 files changed, 12 insertions(+), 5 deletions(-) diff --git a/.github/styles/config/vocabularies/Percona/accept.txt b/.github/styles/config/vocabularies/Percona/accept.txt index 5ec10d84..6476c4ed 100644 --- a/.github/styles/config/vocabularies/Percona/accept.txt +++ b/.github/styles/config/vocabularies/Percona/accept.txt @@ -23,3 +23,8 @@ MinIO [Rr]esync (?i)HMAC retryer +MinIO +S3-compatible +uncomment +CAs +Ceph diff --git a/.vale.ini b/.vale.ini index d3e242e3..27f11fad 100644 --- a/.vale.ini +++ b/.vale.ini @@ -1,6 +1,6 @@ StylesPath = .github/styles -MinAlertLevel = suggestion +MinAlertLevel = warning Packages = Google @@ -10,4 +10,6 @@ Vocab = Percona [*] BasedOnStyles = Vale, Google +Vale.Spelling = warning + Vale.Terms = NO diff --git a/docs/details/endpoint-map.md b/docs/details/endpoint-map.md index b8b00980..1104b854 100644 --- a/docs/details/endpoint-map.md +++ b/docs/details/endpoint-map.md @@ -54,7 +54,7 @@ Here's the example of the configuration file with the endpoint map: ``` -You can define specific nodes using the `endpointUrlMap` (for AWS S3 and Azure) or `endpointMap` (for MinIO and S3-compatible storage) parameters. Both parameters are maps of `'host:port'` to endpoint URL, as shown in the examples above. Nodes not listed in the map will use the endpoint defined by the `endpointUrl` or `endpoint` parameter, respectively. +You can define specific nodes using the `endpointUrlMap` (for AWS S3 and Azure) or `endpointMap` (for MinIO and S3-compatible storage) parameters. Both parameters are maps of `'host:port'` to endpoint URL, as shown in the earlier examples. Nodes not listed in the map use the endpoint defined by the `endpointUrl` or `endpoint` parameter, respectively. For the solution to work, you should also have the mapping mechanism in place. This mechanism should be able to map the custom endpoints to the main endpoint URL of the storage, routing the requests from `pbm-agents` to the storage and back seamlessly. diff --git a/docs/details/minio.md b/docs/details/minio.md index a70175de..5b07c67b 100644 --- a/docs/details/minio.md +++ b/docs/details/minio.md @@ -1,6 +1,6 @@ # MinIO and S3-compatible storage -Percona Backup for MongoDB (PBM) works with both AWS S3 and other S3-compatible storage services. We test S3-compatible storages with PBM using [MinIO :octicons-link-external-16:](https://min.io/) +Percona Backup for MongoDB (PBM) works with both AWS S3 and other S3-compatible storage services. We test S3-compatible storage services with PBM using [MinIO :octicons-link-external-16:](https://min.io/) This document provides an overview of MinIO as the closest S3-compatible storage. To use the native AWS S3 service, see [AWS S3 storage](s3-storage.md). @@ -36,7 +36,7 @@ After the bucket is created, apply the proper [permissions for PBM to use the bu Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#bucket-creation) and managed solely by PBM. -This is the example for the basic configuration of MinIO and other S3-compatible storages in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. +This is the example for the basic configuration of MinIO and other S3-compatible storage services in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. ```yaml storage: @@ -121,7 +121,7 @@ Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path 4. Verify that your custom certificate is recognized. Check PBM logs for successful S3 access. -Alternatively, you can disable the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: +Alternatively, you can turn off the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: ```{.bash data-prompt="$"} $ pbm config --set storage.minio.insecureSkipTLSVerify=True From b4388c3016db82edd45e7b7174810370bfde3f58 Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Thu, 30 Oct 2025 14:48:10 +0100 Subject: [PATCH 15/20] Updated after the review --- docs/details/endpoint-map.md | 2 +- docs/details/minio.md | 8 ++++---- docs/install/backup-storage.md | 2 +- docs/reference/configuration-options.md | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/details/endpoint-map.md b/docs/details/endpoint-map.md index 1104b854..eafba90c 100644 --- a/docs/details/endpoint-map.md +++ b/docs/details/endpoint-map.md @@ -49,7 +49,7 @@ Here's the example of the configuration file with the endpoint map: endpointUrl: https://myaccount.blob.core.windows.net endpointUrlMap: "node01:27017": "did.socf.blob.core.windows.net" - "node03:27017": "https://example.azure.blob.core.windows.net" + "node03:27017": "example.azure.blob.core.windows.net" ... ``` diff --git a/docs/details/minio.md b/docs/details/minio.md index 5b07c67b..5c69f599 100644 --- a/docs/details/minio.md +++ b/docs/details/minio.md @@ -64,13 +64,13 @@ The following sections describe how you can fine-tune your storage configuration ### Debug logging -You can enable debug logging for different types of S3 requests in Percona Backup for MongoDB. Percona Backup for MongoDB prints S3 log messages in the `pbm logs` output so that you can debug and diagnose S3 request issues or failures. +You can enable debug logging for different types of storage requests in Percona Backup for MongoDB. Percona Backup for MongoDB prints log messages in the `pbm logs` output so that you can debug and diagnose storage request issues or failures. -To enable debug logging, set the `storage.minio.debugTrace` option in Percona Backup for MongoDB configuration. This instructs PBM to also print HTTP trace from the S3 storage in the logs. +To enable debug logging, set the `storage.minio.debugTrace` option in Percona Backup for MongoDB configuration. This instructs PBM to also print HTTP trace from the MinIO storage in the logs. ## Upload retries -You can set up the number of attempts for Percona Backup for MongoDB to upload data to S3 storage as well as the min and max time to wait for the next retry. Set the `storage.minio.retryer.numMaxRetries` option in Percona Backup for MongoDB configuration. +You can set up the number of attempts for Percona Backup for MongoDB to upload data to S3 storage. Set the `storage.minio.retryer.numMaxRetries` option in Percona Backup for MongoDB configuration. ```yaml retryer: @@ -118,7 +118,7 @@ Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path $ sudo systemctl start pbm-agent ``` -4. Verify that your custom certificate is recognized. Check PBM logs for successful S3 access. +4. Verify that your custom certificate is recognized. Check PBM logs for successful storage access. Alternatively, you can turn off the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: diff --git a/docs/install/backup-storage.md b/docs/install/backup-storage.md index 56dbc640..699c9be0 100644 --- a/docs/install/backup-storage.md +++ b/docs/install/backup-storage.md @@ -32,7 +32,7 @@ Percona Backup for MongoDB needs its own dedicated S3 bucket exclusively for bac storage: type: minio minio: - endpoint: http://minio.example.com:9000 + endpoint: minio.example.com:9000 bucket: pbm-test-bucket prefix: data/pbm/backup credentials: diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index 892c91d1..53249766 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -313,7 +313,7 @@ The path to the data directory in the bucket. If undefined, backups are stored i *Type*: string
*Required*: YES -The network address (URL or IP) where your MinIO server is accessible. +The network address (URL or IP:port) where your MinIO server is accessible. ### storage.minio.endpointMap From a38a5857a92166d2f49a8a16066d5f93544c3461 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 3 Nov 2025 13:03:21 +0100 Subject: [PATCH 16/20] Update docs/reference/configuration-options.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/reference/configuration-options.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index 53249766..8e315a91 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -288,7 +288,7 @@ storage: ### storage.minio.region *Type*: string
-*Required*: YES +*Required*: NO The location of the storage bucket. Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region. If not specified, the default `us-east-1` region is used. From ec1c6fa69adfdab80b79d3183dde5d68ecc0e489 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 3 Nov 2025 13:03:45 +0100 Subject: [PATCH 17/20] Update docs/reference/configuration-options.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- docs/reference/configuration-options.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index 8e315a91..154240cf 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -351,14 +351,14 @@ Enforces the use of [path style access](../reference/glossary.md#path-style-acce *Type*: string *Required*: YES -Your access key to the storage bucket. This option can be omitted when you run Percona Backup for MongoDB using an EC2 instance profile. To learn more, refer to [Automate access to S3 buckets for Percona Backup for MongoDB](../manage/automate-s3-access.md). +Your access key to the storage bucket. ### storage.minio.credentials.secret-access-key *Type*: string *Required*: YES -The key to sign your programmatic requests to the storage bucket. This option can be omitted when you run Percona Backup for MongoDB using an EC2 instance profile. To learn more, refer to [Automate access to S3 buckets for Percona Backup for MongoDB](../manage/automate-s3-access.md). +The key to sign your programmatic requests to the storage bucket. ### storage.minio.credentials.session-token From 1658c54a5bcbbabe735440d861750ae7e43027bc Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 3 Nov 2025 13:03:57 +0100 Subject: [PATCH 18/20] Update docs/details/minio.md Co-authored-by: Sandra Romanchenko <53295797+sandraromanchenko@users.noreply.github.com> --- docs/details/minio.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/details/minio.md b/docs/details/minio.md index 5c69f599..11994f16 100644 --- a/docs/details/minio.md +++ b/docs/details/minio.md @@ -43,7 +43,6 @@ storage: type: minio minio: endpoint: localhost:9100 - region: us-east-1 bucket: pbm-example prefix: data/pbm/test credentials: From f9151c235557d24f4c513f74ec112caa793c0ea7 Mon Sep 17 00:00:00 2001 From: Anastasia Alexandrova Date: Mon, 3 Nov 2025 13:04:09 +0100 Subject: [PATCH 19/20] Update docs/details/endpoint-map.md Co-authored-by: Sandra Romanchenko <53295797+sandraromanchenko@users.noreply.github.com> --- docs/details/endpoint-map.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/details/endpoint-map.md b/docs/details/endpoint-map.md index eafba90c..77a2d43b 100644 --- a/docs/details/endpoint-map.md +++ b/docs/details/endpoint-map.md @@ -36,7 +36,7 @@ Here's the example of the configuration file with the endpoint map: endpoint: localhost:9100 endpointMap: "node01:27017": "did.socf.s3.com" - "node03:27017": "https://example.min.io" + "node03:27017": "example.min.io" ... ``` From 416f7e09bf43f800d35622584ee5639a3408105a Mon Sep 17 00:00:00 2001 From: Anastasia Alexadrova Date: Mon, 3 Nov 2025 13:09:03 +0100 Subject: [PATCH 20/20] fixed line breaks for options --- docs/reference/configuration-options.md | 36 ++++++++++++------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index 154240cf..fd49024e 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -317,60 +317,60 @@ The network address (URL or IP:port) where your MinIO server is accessible. ### storage.minio.endpointMap -*Type*: array of strings +*Type*: array of strings
*Required*: NO A mapping of custom endpoints for `pbm-agents` on different servers to the same MinIO storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3, MinIO, and Microsoft Azure Blob storages. Available with version 2.8.0. ### storage.minio.secure -*Type*: boolean -*Required*: NO +*Type*: boolean
+*Required*: NO
*Default*: false Defines whether to use HTTP or HTTPS protocol for communication between PBM and S3 storage. Default: `false`. ### storage.minio.insecureSkipTLSVerify -*Type*: boolean -*Required*: NO +*Type*: boolean
+*Required*: NO
*Default*: false Disables the TLS verification of the MinIO / S3-compatible storage. This allows Percona Backup for MongoDB to upload data to MinIO / S3-compatible storages that use self-issued TLS certificates. Use it with caution as it might leave a hole for man-in-the-middle attacks. ### storage.minio.forcePathStyle -*Type*: boolean -*Required*: NO +*Type*: boolean
+*Required*: NO
*Default*: false Enforces the use of [path style access](../reference/glossary.md#path-style-access-to-the-storage) to the storage. Default is `false` which means PBM uses the [virtual-hosted-style](../reference/glossary.md#virtual-hosted-style-access) access to the storage ### storage.minio.credentials.access-key-id -*Type*: string +*Type*: string
*Required*: YES Your access key to the storage bucket. ### storage.minio.credentials.secret-access-key -*Type*: string +*Type*: string
*Required*: YES The key to sign your programmatic requests to the storage bucket. ### storage.minio.credentials.session-token -*Type*: string +*Type*: string
*Required*: NO The MinIO session token used to validate the temporary security credentials for accessing the storage. ### storage.minio.credentials.signature-ver -*Type*: string -*Required*: NO +*Type*: string
+*Required*: NO
*Default*: V4 Specifies the AWS Signature version to use for authentication. Accepted values: `V2`, `V4`. @@ -379,30 +379,30 @@ Allows using the deprecated AWS Signature version 2 for backward compatibility w ### storage.minio.partSize -*Type*: int +*Type*: int
*Required*: NO The size of data chunks in bytes to be uploaded to the storage bucket. Default: 10MB. ### storage.minio.retryer.numMaxRetries -*Type*: int -*Required*: NO +*Type*: int
+*Required*: NO
*Default*: 10 The maximum number of retries to upload data to MinIO / S3-compatible storage. A zero value means no retries will be performed. ### storage.minio.maxObjSizeGB -*Type*: float64 -*Required*: NO +*Type*: float64
+*Required*: NO
*Default*: 5018 The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the limit. Read more about [Managing large backup files](../features/split-merge-backup.md). ### storage.minio.debugTrace -*Type*: boolean +*Type*: boolean
*Required*: NO If set to `true`, outputs all http communication trace in PBM log. Default: false.