diff --git a/.github/styles/config/vocabularies/Percona/accept.txt b/.github/styles/config/vocabularies/Percona/accept.txt index 5ec10d84..6476c4ed 100644 --- a/.github/styles/config/vocabularies/Percona/accept.txt +++ b/.github/styles/config/vocabularies/Percona/accept.txt @@ -23,3 +23,8 @@ MinIO [Rr]esync (?i)HMAC retryer +MinIO +S3-compatible +uncomment +CAs +Ceph diff --git a/.vale.ini b/.vale.ini index d3e242e3..27f11fad 100644 --- a/.vale.ini +++ b/.vale.ini @@ -1,6 +1,6 @@ StylesPath = .github/styles -MinAlertLevel = suggestion +MinAlertLevel = warning Packages = Google @@ -10,4 +10,6 @@ Vocab = Percona [*] BasedOnStyles = Vale, Google +Vale.Spelling = warning + Vale.Terms = NO diff --git a/docs/details/endpoint-map.md b/docs/details/endpoint-map.md new file mode 100644 index 00000000..77a2d43b --- /dev/null +++ b/docs/details/endpoint-map.md @@ -0,0 +1,61 @@ +# Multiple endpoints to the same storage + +!!! admonition "Version added: [2.8.0](../release-notes/2.8.0.md)" + +In environments where `pbm-agents` run on servers that are distributed across several data centers, accessing the same remote backup storage can become challenging. This can be due to complex network configurations or strict policies that prevent direct connections to the outside world. As a result, `pbm-agents` can't use the same URL to reach the storage, which is necessary for Percona Backup for MongoDB to work properly. + +To address these challenges, you can configure custom endpoint URLs for specific nodes in the PBM configuration. This allows all `pbm-agents` to access the same storage while respecting the network settings of their data centers. + +The supported storage types are: + +* AWS S3 +* MinIO and S3-compatible storage services +* Microsoft Azure Blob storage + +Here's the example of the configuration file with the endpoint map: + +=== ":fontawesome-brands-amazon: AWS S3" + + ```yaml + storage: + type: s3 + s3: + endpointUrl: http://S3:9000 + endpointUrlMap: + "node01:27017": "did.socf.s3.com" + "node03:27017": "https://example.aws.s3.com" + ... + ``` + +=== ":simple-minio: MinIO and S3-compatible storage" + + ```yaml + storage: + type: minio + minio: + endpoint: localhost:9100 + endpointMap: + "node01:27017": "did.socf.s3.com" + "node03:27017": "example.min.io" + ... + ``` + +=== ":material-microsoft-azure: Microsoft Azure Blob storage" + + ```yaml + storage: + type: azure + azure: + endpointUrl: https://myaccount.blob.core.windows.net + endpointUrlMap: + "node01:27017": "did.socf.blob.core.windows.net" + "node03:27017": "example.azure.blob.core.windows.net" + ... + ``` + + +You can define specific nodes using the `endpointUrlMap` (for AWS S3 and Azure) or `endpointMap` (for MinIO and S3-compatible storage) parameters. Both parameters are maps of `'host:port'` to endpoint URL, as shown in the earlier examples. Nodes not listed in the map use the endpoint defined by the `endpointUrl` or `endpoint` parameter, respectively. + +For the solution to work, you should also have the mapping mechanism in place. This mechanism should be able to map the custom endpoints to the main endpoint URL of the storage, routing the requests from `pbm-agents` to the storage and back seamlessly. + +With this ability to control the endpoints for `pbm-agents` to reach the same storage, you reduce the administrative overhead on PBM configuration and ensure its proper functioning. \ No newline at end of file diff --git a/docs/details/minio.md b/docs/details/minio.md new file mode 100644 index 00000000..11994f16 --- /dev/null +++ b/docs/details/minio.md @@ -0,0 +1,132 @@ +# MinIO and S3-compatible storage + +Percona Backup for MongoDB (PBM) works with both AWS S3 and other S3-compatible storage services. We test S3-compatible storage services with PBM using [MinIO :octicons-link-external-16:](https://min.io/) + +This document provides an overview of MinIO as the closest S3-compatible storage. To use the native AWS S3 service, see [AWS S3 storage](s3-storage.md). + +[Configuration example :material-arrow-down:](#configuration-example){.md-button} + +## Bucket creation + +1. Install a [MinIO client :octicons-link-external-16:](https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc). After the installation, the `mc` is available for you. + +2. Configure the `mc` command line tool with a MinIO Server + + ```{.bash data-prompt="$"} + $ mc alias set myminio http://127.0.0.1:9000 MINIO_ACCESS_KEY MINIO_SECRET_KEY + ``` + +3. Create a bucket + + ```{.bash data-prompt="$"} + $ mc mb myminio/my-minio-bucket + ``` + +4. Verify the bucket creation + + ```{.bash data-prompt="$"} + $ mc ls myminio + ``` + +After the bucket is created, apply the proper [permissions for PBM to use the bucket](storage-configuration.md#permissions-setup). + +## Configuration example + +!!! important + + Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#bucket-creation) and managed solely by PBM. + +This is the example for the basic configuration of MinIO and other S3-compatible storage services in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. + +```yaml +storage: + type: minio + minio: + endpoint: localhost:9100 + bucket: pbm-example + prefix: data/pbm/test + credentials: + access-key-id: + secret-access-key: +``` + +For the description of configuration options, see [Configuration file options](../reference/configuration-options.md). + +## Fine-tune storage configuration + +The following sections describe how you can fine-tune your storage configuration: + +* [debug logging](#debug-logging) +* [upload retries](#upload-retries) +* [data upload to storage with self-signed TLS certificates](#data-upload-to-storage-with-self-signed-tls-certificates) +* [multiple endpoints to the same S3 storage](endpoint-map.md) + +### Debug logging + +You can enable debug logging for different types of storage requests in Percona Backup for MongoDB. Percona Backup for MongoDB prints log messages in the `pbm logs` output so that you can debug and diagnose storage request issues or failures. + +To enable debug logging, set the `storage.minio.debugTrace` option in Percona Backup for MongoDB configuration. This instructs PBM to also print HTTP trace from the MinIO storage in the logs. + +## Upload retries + +You can set up the number of attempts for Percona Backup for MongoDB to upload data to S3 storage. Set the `storage.minio.retryer.numMaxRetries` option in Percona Backup for MongoDB configuration. + +```yaml +retryer: + numMaxRetries: 3 +``` + +This upload retry increases the chances of data upload completion in cases of unstable connection. + +## Data upload to storage with self-signed TLS certificates + +Percona Backup for MongoDB supports data upload to S3-compatible storage service over HTTPS with a self-signed or a private CA certificate. This feature is especially important when you use services like MinIO, Ceph, or internal S3 gateways that don't use certificates signed by public Certificate Authorities (CAs). + +Providing a whole chain of certificates is recommended to ensure the connection is legit. The `SSL_CERT_FILE` environment variable specifies the path to a custom certificate chain file in PEM-format that PBM uses to validate TLS/SSL connection. + +### Usage example + +Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path and your S3 endpoint is `https://minio.internal.local:9000`. To use self-issued TLS certificates, do the following: + +1. Ensure the cert file is in PEM format. Use the following command to check it: + + ```bash + cat /etc/ssl/minio-ca.crt + ``` + + ??? example "Sample output" + + + ```{text .no-copy} + -----BEGIN CERTIFICATE----- + MIIC+TCCAeGgAwIBAgIJANH3WljB... + -----END CERTIFICATE----- + ``` + +2. Set the `SSL_CERT_FILE` environment variable to that file's path on each host where `pbm-agent` and PBM CLI are running: + + ```{.bash data-prompt="$"} + $ export SSL_CERT_FILE=/etc/ssl/minio-ca.crt + ``` + + If this variable isn't set, PBM uses the system root certificates. + +3. Restart `pbm-agent`: + + ```{.bash data-prompt="$"} + $ sudo systemctl start pbm-agent + ``` + +4. Verify that your custom certificate is recognized. Check PBM logs for successful storage access. + + +Alternatively, you can turn off the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: + +```{.bash data-prompt="$"} +$ pbm config --set storage.minio.insecureSkipTLSVerify=True +``` + +!!! warning + + Use this option with caution as it might leave a hole for man-in-the-middle attacks. + diff --git a/docs/details/s3-storage.md b/docs/details/s3-storage.md index 5616f076..90808b4b 100644 --- a/docs/details/s3-storage.md +++ b/docs/details/s3-storage.md @@ -1,76 +1,43 @@ -# AWS or S3-compatible storage +# AWS S3 storage Percona Backup for MongoDB (PBM) works with AWS S3 and other S3-compatible storage services. We test PBM with the following services: * [Amazon Simple Storage Service :octicons-link-external-16:](https://docs.aws.amazon.com/s3/index.html) * [MinIO :octicons-link-external-16:](https://min.io/) -[Configuration examples :material-arrow-down:](#configuration-examples){.md-button} +This document provides overview for the native AWS S3 services. To use MinIO and other S3-compatible storage services, see [S3-compatible storage](minio.md). -## Storage bucket creation - -Here are some examples of the steps required to create a bucket. - -=== ":material-aws: Amazon S3" - - 1. Install and configure [AWS CLI :octicons-link-external-16:](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) - - 2. Create an S3 bucket +[Configuration example :material-arrow-down:](#configuration-example){.md-button} - ```{.bash data-prompt="$"} - $ aws s3api create-bucket --bucket my-s3-bucket --region us-east-1 - ``` - - 3. Verify the bucket creation - - ```{.bash data-prompt="$"} - $ aws s3 ls - ``` - -=== ":simple-minio: MinIo" +## Storage bucket creation - 1. Install a [MinIO client :octicons-link-external-16:](https://min.io/docs/minio/linux/reference/minio-mc.html#install-mc). After the installation, the `mc` is available for you. +To create a bucket, do the following. - 2. Configure the `mc` command line tool with a MinIO Server +1. Install and configure [AWS CLI :octicons-link-external-16:](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) - ```{.bash data-prompt="$"} - $ mc alias set myminio http://127.0.0.1:9000 MINIO_ACCESS_KEY MINIO_SECRET_KEY - ``` - - 3. Create a bucket +2. Create an S3 bucket - ```{.bash data-prompt="$"} - $ mc mb myminio/my-minio-bucket - ``` - - 4. Verify the bucket creation + ```{.bash data-prompt="$"} + $ aws s3api create-bucket --bucket my-s3-bucket --region us-east-1 + ``` + +3. Verify the bucket creation - ```{.bash data-prompt="$"} - $ mc ls myminio - ``` + ```{.bash data-prompt="$"} + $ aws s3 ls + ``` After the bucket is created, apply the proper [permissions for PBM to use the bucket](storage-configuration.md#permissions-setup). -## Configuration examples +## Configuration example !!! important Percona Backup for MongoDB (PBM) needs its own dedicated S3 bucket exclusively for backup-related files. Ensure that this [bucket is created](#storage-bucket-creation) and managed solely by PBM. -These are the examples for the basic configuration of S3-compatible storage in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. - -The following sections describe how you can fine-tune your storage configuration: - -* [server-side encryption](#server-side-encryption), -* [multiple endpoints to the same S3 storage](#multiple-endpoints-to-the-same-s3-storage), -* [debug logging](#debug-logging), -* [storage classes](#storage-classes), -* [upload retries](#upload-retries), -* [data upload to storage with self-signed TLS certificates](#data-upload-to-storage-with-self-signed-tls-certificates). +This is the example for the basic configuration of AWS S3 storage in Percona Backup for MongoDB. You can find [the configuration file template :octicons-link-external-16:](https://github.com/percona/percona-backup-mongodb/blob/v{{release}}/packaging/conf/pbm-conf-reference.yml) and uncomment the required fields. -### Amazon Simple Storage Service - ```yaml storage: type: s3 @@ -86,25 +53,19 @@ storage: kmsKeyID: ``` -### MinIO +For the description of configuration options, see [Configuration file options](../reference/configuration-options.md). -```yaml -storage: - type: s3 - s3: - endpointUrl: "http://localhost:9000" - region: my-region - bucket: pbm-example - prefix: data/pbm/test - credentials: - access-key-id: - secret-access-key: -``` +## Fine-tune storage configuration -For the description of configuration options, see [Configuration file options](../reference/configuration-options.md). +The following sections describe how you can fine-tune your storage configuration: +* [server-side encryption](#server-side-encryption) +* [debug logging](#debug-logging) +* [storage classes](#storage-classes) +* [upload retries](#upload-retries) +* [multiple endpoints to the same S3 storage](endpoint-map.md) -## Server-side encryption +### Server-side encryption Percona Backup for MongoDB supports [server-side encryption](../reference/glossary.md#server-side-encryption) for [S3 buckets](../reference/glossary.md#bucket) with the following encryption types: @@ -112,7 +73,7 @@ Percona Backup for MongoDB supports [server-side encryption](../reference/glossa * [customer-provided keys stored on the client side (SSE-C)](#using-customer-provided-keys-sse-c) * [Amazon S3 managed encryption keys (SSE-S3)](#using-amazon-s3-managed-keys-sse-s3) -### Using AWS KMS keys (SSE-KMS) +#### Using AWS KMS keys (SSE-KMS) To use the SSE-KMS encryption, specify the following parameters in the Percona Backup for MongoDB configuration file: @@ -122,7 +83,7 @@ serverSideEncryption: sseAlgorithm: aws:kms ``` -### Using customer-provided keys (SSE-C) +#### Using customer-provided keys (SSE-C) !!! admonition "Version added: [2.0.1](../release-notes/2.0.1.md)" @@ -141,7 +102,7 @@ serverSideEncryption: sseCustomerKey: ``` -### Using Amazon S3 managed keys (SSE-S3) +#### Using Amazon S3 managed keys (SSE-S3) !!! admonition "Version added: [2.6.0](../release-notes/2.6.0.md)" @@ -154,42 +115,13 @@ serverSideEncryption: sseAlgorithm: AES256 ``` -## Multiple endpoints to the same S3 storage - -!!! admonition "Version added: [2.8.0](../release-notes/2.8.0.md)" - -In environments where `pbm-agents` run on servers that are distributed across several data centers, accessing the same remote backup storage can become challenging. This can be due to complex network configurations or strict policies that prevent direct connections to the outside world. As a result, `pbm-agents` can't use the same URL to reach the storage, which is necessary for Percona Backup for MongoDB to work properly. - -To address these challenges, you can configure custom endpoint URLs for specific nodes in the PBM configuration. This allows all `pbm-agents` to access the same storage while respecting the network settings of their data centers. - -The supported storage types are Amazon S3 and Microsoft Azure Blob storage. - -Here's the example of the configuration file with the endpoint map: - -```yaml -storage: - type: s3 - s3: - endpointUrl: http://S3:9000 - endpointUrlMap: - "node01:27017": "did.socf.s3.com" - "node03:27017": "https://example.aws.s3.com" - ... -``` - -You can define the specific nodes for the `endpointUrlMap` parameter. Not listed nodes use the URL defined for the `endpointUrl` parameter. - -For the solution to work, you should also have the mapping mechanism in place. This mechanism should be able to map the custom endpoint URLs to the main endpoint URL of the storage, routing the requests from `pbm-agents` to the storage and back seamlessly. - -With this ability to control the endpoints for `pbm-agents` to reach the same storage, you reduce the administrative overhead on PBM configuration and ensure its proper functioning. - -## Debug logging +### Debug logging You can enable debug logging for different types of S3 requests in Percona Backup for MongoDB. Percona Backup for MongoDB prints S3 log messages in the `pbm logs` output so that you can debug and diagnose S3 request issues or failures. To enable S3 debug logging, set the `storage.s3.DebugLogLevel` option in Percona Backup for MongoDB configuration. The supported values are: `LogDebug`, `Signing`, `HTTPBody`, `RequestRetries`, `RequestErrors`, `EventStreamBody`. -## Storage classes +### Storage classes Percona Backup for MongoDB supports [Amazon S3 storage classes :octicons-link-external-16:](https://aws.amazon.com/s3/storage-classes/). Knowing your data access patterns, you can set the S3 storage class in Percona Backup for MongoDB configuration. When Percona Backup for MongoDB uploads data to S3, the data is distributed to the corresponding storage class. The support of S3 bucket storage types allows you to effectively manage S3 storage space and costs. @@ -204,7 +136,7 @@ storage: When the option is undefined, the S3 Standard (`STANDARD`) storage type is used. -## Upload retries +### Upload retries You can set up the number of attempts for Percona Backup for MongoDB to upload data to S3 storage as well as the min and max time to wait for the next retry. Set the options `storage.s3.retryer.numMaxRetries`, `storage.s3.retryer.minRetryDelay` and `storage.s3.retryer.maxRetryDelay` in Percona Backup for MongoDB configuration. @@ -217,56 +149,5 @@ retryer: This upload retry increases the chances of data upload completion in cases of unstable connection. -## Data upload to storage with self-signed TLS certificates - -Percona Backup for MongoDB supports data upload to S3-compatible storage service over HTTPS with a self-signed or a private CA certificate. This feature is especially important when you use services like MinIO, Ceph, or internal S3 gateways that don't use certificates signed by public Certificate Authorities (CAs). - -Providing a whole chain of certificates is recommended to ensure the connection is legit. The `SSL_CERT_FILE` environment variable specifies the path to a custom certificate chain file in PEM-format that PBM uses to validate TLS/SSL connection. - -### Usage example - -Let's assume that your custom CA certificate is at `/etc/ssl/minio-ca.crt` path and your S3 endpoint is `https://minio.internal.local:9000`. To use self-issued TLS certificates, do the following: - -1. Ensure the cert file is in PEM format. Use the following command to check it: - - ```bash - cat /etc/ssl/minio-ca.crt - ``` - - ??? example "Sample output" - - - ```{text .no-copy} - -----BEGIN CERTIFICATE----- - MIIC+TCCAeGgAwIBAgIJANH3WljB... - -----END CERTIFICATE----- - ``` - -2. Set the `SSL_CERT_FILE` environment variable to that file's path on each host where `pbm-agent` and PBM CLI are running: - - ```{.bash data-prompt="$"} - $ export SSL_CERT_FILE=/etc/ssl/minio-ca.crt - ``` - - If this variable isn't set, PBM uses the system root certificates. - -3. Restart `pbm-agent`: - - ```{.bash data-prompt="$"} - $ sudo systemctl start pbm-agent - ``` - -4. Verify that your custom certificate is recognized. Check PBM logs for successful S3 access. - - -Alternatively, you can disable the TLS verification of the S3 storage in Percona Backup for MongoDB configuration: - -```{.bash data-prompt="$"} -$ pbm config --set storage.s3.insecureSkipTLSVerify=True -``` - -!!! warning - - Use this option with caution as it might leave a hole for man-in-the-middle attacks. diff --git a/docs/details/storage-configuration.md b/docs/details/storage-configuration.md index 2c970e0d..b839ce1c 100644 --- a/docs/details/storage-configuration.md +++ b/docs/details/storage-configuration.md @@ -8,26 +8,31 @@ The backup storage serves several purposes: * Ensures data durability and availability * Allows for backup data portability across different environments -Percona Backup for MongoDB (PBM) saves backup data to a designated directory on the backup storage. It can be a specific directory you define for the storage or the root folder. Each backup is prefixed with the UTC starting time for easy identification and consists of: +## Supported storage types -* A metadata file containing backup information -* For each replica set: +Percona Backup for MongoDB supports the following storage types: - - A compressed mongodump archive of all collections - - A compressed BSON file containing the oplog entries for the backup period +* [Amazon S3](s3-storage.md) +* [Google Cloud storage](gcs.md) +* [MinIO and S3-compatible storage](minio.md) +* [Filesystem server storage](filesystem-storage.md) +* [Microsoft Azure Blob storage](azure.md) -The oplog entries ensure backup consistency, and the end time of the oplog slice(s) is the data-consistent point in time of a backup snapshot. +## How PBM organizes backups on the storage -Using the [`pbm list`](../reference/pbm-commands.md#pbm-list) or [`pbm status`](../reference/pbm-commands.md#pbm-status) commands, you can scan the backup directory to find existing backups, even if you never used PBM on your computer before. +Percona Backup for MongoDB (PBM) saves backup data to a designated directory on the backup storage. It can be a specific directory you define for the storage or the root folder. -## Supported storage types +Each backup is prefixed with the UTC starting time for easy identification and consists of: -Percona Backup for MongoDB supports the following storage types: +* A metadata file containing backup information +* For each replica set: -* [S3-compatible storage](s3-storage.md) -* [Filesystem server storage](filesystem-storage.md) -* [Microsoft Azure Blob storage](azure.md) + * A compressed mongodump archive of all collections + * A compressed BSON file containing the oplog entries for the backup period +The oplog entries ensure backup consistency, and the end time of the oplog slice(s) is the data-consistent point in time of a backup snapshot. + +Using the [`pbm list`](../reference/pbm-commands.md#pbm-list) or [`pbm status`](../reference/pbm-commands.md#pbm-status) commands, you can scan the backup directory to find existing backups, even if you never used PBM on your computer before. ## Permissions setup @@ -61,6 +66,8 @@ The following example shows the permissions configuration to the `pbm-testing` b } ``` +### Storage-specific documentation + Please refer to the documentation of your selected storage for the data access management. !!! admonition "See also" diff --git a/docs/features/split-merge-backup.md b/docs/features/split-merge-backup.md index b5a96372..d0cb023b 100644 --- a/docs/features/split-merge-backup.md +++ b/docs/features/split-merge-backup.md @@ -9,6 +9,7 @@ The following table provides default maximum size limits for the supported backu | Storage | Default size limit| | :--- | :--- | | **AWS S3** | 4.9 TB | +| **MinIO and S3-compatible storage** | 4.9 TB | | **GCS** | 4.9 TB | | **Azure Blob Storage** | 190 TB | | **Filesystem storage** | 4.9 TB | diff --git a/docs/install/backup-storage.md b/docs/install/backup-storage.md index b8974150..699c9be0 100644 --- a/docs/install/backup-storage.md +++ b/docs/install/backup-storage.md @@ -26,6 +26,20 @@ Percona Backup for MongoDB needs its own dedicated S3 bucket exclusively for bac secret-access-key: ``` + === ":simple-minio: S3-compatible (MinIO)" + + ```yaml + storage: + type: minio + minio: + endpoint: minio.example.com:9000 + bucket: pbm-test-bucket + prefix: data/pbm/backup + credentials: + access-key-id: + secret-access-key: + ``` + === ":material-google-cloud: GCS (SA)" ```yaml diff --git a/docs/reference/configuration-options.md b/docs/reference/configuration-options.md index 4d6aedf8..fd49024e 100644 --- a/docs/reference/configuration-options.md +++ b/docs/reference/configuration-options.md @@ -7,9 +7,9 @@ *Type*: string
*Required*: YES -Remote backup storage type. Supported values: `s3`, `filesystem`, `azure`. +Remote backup storage type. Supported values: `s3`, `minio`, `gcs`, `filesystem`, `azure`. -## S3 type storage options +## AWS S3 storage options ```yaml storage: @@ -64,7 +64,7 @@ The name of the storage bucket. See the [AWS Bucket naming rules](https://docs.a *Required*: YES (for AWS) The location of the storage bucket. -Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) and [GCS region list](https://cloud.google.com/storage/docs/locations) to define the bucket region +Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region ### storage.s3.prefix @@ -85,7 +85,7 @@ The URL to access the bucket. *Type*: array of strings
*Required*: NO -The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/s3-storage.md#multiple-endpoints-to-the-same-s3-storage) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. +The list of custom paths for `pbm-agents` on different servers to the same storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3 and Microsoft Azure Blob storages. Available with version 2.8.0. ### storage.s3.forcePathStyle @@ -255,6 +255,158 @@ The maximum time to wait before the next retry, specified as a *time.Duration*. The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the limit. Read more about [Managing large backup files](../features/split-merge-backup.md). +## MinIO type storage options + +You can use this storage type for other S3-compatible storage services + +```yaml +storage: + type: minio + minio: + region: + bucket: + prefix: + endpoint: + endpointMap: + "node01:2017": + "node02:2017": + secure: false + insecureSkipTLSVerify: false + forcePathStyle: false + credentials: + access-key-id: + secret-access-key: + session-token: + signature-ver: V4 + partSize: 10485760 (10 MB) + retryer: + numMaxRetries: 10 + maxObjSizeGB: 5018 + debugTrace: false +``` + +### storage.minio.region + +*Type*: string
+*Required*: NO + +The location of the storage bucket. Use the [AWS region list](https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) to define the bucket region. If not specified, the default `us-east-1` region is used. + +### storage.minio.bucket + + +*Type*: string
+*Required*: YES + +The name of the storage bucket. See the [AWS Bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html#bucketnamingrules) for bucket name requirements. + +### storage.minio.prefix + + +*Type*: string
+*Required*: NO + +The path to the data directory in the bucket. If undefined, backups are stored in the bucket’s root directory. + +### storage.minio.endpoint + +*Type*: string
+*Required*: YES + +The network address (URL or IP:port) where your MinIO server is accessible. + +### storage.minio.endpointMap + +*Type*: array of strings
+*Required*: NO + +A mapping of custom endpoints for `pbm-agents` on different servers to the same MinIO storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same S3 storage](../details/endpoint-map.md) section. Supported for Amazon S3, MinIO, and Microsoft Azure Blob storages. Available with version 2.8.0. + +### storage.minio.secure + +*Type*: boolean
+*Required*: NO
+*Default*: false + +Defines whether to use HTTP or HTTPS protocol for communication between PBM and S3 storage. Default: `false`. + +### storage.minio.insecureSkipTLSVerify + +*Type*: boolean
+*Required*: NO
+*Default*: false + +Disables the TLS verification of the MinIO / S3-compatible storage. This allows Percona Backup for MongoDB to upload data to MinIO / S3-compatible storages that use self-issued TLS certificates. Use it with caution as it might leave a hole for man-in-the-middle attacks. + +### storage.minio.forcePathStyle + +*Type*: boolean
+*Required*: NO
+*Default*: false + +Enforces the use of [path style access](../reference/glossary.md#path-style-access-to-the-storage) to the storage. Default is `false` which means PBM uses the [virtual-hosted-style](../reference/glossary.md#virtual-hosted-style-access) access to the storage + +### storage.minio.credentials.access-key-id + +*Type*: string
+*Required*: YES + +Your access key to the storage bucket. + +### storage.minio.credentials.secret-access-key + +*Type*: string
+*Required*: YES + +The key to sign your programmatic requests to the storage bucket. + +### storage.minio.credentials.session-token + +*Type*: string
+*Required*: NO + +The MinIO session token used to validate the temporary security credentials for accessing the storage. + +### storage.minio.credentials.signature-ver + +*Type*: string
+*Required*: NO
+*Default*: V4 + +Specifies the AWS Signature version to use for authentication. Accepted values: `V2`, `V4`. + +Allows using the deprecated AWS Signature version 2 for backward compatibility with storages that don't support Signature version 4. Default: `V4`. + +### storage.minio.partSize + +*Type*: int
+*Required*: NO + +The size of data chunks in bytes to be uploaded to the storage bucket. Default: 10MB. + +### storage.minio.retryer.numMaxRetries + +*Type*: int
+*Required*: NO
+*Default*: 10 + +The maximum number of retries to upload data to MinIO / S3-compatible storage. A zero value means no retries will be performed. + +### storage.minio.maxObjSizeGB + +*Type*: float64
+*Required*: NO
+*Default*: 5018 + +The maximum file size to be stored on the backup storage. If the file to upload exceeds this limit, PBM splits it in pieces, each of which falls within the limit. Read more about [Managing large backup files](../features/split-merge-backup.md). + +### storage.minio.debugTrace + +*Type*: boolean
+*Required*: NO + +If set to `true`, outputs all http communication trace in PBM log. Default: false. + ## GCS type storage options ```yaml @@ -411,6 +563,14 @@ The name of the storage container. See the [Container names](https://docs.micro The URL to access the data in Microsoft Azure Blob Storage. The default value is `https://.blob.core.windows.net`. +### storage.azure.endpointUrlMap + +*Type*: object (host:port -> endpoint URL)
+*Required*: NO + +A mapping of custom endpoint URLs for `pbm-agents` on different servers to the same remote storage. Use this option if `pbm-agents` reside on servers hidden behind different network configurations. Read more in the [Support for multiple endpoints to the same remote storage](../details/endpoint-map.md) section. Available with version 2.8.0. + + ### storage.azure.prefix *Type*: string
diff --git a/docs/reference/glossary.md b/docs/reference/glossary.md index ecfa3b36..a272731f 100644 --- a/docs/reference/glossary.md +++ b/docs/reference/glossary.md @@ -50,6 +50,10 @@ Once a transaction is committed, it will remain so. An EBS (Amazon Elastic Block Storage) snapshot is the point-in-time copy of your data, and can be used to enable disaster recovery, migrate data across regions and accounts, and improve backup compliance. +## Endpoint + +The network address (URL or IP) where an S3-compatible storage service (like MinIO) is accessible. + ## GCP GCP (Google Cloud Platform) is the set of services, including storage service, that runs on Google Cloud infrastructure. @@ -82,6 +86,10 @@ A compressed bundle of [oplog](#oplog) entries stored in the Oplog Store databas A unique identifier of an operation such as backup, restore, resync. When a pbm-agent starts processing an operation, it acquires a lock and an opID. This prevents processing the same operation twice (for example, if there are network issues in distributed systems). Using opID as a log filter allows viewing logs for an operation in progress. +## Path-style access to the storage + +A method of constructing S3 URLs where the bucket name appears in the path portion of the URL. The URL format is ``. Preferred for S3-compatible storage systems like MinIO, especially in environments without wildcard DNS or custom SSL certificates. + ## `pbm-agent` A `pbm-agent` is a PBM process running on the mongod node for backup and restore operations. A pbm-agent instance is required for every mongod node (including replica set secondary members and config server replica set nodes). @@ -122,3 +130,6 @@ Server-side encryption is the encryption of data by the remote storage server as Technical preview features are not yet ready for enterprise use and are not included in support via SLA. They are included in this release so that users can provide feedback prior to the full release of the feature in a future GA release (or removal of the feature if it is deemed not useful). This functionality can change (APIs, CLIs, etc.) from tech preview to GA. +## Virtual-hosted-style access + +A method of constructing S3 URLs where the bucket name is part of the domain name. The URL format is ``. Required by AWS S3 in newer regions; enables better routing and performance in large-scale deployments. diff --git a/docs/troubleshoot/index.md b/docs/troubleshoot/index.md index 4efb2f37..1143f53e 100644 --- a/docs/troubleshoot/index.md +++ b/docs/troubleshoot/index.md @@ -4,6 +4,7 @@ If you encounter issues when working with Percona Backup for MongoDB, consult th - [Diagnose PBM](troubleshooting.md) - [Check PBM status](status.md) +- [Generate diagnostics report](pbm-report.md) - [Partially done physical restores](restore-partial.md) - [Troubleshoot backup management via Percona Monitoring and Management](pbm-pmm.md) - [FAQ](faq.md) diff --git a/mkdocs-base.yml b/mkdocs-base.yml index 49d7758b..9f46eae7 100644 --- a/mkdocs-base.yml +++ b/mkdocs-base.yml @@ -228,10 +228,12 @@ nav: - Backup management via PMM: pmm.md - Storage: - Remote backup storage overview: details/storage-configuration.md - - AWS or s3-compatible storage: details/s3-storage.md + - AWS storage: details/s3-storage.md + - details/minio.md - details/gcs.md - - details/filesystem-storage.md - details/azure.md + - details/filesystem-storage.md + - details/endpoint-map.md - features/multi-storage.md - features/split-merge-backup.md - PBM commands: reference/pbm-commands.md