Skip to content
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ In this example, we assume the Identity Provider does not publish its SAML metad
## Example: Custom JVM trust store bundle [ece-add-custom-bundle-example-cacerts]
If you are using SSL certificates signed by non-public certificate authorities, {{es}} is not able to communicate with the services using those certificates unless you import a custom JVM trust store containing the certificates of your signing authority into your {{ece}} installation. You’ll need the trust store to access snapshot repositories like Minio, for your {{ece}} proxy, or to reindex from remote.
If you are using SSL certificates signed by non-public certificate authorities, {{es}} is not able to communicate with the services using those certificates unless you import a custom JVM trust store containing the certificates of your signing authority into your {{ece}} installation. You’ll need the trust store to access snapshot repositories like MinIO, for your {{ece}} proxy, or to reindex from remote.
To import a JVM trust store:
Expand Down
Binary file not shown.
Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ When a repository is assigned to a deployment, a snapshot is taken every 30 minu
* AWS S3
* Azure Blob Storage
* Google Cloud Storage
* Minio S3
* MinIO S3

::::{note}
No repository types other than those listed are supported in the {{ece}} platform, even if they are supported by {{es}}.
Expand All @@ -36,7 +36,7 @@ For more details about how snapshots are used with {{es}}, check [Snapshot and R
* [Google Cloud Storage documentation](https://cloud.google.com/storage/docs/)

::::{tip}
If you are installing ECE without internet access (commonly called an offline or air-gapped installation), you will need to use an on-premise storage service. We suggest that you use [Minio](https://www.minio.io/). For our installation notes, check [Snapshotting to Minio On-Premise Storage](minio-on-premise-repository.md).
If you are installing ECE without internet access (commonly called an offline or air-gapped installation), you will need to use an on-premise storage service. We suggest that you use [MinIO](https://www.minio.io/). For our installation notes, check [Snapshotting to MinIO On-Premise Storage](minio-on-premise-repository.md).
::::


Expand All @@ -47,7 +47,7 @@ The following guides provide instructions on adding a snapshot repository in ECE
* [AWS S3](/deploy-manage/tools/snapshot-and-restore/ece-aws-custom-repository.md)
* [Azure](/deploy-manage/tools/snapshot-and-restore/azure-storage-repository.md)
* [Google Cloud Storage](/deploy-manage/tools/snapshot-and-restore/google-cloud-storage-gcs-repository.md)
* [Minio](/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md)
* [MinIO](/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md)

## Edit snapshot repository configurations [ece_edit_snapshot_repository_configurations]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ When a platform-level repository is associated with a deployment, the `found-sna
* [AWS S3](/deploy-manage/tools/snapshot-and-restore/ece-aws-custom-repository.md)
* [Azure](/deploy-manage/tools/snapshot-and-restore/azure-storage-repository.md)
* [Google Cloud Storage](/deploy-manage/tools/snapshot-and-restore/google-cloud-storage-gcs-repository.md)
* [Minio](/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md)
* [MinIO](/deploy-manage/tools/snapshot-and-restore/minio-on-premise-repository.md)

:::{note}
No repository types other than those listed are supported in the {{ece}} platform, even if they are supported by {{es}}.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,83 +8,76 @@ products:
- id: cloud-enterprise
---

# Minio on-premise repository [ece-configuring-minio]
# MinIO self-managed repository [ece-configuring-minio]

Minio is a popular, open-source distributed object storage server compatible with the Amazon AWS S3 API. You can use it with {{ece}} installations when you want to store your {{es}} snapshots locally.
[MinIO](https://min.io/docs/minio/container/index.html) is a popular, open-source object storage server compatible with the Amazon AWS S3 API. As an [S3 compatible service](/deploy-manage/tools/snapshot-and-restore/s3-repository.md#repository-s3-compatible-services), MinIO is supported for use as a snapshot repository in {{ece}} (ECE).

This guide walks you through integrating MinIO with ECE to store your {{es}} snapshots.

## Create a test environment [ece-minio-test]
::::{important}
Avoid running MinIO directly on ECE hosts. Sharing infrastructure can lead to resource contention, especially disk I/O, and may affect the performance and stability of your Elastic workloads. It also complicates upgrades, troubleshooting, and supportability.

We recommend following the [Minio Quickstart Guide Docker Container instructions](https://docs.minio.io/docs/minio-docker-quickstart-guide) to create a simple Minio standalone installation for your initial evaluation and development.

Be sure to use the `docker -v` option to map persistent storage to the container.


## Production environment prerequisites [ece-minio-requirements]

Installing Minio for production requires a high-availability configuration where Minio is running in [Distributed mode](https://docs.minio.io/docs/distributed-minio-quickstart-guide).

As mentioned in the Minio documentation, you will need to have 4-16 Minio drive mounts. There is no hard limit on the number of Minio nodes. It might be convenient to place the Minio node containers on your ECE hosts to ensure you have a suitable level of availability, but those can not be located on the same hosts as ECE proxies since they both listen on the same port.

The following illustration is a sample architecture for a [large ECE installation](../../deploy/cloud-enterprise/deploy-large-installation.md). Note that there is at least one MinIO container in *each* availability zone.

There are a number of different ways of orchestrating the Minio deployment (Docker Compose, Kubernetes, and so on). We suggest you use the method most familiar to you.

We recommend:

* Using a single Minio endpoint with the {{ece}} installation, to simplify repository management.
* Securing access to the Minio endpoint with TLS.
If you're evaluating MinIO in a test system, do not place MinIO containers on the same hosts as ECE proxies, as both services use the same port.
::::

:::{image} /deploy-manage/images/cloud-enterprise-ece-minio-large-arch.png
:alt: Architecture diagram
:name: img-ece-minio-large-arch
:::
## Deploy MinIO

This section provides guidance and recommendations for deploying MinIO. It does not include installation steps. As MinIO is a third-party product, its deployment, configuration, and maintenance are outside the scope of Elastic support.

## Create an offline installation [ece-minio-offline-installation]
For installation instructions, refer to the official [MinIO documentation](https://min.io/docs/).

If you are installing MinIO offline, the process is very similar to the [offline installation of {{ece}}](../../deploy/cloud-enterprise/air-gapped-install.md). There are two options:
The performance and reliability of MinIO depend on its configuration and the underlying infrastructure. Consider the following best practices:

* Use a private Docker repository and [install the Minio images in the private repository](https://docs.docker.com/registry/deploying/).
* Download the Minio images from an internet-connected machine, then use docker save to bundle the images into tar files. Copy the TAR files to the target hosts and use `docker load` to install.
* For production use, deploy MinIO in [distributed mode](https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.html#minio-mnmd).
* Use a single MinIO endpoint with the ECE installation, to simplify repository configuration.
* Secure access to the MinIO endpoint with TLS.

Gather the following after your installation:
After deployment, make sure you collect the following values:

* Minio AccessKey
* Minio SecretKey
* Endpoint URL
* MinIO Access Key
* MinIO Secret Key
* MinIO endpoint URL

::::{tip}
Minio might report various Endpoint URLs, be sure to choose the one that will be routable from your {{es}} Docker containers.
MinIO may report multiple endpoint URLs. Be sure to select the one reachable from your {{es}} containers running on ECE allocator hosts.
::::

### Testing and evaluation

Use the [MinIO Quickstart Guide](https://charts.min.io/) or the [container deployment guide](https://min.io/docs/minio/container/index.html) to spin up a simple standalone MinIO container. Use `-v` to map persistent storage when using the `docker` or `podman` options.

## Create the S3 bucket [ece-minio-create-s3-bucket]

How you create the AWS S3 bucket depends on what version of {{es}} you are using:
### Production environments

* For version 7.x:
Set up MinIO in distributed mode across multiple nodes and drives. You can use Docker Compose, Kubernetes, or another orchestration tool of your choice.

1. Using the Minio browser or an S3 client application, create an S3 bucket to store your snapshots.
2. [Log into the Cloud UI](../../deploy/cloud-enterprise/log-into-cloud-ui.md) and [add the S3 repository plugin](elasticsearch://reference/elasticsearch-plugins/plugin-management.md) to your cluster.
## Create the S3 bucket [ece-minio-create-s3-bucket]

* For versions 8.0 and later, {{es}} has built-in support for AWS S3 repositories; no repository plugin is needed. Use the Minio browser or an S3 client application to create an S3 bucket to store your snapshots.
After installing MinIO you will need to create a bucket to store your deployments' snapshots. Use the MinIO browser or an S3 client application to create an S3 bucket to store your snapshots.

::::{tip}
Don’t forget to make the bucket name DNS-friendly, for example no underscores or uppercase letters. For more details, read the [bucket restrictions](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html).
::::

## {{ece}} configuration [ece-install-with-minio]

This section describes the configuration changes required to use MinIO storage within ECE to make periodic snapshots of your {{es}} deployments. The required steps include:

* Configuring the repository at ECE level
* Associating it with your deployments
* Applying specific YAML settings to the deployments

## {{ece}} configuration [ece-install-with-minio]
### Prerequisites

You can configure existing deployments, or create new ones, with the following changes to use Minio storage.
Before integrating ECE with MinIO, ensure you have the following details from your MinIO deployment:

* MinIO Access Key
* MinIO Secret Key
* MinIO endpoint URL
* S3 bucket name

### Add the repository to {{ece}} [ece-add-repository]

You must add the new repository to {{ece}} before it can be used with your {{es}} clusters.
You must add the new repository at ECE platform level before it can be used by your {{es}} deployments.

1. [Log into the Cloud UI](../../deploy/cloud-enterprise/log-into-cloud-ui.md).
2. From the **Platform** menu, select **Repositories**.
Expand All @@ -97,76 +90,48 @@ You must add the new repository to {{ece}} before it can be used with your {{es}
"type": "s3",
"settings": {
"bucket": "ece-backup",
"access_key": "<your Minio AccessKey>",
"secret_key": "<your Minio SecretKey>",
"endpoint": "<your Minio endpoint URL>:9000",
"access_key": "<your MinIO AccessKey>",
"secret_key": "<your MinIO SecretKey>",
"endpoint": "<your MinIO endpoint URL>:9000",
"path_style_access": "true",
"protocol": "http"
}
}
```

:::{image} /deploy-manage/images/cloud-enterprise-ece-minio-repository.png
:alt: Create form
:name: img-ece-minio-repository
:::

6. Select **Save** to submit your configuration.

The Minio repository is now available from the drop-down list of repositories when creating deployments.
### Associate repository with deployments

Once the MinIO repository is created at the ECE platform level, you can associate it with your {{es}} deployments in two ways:

:::{image} /deploy-manage/images/cloud-enterprise-ece-minio-deployment.png
:alt: Create deployment
:name: img-ece-minio-deployment
:::
* For new deployments, select the repository from the **Snapshot repository** drop-down list while [creating the deployment](/deploy-manage/deploy/cloud-enterprise/create-deployment.md).

* For existing deployments, associate the repository by following the instructions in [Manage {{es}} clusters repositories](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md#ece-manage-repositories-clusters).

### Additional settings for 6.x clusters [ece-6.x-settings]
### Additional settings for {{es}} [ece-6.x-settings]

For {{es}} versions 6.0 and later, after selecting the repository, you also need to set your **User Settings** YAML to specify the endpoint and protocol. For example:
After selecting the repository, you also need to configure your [{{es}} user settings YAML](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings-elasticsearch.md) to specify the endpoint and protocol. For example:

```
s3.client.default.endpoint: "<your Minio endpoint>:9000"
s3.client.default.endpoint: "<your MinIO endpoint>:9000"
s3.client.default.protocol: http
```
Check the [{{es}} S3 plugin details](https://www.elastic.co/guide/en/elasticsearch/plugins/6.8/repository-s3-client.html) for more information.

Refer to the [{{es}} S3 plugin details](/deploy-manage/tools/snapshot-and-restore/s3-repository.md) for more information.

## Upgrade from 5.x to 6.x {{es}} clusters [ece-upgrade-minio]
#### Add S3 repository plugin (only for {{es}} 7.x)

The configuration options for the {{es}} S3 repository plugin have changed from 5.x to 6.x versions and you must copy the endpoint and protocol values from your repository configuration to your **User Settings** YAML before you upgrade.
For {{es}} clusters in version 7.x you must add the S3 repository plugin to your cluster. Refer to [Managing plugins for ECE](elasticsearch://reference/elasticsearch-plugins/plugin-management.md#managing-plugins-for-ece) for more details.

::::{note}
For versions 8.0 and later, {{es}} has built-in support for AWS S3 repositories; no repository plugin is needed.
::::

## Verify snapshots [ece-minio-verify-snapshot]

The cluster should make a snapshot when the repository is set up. You can check that by going to the **Elasticsearch** and then the **Snapshots** page.

As an extra verification step, you can restore a cluster using the snapshots that have been taken.

1. [Log into the Cloud UI](../../deploy/cloud-enterprise/log-into-cloud-ui.md).
2. Get the plan from your test cluster.

1. From the **Deployments** page, select your deployment.

Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters.

2. From your deployment menu, go to the **Edit** page then go to the bottom of the page and select **advanced {{es}} configuration**.
3. Copy the JSON format under the **Deployment configuration** heading.

3. Create a new {{es}} cluster as your target.
4. On the new cluster, open the advanced cluster configuration editor. In the transient section, add the `restore_snapshot` settings to the plan.

```json
...
"transient": {
"restore_snapshot": {
"repository_name": "<Minio repository name>",
"snapshot_name": "latest_success"
}
}
```

5. Select **Save** to restore from the snapshot. When the plan update is complete, you can check the restored indexes in your target cluster.
The cluster should make periodic snapshots when the repository is set up and associated to it. You can check this in the **Elasticsearch > Snapshots** section of the deployment page in the [Cloud UI](../../deploy/cloud-enterprise/log-into-cloud-ui.md).

More details are available to [work with snapshots](../snapshot-and-restore.md).
As an extra verification step, you can [restore snapshots across clusters](/deploy-manage/tools/snapshot-and-restore/ece-restore-across-clusters.md).

Refer to [work with snapshots](../snapshot-and-restore.md) for more information around {{es}} snapshot and restore.
Loading