Skip to content

Small changes to S3 repo docs #3350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 7, 2025
Merged

Small changes to S3 repo docs #3350

Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b759bef
Small changes to S3 repo docs
DaveCTurner Oct 6, 2025
f3e8f2a
IMDSv1 is not supported
DaveCTurner Oct 6, 2025
e413008
Merge branch 'main' into 2025/10/06/s3-repo-more-clarifications
DaveCTurner Oct 7, 2025
3213015
Update deploy-manage/tools/snapshot-and-restore/s3-repository.md
DaveCTurner Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions deploy-manage/tools/snapshot-and-restore/s3-repository.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,7 @@ See [this video](https://www.youtube.com/watch?v=ACqfyzWf-xs) for a walkthrough

## Getting started [repository-s3-usage]

To register an S3 repository, specify the type as `s3` when creating the repository. The repository defaults to using [ECS IAM Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) credentials for authentication. You can also use [Kubernetes service accounts](#iam-kubernetes-service-accounts) for authentication.

The only mandatory setting is the bucket name:
To register an S3 repository, specify the type as `s3` when creating the repository. The only mandatory setting is the bucket name:

```console
PUT _snapshot/my_s3_repository
Expand All @@ -36,6 +34,7 @@ PUT _snapshot/my_s3_repository
}
```

By default a S3 repository will attempt to obtain its credentials automatically from the environment. For instance, if {{es}} is running on an AWS EC2 instance then it will attempt to use the [EC2 Instance Metadata Service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) to obtain temporary credentials for the [instance IAM role](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html). Likewise, if {{es}} is running in AWS EC2 then it will automatically obtain temporary [ECS IAM Role](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html) credentials for authentication. You can also use [Kubernetes service accounts](#iam-kubernetes-service-accounts) for authentication. To disable this behaviour, specify an access key, a secret key, and optionally a session token, in the {{es}} keystore.

## Client settings [repository-s3-client]

Expand Down Expand Up @@ -65,7 +64,7 @@ bin/elasticsearch-keystore add s3.client.default.session_token

If you do not configure these settings then {{es}} will attempt to automatically obtain credentials from the environment in which it is running:

* Nodes running on an instance in AWS EC2 will attempt to use the EC2 Instance Metadata Service (IMDS) to obtain instance role credentials. {{es}} supports both IMDS version 1 and IMDS version 2.
* Nodes running on an instance in AWS EC2 will attempt to use the EC2 Instance Metadata Service (IMDS) to obtain instance role credentials. {{es}} supports IMDS version 2 only.
* Nodes running in a container in AWS ECS and AWS EKS will attempt to obtain container role credentials similarly.

You can switch from using specific credentials back to the default of using the instance role or container role by removing these settings from the keystore as follows:
Expand Down Expand Up @@ -385,7 +384,7 @@ There are a number of storage systems that provide an S3-compatible API, and the

By default {{es}} communicates with your storage system using HTTPS, and validates the repository’s certificate chain using the JVM-wide truststore. Ensure that the JVM-wide truststore includes an entry for your repository. If you wish to use unsecured HTTP communication instead of HTTPS, set `s3.client.CLIENT_NAME.protocol` to `http`.

There are many systems, including some from very well-known storage vendors, which claim to offer an S3-compatible API despite failing to emulate S3’s behavior in full. If you are using such a system for your snapshots, consider using a [shared filesystem repository](shared-file-system-repository.md) based on a standardized protocol such as NFS to access your storage system instead. The `s3` repository type requires full compatibility with S3. In particular it must support the same set of API endpoints, with the same parameters, return the same errors in case of failures, and offer consistency and performance at least as good as S3 even when accessed concurrently by multiple nodes. You will need to work with the supplier of your storage system to address any incompatibilities you encounter. Don't report {{es}} issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository.
There are many systems, including some from very well-known storage vendors, which claim to offer an S3-compatible API despite failing to emulate S3’s behavior in full. If you are using such a system for your snapshots, consider using a [shared filesystem repository](shared-file-system-repository.md) based on a standardized protocol such as NFS to access your storage system instead. The `s3` repository type requires full compatibility with S3. In particular it must support the same set of API endpoints, with the same parameters, return the same errors in case of failures, and offer consistency, performance, and reliability at least as good as S3 even when accessed concurrently by multiple nodes. You will need to work with the supplier of your storage system to address any incompatibilities you encounter. Don't report {{es}} issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DaveCTurner shouldn't this be checked by the verify repo API call?
Said differently, can a customer be 100% certain that their system will be ok as long as the API call gives the green light.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, as per the docs you linked the verify repo API call is just there to ...

Check for common misconfigurations in a snapshot repository

No certainty here.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel we need to be more prescriptive here. Either have an API call that can give full clearance as to repo capabilities, or specifically call out support for specific S3 providers (and as a result, be explicit about non-support of others).
Thoughts?

Copy link
Contributor Author

@DaveCTurner DaveCTurner Oct 8, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have an API call that can give full clearance as to repo capabilities

We do what we can in terms of testing compatibility by experiment but such tests cannot be a positive indicator of compatibility, they can only report failures. We call out (e.g. in these docs) the limitations of this approach:

If the analysis is successful then it detected no incorrect behaviour, but this does not mean that correct behaviour is guaranteed. The analysis attempts to detect common bugs but it does not offer 100% coverage. Additionally, it does not test the following:

  • Your repository must perform durable writes. Once a blob has been written it must remain in place until it is deleted, even after a power loss or similar disaster.
  • Your repository must not suffer from silent data corruption. Once a blob has been written, its contents must remain unchanged until it is deliberately modified or deleted.
  • Your repository must behave correctly even if connectivity from the cluster is disrupted. Reads and writes may fail in this case, but they must not return incorrect results.

specifically call out support for specific S3 providers

Right, we call out MinIO here in these docs and redirect everyone else to contact their supplier for the relevant support rather than us.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...and testing durability, consistency and erroneous behavior is indeed not something we can do as part of an API call, I understand the conundrum 😅
Thanks for your patience here @DaveCTurner , this is clear :)


You can perform some basic checks of the suitability of your storage system using the [repository analysis API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-repository-analyze). If this API does not complete successfully, or indicates poor performance, then your storage system is not fully compatible with AWS S3 and therefore unsuitable for use as a snapshot repository. However, these checks do not guarantee full compatibility.

Expand Down
Loading