Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
4f6711a
Adds Migrate Elasticsearch with minimal downtime page
KOTungseth Aug 12, 2025
6e952fa
Fixes toc
KOTungseth Aug 12, 2025
fd84629
Fixes broken links
KOTungseth Aug 12, 2025
1508c9a
Improves main title and link titles
KOTungseth Aug 12, 2025
85dff1b
Merge branch 'main' into manage-data/zero-downtime-migrate
LolloneS Aug 13, 2025
7af3e17
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 13, 2025
28a01e5
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 13, 2025
69f5bf1
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 13, 2025
96181a1
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 13, 2025
fab7ced
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 14, 2025
83d025c
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 14, 2025
d035ecc
Update migrate-data-between-elasticsearch-clusters-with-minimal-downt…
LolloneS Aug 14, 2025
78355c4
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 14, 2025
7984aec
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 14, 2025
f6b83cf
Update migrate-data-between-elasticsearch-clusters-with-minimal-downt…
LolloneS Aug 14, 2025
5e4726b
Merge branch 'main' into manage-data/zero-downtime-migrate
LolloneS Aug 14, 2025
d86b5b1
Update migrate-data-between-elasticsearch-clusters-with-minimal-downt…
LolloneS Aug 14, 2025
eb31d2f
Update migrate-data-between-elasticsearch-clusters-with-minimal-downt…
LolloneS Aug 14, 2025
981ce4b
Review part 1
KOTungseth Aug 14, 2025
18ab76d
Merge branch 'main' into manage-data/zero-downtime-migrate
LolloneS Aug 18, 2025
2ee8c59
Update migrate.md
LolloneS Aug 19, 2025
841e2c2
Integrates additional resource links into main page content
KOTungseth Aug 19, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions manage-data/migrate.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,18 +126,23 @@ Restoring from a snapshot is often the fastest and most reliable way to migrate

System indices can be restored by including their corresponding [feature states](/deploy-manage/tools/snapshot-and-restore.md#feature-state) in the restore operation, allowing you to retain internal configurations related to security, {{kib}}, or other stack features.

This method is especially useful when you want to fully replicate the source cluster or when remote reindexing is not possible, for example if the source cluster is in a degraded or unreachable state.
This method is especially useful when:

To use this method, the new cluster must have access to the snapshot repository that contains the data from the old cluster. Also ensure that both clusters use [compatible versions](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility).
* You want to fully replicate the source cluster or when remote reindexing is not possible, for example if the source cluster is in a degraded or unreachable state.
* Your old cluster contains mostly static data, allowing you to snapshot the source cluster, restore in the new cluster, and continue operations.

For more information, refer to [Restore into a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster)
When your source cluster is actively ingesting data, such as logs, metrics, or traces, and you need a seamless migration with minimal downtime, consider using the [minimal downtime migration](migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md).

### Requirements [ec-restore-snapshots-requirements]
* The new cluster must have access to the snapshot repository that contains the data from the old cluster.
* Both clusters must use [compatible versions](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility).

For more information, refer to [Restore into a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster).

::::{note}
For {{ece}} users, while it is most common to have Amazon S3 buckets, you should be able to restore from any addressable external storage that has your {{es}} snapshots.
For {{ece}}, Amazon S3 us the most common snapshot storage, but you can restor from any accesible external storage that contains your {{es}} snapshots.
::::

The following steps assume you already have a snapshot repository configured in the old cluster, with at least one valid snapshot containing the data you want to migrate.

### Step 1: Set up the repository in the new cluster [migrate-repo-setup]

In this step, you’ll configure a snapshot repository in the new cluster that points to the storage location used by the old cluster. This allows the new cluster to access and restore snapshots created in the original environment.
Expand Down
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a reference in the parent page: KOTungseth@2ee8c59

Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
navigation_title: Migrate with minimal downtime
applies_to:
stack: ga
products:
- id: elasticsearch
- id: cloud-hosted
- id: cloud-enterprise
- id: cloud-kubernetes
---

# Migrate {{es}} data with minimal downtime [migrate-elasticsearch-data-with-minimal-downtime]
When moving your data and services from one {{es}} cluster to another, such as to {{ech}}, {{ece}}, new on-premises hardware, or any other {{es}} environment, you can migrate with minimal downtime using incremental snapshots.

Migrating with incremental snapshots is useful when you want to:

* Migrate all data in your indices and configurations, such as roles and {{kib}} dashboards, from the old cluster to a new cluster.
* Ensure data ingestion, such as {{ls}} or {{beats}}, and data consumption, such as applications using {{es}} as a backend, seamlessly migrate to the new cluster.
* Maintain data consistency and minimize disruption.

## How incremental snapshots work [how-incremental-snapshots-work]
Incremental snapshots save only the data that has changed since the last snapshot. The first snapshot is a full copy of the data. Each subsequent snapshot contains only the differences, which makes creating and restoring snapshots faster and more efficient over time.

When restoring, {{es}} copies only the missing data segments from the snapshot repository to the new cluster local storage. When the changes between snapshots are small, the restore process is significantly faster.

By taking and restoring incremental snapshots in sequence, you can keep a new cluster closely synchronized with the old cluster, allowing you to migrate most of your data ahead of time and minimize downtime during the final cutover.

For more information about migrating your data with snapshot and restore, check [Snapshot and restore](/deploy-manage/tools/snapshot-and-restore.md).

## Before you begin [incremental-snapshots-before-you-begin]
Before you migrate, review the prerequisites and requirements.

### Prerequisites
* Learn how to [set up and manage snapshot repositories](/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md).
* If restoring to a different cluster, review [Restore to a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster).
* As an alternative migration method, you can [reindex from a remote cluster](/manage-data/migrate.md#ech-reindex-remote).

### Requirements
* **Cluster size** – The new cluster must be the same size or larger than the old cluster.
* **Version compatibility** – Both clusters must use compatible {{es}} versions. To check if your cluster versions are compatible, check [Snapshot version compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility).
* **Storage requirements** - Ensure sufficient repository storage. Usage grows with snapshot frequency and data volume.
* **Network overhead** – Transferring snapshots across networks, regions, or providers can be time consuming and incur costs.
* **Resource usage** – Snapshot and restore operations can be resource intensive and affect cluster performance.
* **Custom integrations** – Some integrations that use the {{es}} API directly, such as the [Elasticsearch Java Client library](elasticsearch-java://reference/index.md), can require additional handling during cutover.

## Recommended migration timeline [recommended-migration-timeline]
Tp complete the migration with minimal downtime, use incremental snapshots. While the exact sequence may differ depending on your infrastructure and operational requirements, you can use the recommended migration timeline as a reliable baseline that you can adapt. Adjust the steps and times to fit your own operational needs.

1. **09:00**: Take the initial full snapshot of the old cluster. You can also take the initial full snapshot the day before.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably should link to some snapshot and restore docs in the body of the tutorial, and maybe add a prereq to set up snapshot and restore (maybe clarifying how much of this process you should do)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to send users off to other docs in the middle of the page, which is why I included the list of relevant docs at the bottom

2. **09:30**: Restore the snapshot to the new cluster.
3. **09:55**: Take another snapshot of the old cluster and restore it to the new cluster. Repeat this process until the snapshot and restore operations take only a few seconds or minutes.
4. **10:15**: Perform the final cutover.
1. In the old cluster, pause indexing or set indices to read-only. For details on setting indices to read-only to safely pause indexing during migration, check [Index lifecycle actions: Read-only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md).
2. Take a final snapshot.
3. Restore the snapshot to the new cluster.
4. Change ingestion and querying to the new cluster.
5. Open the indices in the new cluster.

## Additional support [incremental-snapshots-additional-support]
To get expert assistance for your {{es}} migrations, go to [Elastic Professional Services](https://www.elastic.co/consulting).
1 change: 1 addition & 0 deletions manage-data/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -153,4 +153,5 @@ toc:
children:
- file: migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md
- file: migrate/migrate-internal-indices.md
- file: migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md
- file: use-case-use-elasticsearch-to-manage-time-series-data.md
Loading