Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
4f6711a
Adds Migrate Elasticsearch with minimal downtime page
KOTungseth Aug 12, 2025
6e952fa
Fixes toc
KOTungseth Aug 12, 2025
fd84629
Fixes broken links
KOTungseth Aug 12, 2025
1508c9a
Improves main title and link titles
KOTungseth Aug 12, 2025
85dff1b
Merge branch 'main' into manage-data/zero-downtime-migrate
LolloneS Aug 13, 2025
7af3e17
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 13, 2025
28a01e5
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 13, 2025
69f5bf1
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 13, 2025
96181a1
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 13, 2025
fab7ced
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 14, 2025
83d025c
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 14, 2025
d035ecc
Update migrate-data-between-elasticsearch-clusters-with-minimal-downt…
LolloneS Aug 14, 2025
78355c4
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 14, 2025
7984aec
Update manage-data/migrate/migrate-data-between-elasticsearch-cluster…
LolloneS Aug 14, 2025
f6b83cf
Update migrate-data-between-elasticsearch-clusters-with-minimal-downt…
LolloneS Aug 14, 2025
5e4726b
Merge branch 'main' into manage-data/zero-downtime-migrate
LolloneS Aug 14, 2025
d86b5b1
Update migrate-data-between-elasticsearch-clusters-with-minimal-downt…
LolloneS Aug 14, 2025
eb31d2f
Update migrate-data-between-elasticsearch-clusters-with-minimal-downt…
LolloneS Aug 14, 2025
981ce4b
Review part 1
KOTungseth Aug 14, 2025
18ab76d
Merge branch 'main' into manage-data/zero-downtime-migrate
LolloneS Aug 18, 2025
2ee8c59
Update migrate.md
LolloneS Aug 19, 2025
841e2c2
Integrates additional resource links into main page content
KOTungseth Aug 19, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a reference in the parent page: KOTungseth@2ee8c59

Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
---
navigation_title: Migrate Elasticsearch data with minimal downtime
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because this is in the migration section, we could try:

Suggested change
navigation_title: Migrate Elasticsearch data with minimal downtime
navigation_title: Minimize downtime with incremental snapshots

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shainaraskas I think the goal of this article is to provide an answer to all our customers asking for "zero-downtime" migrations. Incremental snapshots are the technology enabling it, not the end goal. I am totally ok with changing the title but I would like to keep the "downtime" part in there :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a keyword search, and to optimize this page for SEO, we need to keep Elasticsearch in the title.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the nav title, not the page title, so we could simplify this so it's easier to read in the sidebar.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it! I think for the nav title @shainaraskas's suggestion makes sense as we are already under the "migrate" section.

applies_to:
stack:
deployment:
ess:
ece:
eck:
products:
- id: elasticsearch
- id: cloud-hosted
- id: cloud-enterprise
---

# Migrate {{es}} data with minimal downtime [migrate-elasticsearch-data-with-minimal-downtime]
When moving your data and services from one {{es}} cluster to another, such as to {{ech}}, {{ece}}, new on-premises hardware, or any other {{es}} environment, you can use incremental snapshots to minimize downtime.

Migrating with incremental snapshots is useful when you want to:

* Migrate all data in your indices and configurations, such as roles and {{kib}} dashboards, from the old cluster to a new cluster.
* Ensure data ingestion, such as {{ls}} or {{beats}}, and data consumption, such as applications using {{es}} as a backend, seamlessly migrate to the new cluster.
* Maintain data consistency and minimize disruption.

## How incremental snapshots works [how-incremental-snapshots-work]
Incremental snapshots capture only the data that has changed since the previous snapshot.

After the initial full snapshot, each subsequent snapshot contains only the differences, which makes the snapshot process faster over time. When you restore snapshots, only the missing data segments are copied from the snapshot repository to the cluster local storage, speeding up restores when the changes between snapshots are small.

When you incrementally create and restore snapshots, you can repeatedly synchronize the new cluster with the old cluster by taking and restoring multiple snapshots before performing the final cutover.

## Recommended migration timeline [recommended-migration-timeline]
Complete the minimal-downtime migration using incremental snapshots. While the exact sequence may differ depending on your infrastructure and operational requirements, you can use the recommended migration timeline as a reliable baseline that you can adapt. Adjust the steps and times to fit your own operational needs.

1. **09:00**: Take the initial full snapshot of the old cluster. You can also take the initial full snapshot the day before.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably should link to some snapshot and restore docs in the body of the tutorial, and maybe add a prereq to set up snapshot and restore (maybe clarifying how much of this process you should do)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to send users off to other docs in the middle of the page, which is why I included the list of relevant docs at the bottom

2. **09:30**: Restore the snapshot to the new cluster.
3. **09:55**: Take another snapshot of the old cluster and restore it to the new cluster. Repeat this process until the snapshot and restore operations take only a few seconds or minutes.
4. **10:15**: Perform the final cutover.
1. In the old cluster, pause indexing or set indices to ready-only.
2. Take a final snapshot.
3. Restore the snapshot to the new cluster.
4. Change ingestion and querying to the new cluster.
5. Open the indices in the new cluster.

## Incremental snapshot limitations [incremental-snapshot-limitations]
While incremental snapshots allow efficient migration with minimal downtime, consider the limitations when planning your migration.

Limitations include the following:
* **Storage requirements** – Sufficient repository storage is required, and usage can grow based on snapshot frequency and data volume.
* **Network overhead** – Transferring snapshots across networks, regions, or providers can be time consuming and incur costs.
* **Version compatibility** – Old and new clusters must use compatible {{es}} versions. To check if your cluster versions are compatible, check [Snapshot compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility).
* **Custom integrations** – Some custom integrations that directly use the {{es}} API can require additional handling during the cutover from the old cluster to the new cluster.
* **Resource usage** – Initial and incremental snapshot and restore operations can be resource-intensive, potentially affecting cluster performance.

## Additional topics [additional-incremental-snapshot-topics]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider integrating most of these into the body.

For more information on migrating {{es}} data with minimal downtime using incremental snapshots, review the related resources.

### Snapshot and restore
* For more information about snapshot and restore concepts, check [Snapshot and Restore](/deploy-manage/tools/snapshot-and-restore.md).
* To learn how to configure snapshot repositories before taking or restoring snapshots, check [Manage snapshot repositories](/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md).
* To learn how to restore snapshots to clusters other than the source, check [Restore to a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster).

### Cluster and index management
* For details on setting indices to read-only to safely pause indexing during migration, check [Index lifecycle actions: Read-only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md).

### Data ingestion
* For information about using {{ls}} for data ingestion, check the [{{ls}} documentation](logstash://reference/index.md).
* For information about using Beats for data ingestion, check the [{{beats}} documentation](beats://reference/index.md).

### Alternative migration methods
* To learn about reindexing from remote clusters as an alternative migration method, check [Reindex documents API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex).

### Elastic Cloud environments
* To explore {{ech}} environments for migration targets, check the [{{ech}} documentation](/deploy-manage/deploy/elastic-cloud/cloud-hosted.md).
* To explore {{ece}} environments for migration targets, check the [{{ece}} documentation](/deploy-manage/deploy/cloud-enterprise.md).

### Additional support
* To get expert assistance for your {{es}} migrations, go to [Elastic Professional Services](https://www.elastic.co/consulting).
1 change: 1 addition & 0 deletions manage-data/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ toc:
- file: lifecycle/rollup/migrating-from-rollup-to-downsampling.md
- file: migrate.md
children:
- file: migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md
- file: migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md
- file: migrate/migrate-internal-indices.md
- file: use-case-use-elasticsearch-to-manage-time-series-data.md
Loading