Skip to content

Commit 3458e90

Browse files
KOTungsethLolloneSvishaangelovashainaraskas
authored
Adds Migrate Elasticsearch with minimal downtime page (#2533)
Adds content from the [How to migrate Elasticsearch clusters with zero downtime](https://docs.google.com/document/d/1GE-nY6ZIIsERN9D7YTpDsyDaZuzncNeq2JrQRtNpK5Y/edit?tab=t.0#heading=h.46xvrgy4mbk6) blog post. Closes elastic/docs-content-internal#53 --------- Co-authored-by: Lorenzo <[email protected]> Co-authored-by: Visha Angelova <[email protected]> Co-authored-by: shainaraskas <[email protected]> Co-authored-by: Lorenzo <[email protected]>
1 parent f165c99 commit 3458e90

File tree

3 files changed

+72
-6
lines changed

3 files changed

+72
-6
lines changed

manage-data/migrate.md

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -126,18 +126,23 @@ Restoring from a snapshot is often the fastest and most reliable way to migrate
126126

127127
System indices can be restored by including their corresponding [feature states](/deploy-manage/tools/snapshot-and-restore.md#feature-state) in the restore operation, allowing you to retain internal configurations related to security, {{kib}}, or other stack features.
128128

129-
This method is especially useful when you want to fully replicate the source cluster or when remote reindexing is not possible, for example if the source cluster is in a degraded or unreachable state.
129+
This method is especially useful when:
130130

131-
To use this method, the new cluster must have access to the snapshot repository that contains the data from the old cluster. Also ensure that both clusters use [compatible versions](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility).
131+
* You want to fully replicate the source cluster or when remote reindexing is not possible, for example if the source cluster is in a degraded or unreachable state.
132+
* Your old cluster contains mostly static data, allowing you to snapshot the source cluster, restore in the new cluster, and continue operations.
132133

133-
For more information, refer to [Restore into a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster)
134+
When your source cluster is actively ingesting data, such as logs, metrics, or traces, and you need a seamless migration with minimal downtime, consider using the [minimal downtime migration](migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md).
135+
136+
### Requirements [ec-restore-snapshots-requirements]
137+
* The new cluster must have access to the snapshot repository that contains the data from the old cluster.
138+
* Both clusters must use [compatible versions](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility).
139+
140+
For more information, refer to [Restore into a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster).
134141

135142
::::{note}
136-
For {{ece}} users, while it is most common to have Amazon S3 buckets, you should be able to restore from any addressable external storage that has your {{es}} snapshots.
143+
For {{ece}}, Amazon S3 us the most common snapshot storage, but you can restor from any accesible external storage that contains your {{es}} snapshots.
137144
::::
138145

139-
The following steps assume you already have a snapshot repository configured in the old cluster, with at least one valid snapshot containing the data you want to migrate.
140-
141146
### Step 1: Set up the repository in the new cluster [migrate-repo-setup]
142147

143148
In this step, you’ll configure a snapshot repository in the new cluster that points to the storage location used by the old cluster. This allows the new cluster to access and restore snapshots created in the original environment.
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
navigation_title: Migrate with minimal downtime
3+
applies_to:
4+
stack: ga
5+
products:
6+
- id: elasticsearch
7+
- id: cloud-hosted
8+
- id: cloud-enterprise
9+
- id: cloud-kubernetes
10+
---
11+
12+
# Migrate {{es}} data with minimal downtime [migrate-elasticsearch-data-with-minimal-downtime]
13+
When moving your data and services from one {{es}} cluster to another, such as to {{ech}}, {{ece}}, new on-premises hardware, or any other {{es}} environment, you can migrate with minimal downtime using incremental snapshots.
14+
15+
Migrating with incremental snapshots is useful when you want to:
16+
17+
* Migrate all data in your indices and configurations, such as roles and {{kib}} dashboards, from the old cluster to a new cluster.
18+
* Ensure data ingestion, such as {{ls}} or {{beats}}, and data consumption, such as applications using {{es}} as a backend, seamlessly migrate to the new cluster.
19+
* Maintain data consistency and minimize disruption.
20+
21+
## How incremental snapshots work [how-incremental-snapshots-work]
22+
Incremental snapshots save only the data that has changed since the last snapshot. The first snapshot is a full copy of the data. Each subsequent snapshot contains only the differences, which makes creating and restoring snapshots faster and more efficient over time.
23+
24+
When restoring, {{es}} copies only the missing data segments from the snapshot repository to the new cluster local storage. When the changes between snapshots are small, the restore process is significantly faster.
25+
26+
By taking and restoring incremental snapshots in sequence, you can keep a new cluster closely synchronized with the old cluster, allowing you to migrate most of your data ahead of time and minimize downtime during the final cutover.
27+
28+
For more information about migrating your data with snapshot and restore, check [Snapshot and restore](/deploy-manage/tools/snapshot-and-restore.md).
29+
30+
## Before you begin [incremental-snapshots-before-you-begin]
31+
Before you migrate, review the prerequisites and requirements.
32+
33+
### Prerequisites
34+
* Learn how to [set up and manage snapshot repositories](/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md).
35+
* If restoring to a different cluster, review [Restore to a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster).
36+
* As an alternative migration method, you can [reindex from a remote cluster](/manage-data/migrate.md#ech-reindex-remote).
37+
38+
### Requirements
39+
* **Cluster size** – The new cluster must be the same size or larger than the old cluster.
40+
* **Version compatibility** – Both clusters must use compatible {{es}} versions. To check if your cluster versions are compatible, check [Snapshot version compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility).
41+
* **Storage requirements** - Ensure sufficient repository storage. Usage grows with snapshot frequency and data volume.
42+
* **Network overhead** – Transferring snapshots across networks, regions, or providers can be time consuming and incur costs.
43+
* **Resource usage** – Snapshot and restore operations can be resource intensive and affect cluster performance.
44+
* **Custom integrations** – Some integrations that use the {{es}} API directly, such as the [Elasticsearch Java Client library](elasticsearch-java://reference/index.md), can require additional handling during cutover.
45+
46+
## Recommended migration timeline [recommended-migration-timeline]
47+
Tp complete the migration with minimal downtime, use incremental snapshots. While the exact sequence may differ depending on your infrastructure and operational requirements, you can use the recommended migration timeline as a reliable baseline that you can adapt. Adjust the steps and times to fit your own operational needs.
48+
49+
1. **09:00**: Take the initial full snapshot of the old cluster. You can also take the initial full snapshot the day before.
50+
2. **09:30**: Restore the snapshot to the new cluster.
51+
3. **09:55**: Take another snapshot of the old cluster and restore it to the new cluster. Repeat this process until the snapshot and restore operations take only a few seconds or minutes.
52+
4. **10:15**: Perform the final cutover.
53+
1. In the old cluster, pause indexing or set indices to read-only. For details on setting indices to read-only to safely pause indexing during migration, check [Index lifecycle actions: Read-only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md).
54+
2. Take a final snapshot.
55+
3. Restore the snapshot to the new cluster.
56+
4. Change ingestion and querying to the new cluster.
57+
5. Open the indices in the new cluster.
58+
59+
## Additional support [incremental-snapshots-additional-support]
60+
To get expert assistance for your {{es}} migrations, go to [Elastic Professional Services](https://www.elastic.co/consulting).

manage-data/toc.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,4 +153,5 @@ toc:
153153
children:
154154
- file: migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md
155155
- file: migrate/migrate-internal-indices.md
156+
- file: migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md
156157
- file: use-case-use-elasticsearch-to-manage-time-series-data.md

0 commit comments

Comments
 (0)