You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: manage-data/migrate.md
+11-6Lines changed: 11 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -126,18 +126,23 @@ Restoring from a snapshot is often the fastest and most reliable way to migrate
126
126
127
127
System indices can be restored by including their corresponding [feature states](/deploy-manage/tools/snapshot-and-restore.md#feature-state) in the restore operation, allowing you to retain internal configurations related to security, {{kib}}, or other stack features.
128
128
129
-
This method is especially useful when you want to fully replicate the source cluster or when remote reindexing is not possible, forexample if the source cluster isin a degraded or unreachable state.
129
+
This method is especially useful when:
130
130
131
-
To use this method, the new cluster must have access to the snapshot repository that contains the data from the old cluster. Also ensure that both clusters use [compatible versions](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility).
131
+
* You want to fully replicate the source cluster or when remote reindexing is not possible, forexample if the source cluster isin a degraded or unreachable state.
132
+
* Your old cluster contains mostly static data, allowing you to snapshot the source cluster, restore in the new cluster, and continue operations.
132
133
133
-
For more information, refer to [Restore into a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster)
134
+
When your source cluster is actively ingesting data, such as logs, metrics, or traces, and you need a seamless migration with minimal downtime, consider using the [minimal downtime migration](migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md).
* The new cluster must have access to the snapshot repository that contains the data from the old cluster.
138
+
* Both clusters must use [compatible versions](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility).
139
+
140
+
For more information, refer to [Restore into a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster).
134
141
135
142
::::{note}
136
-
For {{ece}} users, while it is most common to have Amazon S3 buckets, you should be able to restore from any addressable external storage that has your {{es}} snapshots.
143
+
For {{ece}}, Amazon S3 us the most common snapshot storage, but you can restor from any accesible external storage that contains your {{es}} snapshots.
137
144
::::
138
145
139
-
The following steps assume you already have a snapshot repository configured in the old cluster, with at least one valid snapshot containing the data you want to migrate. While these steps are valid in general, they assume that the migration is from a cluster containing mostly "static" data. In other words, they assume that it is possible to take a snapshot of the source (old) cluster, restore it in the target (new) cluster, and move on with operations. In use-cases in which the source cluster is consistently being used fordata ingestion -for instance, to ingest time-series data such as logs, metrics, and traces- and the migration needs to be as transparent as possible, a minimal downtime approach can be of great help. More information is availablein a [dedicated section](migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md).
140
-
141
146
### Step 1: Set up the repository in the new cluster [migrate-repo-setup]
142
147
143
148
In this step, you’ll configure a snapshot repository in the new cluster that points to the storage location used by the old cluster. This allows the new cluster to access and restore snapshots created in the original environment.
Copy file name to clipboardExpand all lines: manage-data/migrate/migrate-data-between-elasticsearch-clusters-with-minimal-downtime.md
+22-32Lines changed: 22 additions & 32 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,21 +19,29 @@ Migrating with incremental snapshots is useful when you want to:
19
19
* Maintain data consistency and minimize disruption.
20
20
21
21
## How incremental snapshots work [how-incremental-snapshots-work]
22
-
Incremental snapshots capture only the data that has changed since the previous snapshot.
22
+
Incremental snapshots save only the data that has changed since the last snapshot. The first snapshot is a full copy of the data. Each subsequent snapshot contains only the differences, which makes creating and restoring snapshots faster and more efficient over time.
23
23
24
-
After the initial full snapshot, each subsequent snapshot contains only the differences, which makes the snapshot process faster over time. When you restore snapshots, only the missing data segments are copied from the snapshot repository to the cluster local storage, speeding up restores when the changes between snapshots are small.
24
+
When restoring, {{es}} copies only the missing data segments from the snapshot repository to the new cluster local storage. When the changes between snapshots are small, the restore process is significantly faster.
25
25
26
-
When you incrementally create and restore snapshots, you can repeatedly synchronize the new cluster with the old cluster by taking and restoring multiple snapshots before performing the final cutover. This mechanism enables you to copy the vast majority of your data into the new cluster ahead of time, making the future transition from the old cluster to your new one as quick as possible, since only few segments will need to go through the snapshot-and-restore procedure upon cutover.
26
+
By taking and restoring incremental snapshots in sequence, you can keep a new cluster closely synchronized with the old cluster, allowing you to migrate most of your data ahead of time and minimize downtime during the final cutover.
While incremental snapshots allow efficient migration with minimal downtime, consider the limitations when planning your migration.
28
+
For more information about migrating your data with snapshot and restore, check [Snapshot and restore](/deploy-manage/tools/snapshot-and-restore.md).
30
29
31
-
Limitations include the following:
32
-
***Storage requirements** – Sufficient repository storage is required, and usage can grow based on snapshot frequency and data volume.
33
-
***Network overhead** – Transferring snapshots across networks, regions, or providers can be time-consuming and incur costs.
34
-
***Version compatibility** – Old and new clusters must use compatible {{es}} versions. To check if your cluster versions are compatible, check [Snapshot version compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility).
35
-
***Custom integrations** – Some custom integrations that directly use the {{es}} API, such as the [Elasticsearch Java Client library](elasticsearch-java://reference/index.md), can require additional handling during the cutover from the old cluster to the new cluster.
36
-
***Resource usage** – Initial and incremental snapshot and restore operations can be resource-intensive, potentially affecting cluster performance.
30
+
## Before you begin [incremental-snapshots-before-you-begin]
31
+
Before you migrate, review the prerequisites and requirements.
32
+
33
+
### Prerequisites
34
+
* Learn how to [set up and manage snapshot repositories](/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md).
35
+
* If restoring to a different cluster, review [Restore to a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster).
36
+
* As an alternative migration method, you can [reindex from a remote cluster](/manage-data/migrate.md#ech-reindex-remote).
37
+
38
+
### Requirements
39
+
***Cluster size** – The new cluster must be the same size or larger than the old cluster.
40
+
***Version compatibility** – Both clusters must use compatible {{es}} versions. To check if your cluster versions are compatible, check [Snapshot version compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility).
41
+
***Storage requirements** - Ensure sufficient repository storage. Usage grows with snapshot frequency and data volume.
42
+
***Network overhead** – Transferring snapshots across networks, regions, or providers can be time consuming and incur costs.
43
+
***Resource usage** – Snapshot and restore operations can be resource intensive and affect cluster performance.
44
+
***Custom integrations** – Some integrations that use the {{es}} API directly, such as the [Elasticsearch Java Client library](elasticsearch-java://reference/index.md), can require additional handling during cutover.
Tp complete the migration with minimal downtime, use incremental snapshots. While the exact sequence may differ depending on your infrastructure and operational requirements, you can use the recommended migration timeline as a reliable baseline that you can adapt. Adjust the steps and times to fit your own operational needs.
@@ -42,29 +50,11 @@ Tp complete the migration with minimal downtime, use incremental snapshots. Whil
42
50
2.**09:30**: Restore the snapshot to the new cluster.
43
51
3.**09:55**: Take another snapshot of the old cluster and restore it to the new cluster. Repeat this process until the snapshot and restore operations take only a few seconds or minutes.
44
52
4.**10:15**: Perform the final cutover.
45
-
1. In the old cluster, pause indexing or set indices to read-only.
53
+
1. In the old cluster, pause indexing or set indices to read-only. For details on setting indices to read-only to safely pause indexing during migration, check [Index lifecycle actions: Read-only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md).
46
54
2. Take a final snapshot.
47
55
3. Restore the snapshot to the new cluster.
48
56
4. Change ingestion and querying to the new cluster.
49
57
5. Open the indices in the new cluster.
50
58
51
-
## Related resources [related-incremental-snapshot-resources]
52
-
For more information on migrating {{es}} data with minimal downtime using incremental snapshots, review the related resources.
53
-
54
-
### Snapshot and restore
55
-
* For more information about snapshot and restore concepts, check [Snapshot and restore](/deploy-manage/tools/snapshot-and-restore.md).
56
-
* To learn how to configure snapshot repositories before taking or restoring snapshots, check [Manage snapshot repositories](/deploy-manage/tools/snapshot-and-restore/manage-snapshot-repositories.md).
57
-
* To learn how to restore snapshots to clusters other than the source, check [Restore to a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster).
58
-
59
-
### Cluster and index management
60
-
* For details on setting indices to read-only to safely pause indexing during migration, check [Index lifecycle actions: Read-only](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-readonly.md).
61
-
62
-
### Data ingestion
63
-
* For information about using {{ls}} for data ingestion, check the [{{ls}} documentation](logstash://reference/index.md).
64
-
* For information about using {{beats}} for data ingestion, check the [{{beats}} documentation](beats://reference/index.md).
65
-
66
-
### Alternative migration methods
67
-
* To learn about reindexing from remote clusters as an alternative migration method, check [Reindex from a remote cluster](/manage-data/migrate.md#ech-reindex-remote).
68
-
69
-
### Additional support
70
-
* To get expert assistance for your {{es}} migrations, go to [Elastic Professional Services](https://www.elastic.co/consulting).
59
+
## Additional support [incremental-snapshots-additional-support]
60
+
To get expert assistance for your {{es}} migrations, go to [Elastic Professional Services](https://www.elastic.co/consulting).
0 commit comments