|
| 1 | +--- |
| 2 | +navigation_title: Migrate Elasticsearch data with minimal downtime |
| 3 | +applies_to: |
| 4 | + stack: |
| 5 | + deployment: |
| 6 | + ess: |
| 7 | + ece: |
| 8 | + eck: |
| 9 | +products: |
| 10 | + - id: elasticsearch |
| 11 | + - id: cloud-hosted |
| 12 | + - id: cloud-enterprise |
| 13 | +--- |
| 14 | + |
| 15 | +# Migrate {{es}} data with minimal downtime using incremental snapshots [migrate-elasticsearch-data-with-minimal-downtime] |
| 16 | +When moving your data and services from one {{es}} cluster to another, such as to {{ech}}, {{ece}}, new on-premises hardware, or any other {{es}} environment, you can use incremental snapshots to minimize downtime. |
| 17 | + |
| 18 | +Migrating with incremental snapshots is useful when you want to: |
| 19 | + |
| 20 | +* Migrate all data in your indices and configurations, such as roles and {{kib}} dashboards, from the old cluster to a new cluster. |
| 21 | +* Ensure data ingestion, such as {{ls}} or {{beats}}, and data consumption, such as applications using {{es}} as a backend, seamlessly migrate to the new cluster. |
| 22 | +* Maintain data consistency and minimize disruption. |
| 23 | + |
| 24 | +## How incremental snapshots works [how-incremental-snapshots-work] |
| 25 | +Incremental snapshots capture only the data that has changed since the previous snapshot. |
| 26 | + |
| 27 | +After the initial full snapshot, each subsequent snapshot contains only the differences, which makes the snapshot process faster over time. When you restore snapshots, only the missing data segments are copied from the snapshot repository to the cluster local storage, speeding up restores when the changes between snapshots are small. |
| 28 | + |
| 29 | +When you incrementally create and restore snapshots, you can repeatedly synchronize the new cluster with the old cluster by taking and restoring multiple snapshots before performing the final cutover. |
| 30 | + |
| 31 | +## Recommended migration timeline [recommended-migration-timeline] |
| 32 | +Complete the minimal-downtime migration using incremental snapshots. While the exact sequence may differ depending on your infrastructure and operational requirements, you can use the recommended migration timeline as a reliable baseline that you can adapt. Adjust the steps and times to fit your own operational needs. |
| 33 | + |
| 34 | +1. **09:00**: Take the initial full snapshot of the old cluster. You can also take the initial full snapshot the day before. |
| 35 | +2. **09:30**: Restore the snapshot to the new cluster. |
| 36 | +3. **09:55**: Take another snapshot of the old cluster and restore it to the new cluster. Repeat this process until the snapshot and restore operations take only a few seconds or minutes. |
| 37 | +4. **10:15**: Perform the final cutover. |
| 38 | + 1. In the old cluster, pause indexing or set indices to ready-only. |
| 39 | + 2. Take a final snapshot. |
| 40 | + 3. Restore the snapshot to the new cluster. |
| 41 | + 4. Change ingestion and querying to the new cluster. |
| 42 | + 5. Open the indices in the new cluster. |
| 43 | + |
| 44 | +## Incremental snapshot limitations [incremental-snapshot-limitations] |
| 45 | +While incremental snapshots allow efficient migration with minimal downtime, consider the limitations when planning your migration. |
| 46 | + |
| 47 | +Limitations include the following: |
| 48 | +* **Storage requirements** – Sufficient repository storage is required, and usage can grow based on snapshot frequency and data volume. |
| 49 | +* **Network overhead** – Transferring snapshots across networks, regions, or providers can be time consuming and incur costs. |
| 50 | +* **Version compatibility** – Old and new clusters must use compatible {{es}} versions. To check if your cluster versions are compatible, check [Snapshot compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility). |
| 51 | +* **Custom integrations** – Some custom integrations that directly uses the {{es}} API may require additional handling during the cutover from the old cluster to the new cluster. |
| 52 | +* **Resource usage** – Initial and incremental snapshot and restore operations can be resource-intensive, potentially affecting cluster performance. |
| 53 | + |
| 54 | +## Related resources for incremental snapshots [related-incremental-snapshot-resources] |
| 55 | +For more information on migrating {{es}} data with minimal downtime using incremental snapshots, review the related resources. |
| 56 | + |
| 57 | +### Snapshot and restore |
| 58 | +* For more information about snapshot and restore concepts, check [Snapshot and Restore](/deploy-manage/tools/snapshot-and-restore.md). |
| 59 | +* To learn how to configure snapshot repositories before taking or restoring snapshots, check [Set up snapshot repositories](/deploy-manage/tools/snapshot-and-restore/self-managed.md). |
| 60 | +* To learn how to restore snapshots to clusters other than the source, check [Restore into a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster). |
| 61 | + |
| 62 | +### Cluster and index management |
| 63 | +* For details on setting indices to read-only to safely pause indexing during migration, check [Pausing indexing and read-only indices](/manage-data/index-management/index-modules.md#readonly-index). |
| 64 | +* For guidance on managing cluster-wide settings during migration, [Managing Elasticsearch cluster settings](/manage-data/cluster/index.html). |
| 65 | + |
| 66 | +### Data ingestion |
| 67 | +* For information about using {{ls}} for data ingestion, check the [Logstash documentation](/logs/logstash/index.html). |
| 68 | +* For information about using Beats for data ingestion, check [Beats documentation](/beats/index.html). |
| 69 | + |
| 70 | +### Alternative migration methods |
| 71 | +* To learn about reindexing from remote clusters as an alternative migration method, check [Reindex API](/api-docs/elasticsearch/reference/current/docs-reindex.html). |
| 72 | + |
| 73 | +### Elastic Cloud environments |
| 74 | +* To explore {{ech}} environments for migration targets, check the [Elastic Cloud (ECH) documentation](/cloud/index.html). |
| 75 | +* To explore {{ece}} environments for migration targets, check the [Elastic Cloud Enterprise (ECE) documentation](/deploy-manage/cloud-enterprise/index.html). |
| 76 | + |
| 77 | +### Additional support |
| 78 | +* To get expert assistance for your {{es}} migrations, go to [Elastic Professional Services](https://www.elastic.co/consulting). |
0 commit comments