Skip to content

Commit 37e54c2

Browse files
Manage data -> Data migration to ECH or ECE updated (#1426)
When working on the issue #1391 I realized that the migration guide was a bit outdated and had multiple important issues to address. I have refined the content and applied major updates. Updated docs: - [Migrate your Elasticsearch data](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/1426/manage-data/migrate) - [Migrate system indices](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/1426/manage-data/migrate/migrate-internal-indices) Summary of fixes and changes: - Internal / system indices migration based on snapshot was incorrect -> feature states should be used. - Internal / system indices migration based on reindexing was incorrect -> the procedure was faulty and needed extra love and research. But considering that based on Elasticsearch docs we state that from 8.0 onwards, `feature states` are the only way to backup and restore system indices and system data streams, I have removed the reindex option for this. - Snapshot repository creation for ECE was invalid, as users shouldn't create a repository for data migration at ECE-level, but directly at deployment-level (this is a gap in our docs as we don't have specific ECE docs to create custom repos only at deployment level) -> The ECH doc is valid for this, so ECE users can follow the ECH doc to create a custom repo at deployment level. - Added a lot of links to other relevant documents of the new IA. The intention of this document is to feel like a narrative document explaining what to do and how, but with links to the official snapshots or reindex documentation for more information. - For reindex method, I have added explanations and recommendations for mappings and index templates handling, without specific instructions (just pointers). - The previous doc presented the official doc about Backup Snapshot repositories in a context where it didn't belong. I've included a tip explaining a use case where that can be useful. - The previous doc suggested the usage of a `url` type repository in a way that felt incorrect (together with AWS / GCS / Azure keystore credentials which in my opinion cannot be related). I've added it as another possible use case (I'm still not 100% sure about this, and we have some contradictions in other documents that we have to address separately). Closes: #1391 --------- Co-authored-by: shainaraskas <[email protected]>
1 parent 2e578e9 commit 37e54c2

File tree

2 files changed

+110
-165
lines changed

2 files changed

+110
-165
lines changed

manage-data/migrate.md

Lines changed: 89 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -6,26 +6,30 @@ mapped_pages:
66
applies_to:
77
stack: ga
88
deployment:
9-
eck: unavailable
109
ess: ga
1110
ece: ga
12-
serverless: unavailable
1311
products:
1412
- id: cloud-hosted
1513
- id: cloud-enterprise
1614
---
1715

1816
# Migrate your {{es}} data
1917

20-
You might have switched to {{ech}} or {{ece}} for any number of reasons, and you’re likely wondering how to get your existing {{es}} data into your new infrastructure. Along with easily creating as many new deployments with {{es}} clusters that you need, you have several options for moving your data over. Choose the option that works best for you:
18+
You might have switched to {{ech}} (ECH) or {{ece}} (ECE) for any number of reasons, and you’re likely wondering how to get your existing {{es}} data into your new infrastructure. Along with easily creating as many new deployments with {{es}} clusters that you need, you have several options for moving your data over. Choose the option that works best for you:
2119

2220
* Index your data from the original source, which is the simplest method and provides the greatest flexibility for the {{es}} version and ingestion method.
2321
* Reindex from a remote cluster, which rebuilds the index from scratch.
2422
* Restore from a snapshot, which copies the existing indices.
2523

26-
### Before you begin [ec_migrate_before_you_begin]
24+
::::{note}
25+
Although this guide focuses on migrating data from a self-managed cluster to an {{ech}} or {{ece}} deployment, the steps can also be adapted for other scenarios, such as when the source cluster is managed by {{eck}}, or when migrating from {{ece}} to {{ech}}.
26+
27+
If both the source and destination clusters belong to the same {{ech}} or {{ece}} environment, refer to [](/deploy-manage/tools/snapshot-and-restore/ece-restore-across-clusters.md).
28+
::::
29+
30+
## Before you begin [ec_migrate_before_you_begin]
2731

28-
Depending on which option that you choose, you might have limitations or need to do some preparation beforehand.
32+
Depending on which option you choose, you might have limitations or need to do some preparation beforehand.
2933

3034
Indexing from the source
3135
: The new cluster must be the same size as your old one, or larger, to accommodate the data.
@@ -38,33 +42,37 @@ Reindex from a remote cluster
3842
Restore from a snapshot
3943
: The new cluster must be the same size as your old one, or larger, to accommodate the data. The new cluster must also be an Elasticsearch version that is compatible with the old cluster (check [Elasticsearch snapshot version compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility) for details). If you have not already done so, you will need to [set up snapshots for your old cluster](/deploy-manage/tools/snapshot-and-restore/self-managed.md) using a repository that can be accessed from the new cluster.
4044

41-
Migrating internal {{es}} indices
42-
: For {{ech}}, if you are migrating internal {{es}} indices from another cluster, specifically the `.kibana` index or the `.security` index, there are two options:
43-
44-
* Use the steps on this page to reindex the internal indices from a remote cluster. The steps for reindexing internal indices and regular, data indices are the same.
45-
* Check [Migrating internal indices](migrate/migrate-internal-indices.md) to restore the internal {{es}} indices from a snapshot.
46-
47-
::::{warning}
48-
Before you migrate your {{es}} data, [define your index mappings](/manage-data/data-store/mapping.md) on the new cluster. Index mappings are unable to migrate during reindex operations.
49-
::::
45+
:::{admonition} Migrating system {{es}} indices
46+
In {{es}} 8.0 and later versions, to back up and restore system indices and system data streams such as `.kibana` or `.security`, you must snapshot and restore the related feature's [feature state](/deploy-manage/tools/snapshot-and-restore.md#feature-state).
47+
48+
Refer to [Migrate system indices](./migrate/migrate-internal-indices.md) to learn how to restore the internal {{es}} system indices from a snapshot.
49+
:::
5050

51-
### Index from the source [ec-index-source]
51+
## Index from the source [ec-index-source]
5252

5353
If you still have access to the original data source, outside of your old {{es}} cluster, you can load the data from there. This might be the simplest option, allowing you to choose the {{es}} version and take advantage of the latest features. You have the option to use any ingestion method that you want—Logstash, Beats, the {{es}} clients, or whatever works best for you.
5454

5555
If the original source isn’t available or has other issues that make it non-viable, there are still two more migration options, getting the data from a remote cluster or restoring from a snapshot.
5656

57-
### Reindex from a remote cluster [ech-reindex-remote]
57+
## Reindex from a remote cluster [ech-reindex-remote]
5858

59-
Through the {{es}} reindex API, you can connect your new {{es}} Service deployment remotely to your old {{es}} cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run.
59+
Through the {{es}} [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex), you can connect your new {{es}} deployment remotely to your old {{es}} cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run than a [snapshot restore](#ec-restore-snapshots).
60+
61+
::::{warning}
62+
Reindex operations do not migrate index mappings, settings, or associated index templates from the source cluster.
63+
64+
Before migrating your {{es}} data, define the necessary [mappings](/manage-data/data-store/mapping.md) and [templates](/manage-data/data-store/templates.md) on the new cluster. The easiest way to do this is to copy the relevant index templates from the old cluster to the new one before starting reindex operations.
65+
::::
66+
67+
Follow these steps to reindex data remotely:
6068

6169
1. Log in to {{ech}} or {{ece}}.
6270
2. Select a deployment or create one.
63-
3. If the old {{es}} cluster is on a remote host (any type of host accessible over the internet), you need to make sure that the host can be accessed. Access is determined by the {{es}} `reindex.remote.whitelist` user setting.
71+
3. Ensure that the new {{es}} cluster can access the remote source cluster to perform the reindex operation. Access is controlled by the {{es}} `reindex.remote.whitelist` user setting.
6472

65-
Domains matching the pattern `["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`.
73+
Domains matching the patterns `["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`.
6674

67-
Otherwise, if your remote endpoint is not covered by the default settings, adjust the setting to add the remote {{es}} cluster as an allowed host:
75+
Otherwise, if your remote endpoint is not covered by the default patterns, adjust the setting to add the remote {{es}} cluster as an allowed host:
6876

6977
1. From your deployment menu, go to the **Edit** page.
7078
2. In the **Elasticsearch** section, select **Manage user settings and extensions**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for each node type instead.
@@ -78,8 +86,9 @@ Through the {{es}} reindex API, you can connect your new {{es}} Service deployme
7886

7987
4. Save your changes.
8088

81-
4. From the **API Console** or in the Kibana Console app, create the destination index.
82-
5. Copy the index from the remote cluster:
89+
4. Using the **API Console** or within {{kib}}, either create the destination index with the appropriate settings and [mappings](/manage-data/data-store/mapping.md), or ensure that the relevant [index templates](/manage-data/data-store/templates.md) are in place.
90+
91+
5. Using the **API Console** or [{{kib}} DevTools Console](/explore-analyze/query-filter/tools/console.md), reindex the data remotely from the old cluster:
8392

8493
```sh
8594
POST _reindex
@@ -101,92 +110,102 @@ Through the {{es}} reindex API, you can connect your new {{es}} Service deployme
101110
}
102111
```
103112

113+
For additional options and details, refer to the [reindex API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex).
114+
104115
6. Verify that the new index is present:
105116

106117
```sh
107118
GET INDEX-NAME/_search?pretty
108119
```
109120

110-
7. You can remove the reindex.remote.whitelist user setting that you added previously.
121+
7. If you are not planning to reindex more data from the remote, you can remove the `reindex.remote.whitelist` user setting that you added previously.
122+
123+
## Restore from a snapshot [ec-restore-snapshots]
111124

125+
Restoring from a snapshot is often the fastest and most reliable way to migrate data between {{es}} clusters. It preserves mappings, settings, and optionally parts of the cluster state such as index templates, component templates, and system indices.
112126

113-
### Restore from a snapshot [ec-restore-snapshots]
127+
System indices can be restored by including their corresponding [feature states](/deploy-manage/tools/snapshot-and-restore.md#feature-state) in the restore operation, allowing you to retain internal configurations related to security, {{kib}}, or other stack features.
114128

115-
If you cannot connect to a remote index for whatever reason, such as if it’s in a non-working state, you can try restoring from the most recent working snapshot.
129+
This method is especially useful when you want to fully replicate the source cluster or when remote reindexing is not possible, for example if the source cluster is in a degraded or unreachable state.
130+
131+
To use this method, the new cluster must have access to the snapshot repository that contains the data from the old cluster. Also ensure that both clusters use [compatible versions](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility).
132+
133+
For more information, refer to [Restore into a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster)
116134

117135
::::{note}
118136
For {{ece}} users, while it is most common to have Amazon S3 buckets, you should be able to restore from any addressable external storage that has your {{es}} snapshots.
119137
::::
120138

121-
1. On your old {{es}} cluster, choose an option to get the name of your snapshot repository bucket:
139+
The following steps assume you already have a snapshot repository configured in the old cluster, with at least one valid snapshot containing the data you want to migrate.
122140

123-
```sh
124-
GET /_snapshot
125-
GET /_snapshot/_all
126-
```
141+
### Step 1: Set up the repository in the new cluster [migrate-repo-setup]
127142

128-
2. Get the snapshot name:
143+
In this step, you’ll configure a snapshot repository in the new cluster that points to the storage location used by the old cluster. This allows the new cluster to access and restore snapshots created in the original environment.
129144

130-
```sh
131-
GET /_snapshot/NEW-REPOSITORY-NAME/_all
132-
```
145+
::::{tip}
146+
If your new {{ech}} or {{ece}} deployment cannot connect to the same repository used by your self-managed cluster, for example if it's a private NFS share, consider one of the following alternatives:
147+
148+
* [Back up your repository](/deploy-manage/tools/snapshot-and-restore/self-managed.md#snapshots-repository-backup) to a supported storage system such as AWS S3, Google Cloud Storage, or Azure Blob Storage, and then configure your new cluster to use that location for the data migration.
149+
* Expose the repository contents over `ftp`, `http`, or `https`, and use a [read-only URL repository](/deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md) type in your new deployment to access the snapshots.
150+
::::
133151
134-
The output for each entry provides a `"snapshot":` value which is the snapshot name.
152+
1. On your old {{es}} cluster, retrieve the snapshot repository configuration:
135153
136-
```json
137-
{
138-
"snapshots": [
139-
{
140-
"snapshot": "scheduled-1527616008-instance-0000000004",
141-
...
142-
},
143-
...
144-
]
145-
}
154+
```sh
155+
GET /_snapshot/_all
146156
```
147157
158+
Take note of the repository name and type (for example, `s3`, `gcs`, or `azure`), its base path, and any additional settings. Authentication credentials are often stored in the [secure settings](/deploy-manage/security/secure-settings.md) on each node. You’ll need to replicate all this configuration when registering the repository in the new ECH or ECE deployment.
148159
149-
3. Add the snapshot repository:
160+
If your old cluster has multiple repositories configured, identify the repository with the snapshots containing the data that you want to migrate.
150161
151-
::::{tab-set}
162+
2. Add the snapshot repository on the new cluster.
152163
153-
:::{tab-item} {{ech}}
164+
Considerations:
154165
155-
From the [console](https://cloud.elastic.co?page=docs&placement=docs-body) of the **new** {{es}} cluster, add the snapshot repository.
166+
* If you’re migrating [searchable snapshots](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters.
167+
* If the old cluster still has write access to the repository, register the repository as read-only to avoid data corruption. This can be done using the `readonly: true` option.
156168
157-
For details, check our guidelines for:
158-
* [Amazon Web Services (AWS) Storage](../deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md)
159-
* [Google Cloud Storage (GCS)](../deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md)
160-
* [Azure Blob Storage](../deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md).
169+
To connect the existing snapshot repository to your new deployment, follow the steps for the storage provider where the repository is hosted:
161170
162-
If you’re migrating [searchable snapshots](../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters.
171+
* **Amazon Web Services (AWS) Storage**
172+
* [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md#ec-snapshot-secrets-keystore)
173+
* [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md#ec-create-aws-repository)
174+
* **Google Cloud Storage (GCS)**
175+
* [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md#ec-configure-gcs-keystore)
176+
* [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md#ec-create-gcs-repository)
177+
* **Azure Blob Storage**
178+
* [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md#ec-configure-azure-keystore).
179+
* [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md#ec-create-azure-repository).
180+
181+
::::{important}
182+
Although these instructions are focused on {{ech}}, you should follow the same steps for {{ece}} by configuring the repository directly **at the deployment level**.
163183
164-
If the source cluster is still writing to the repository, you need to set the destination cluster’s repository connection to `readonly:true` to avoid data corruption. Refer to [backup a repository](../deploy-manage/tools/snapshot-and-restore/self-managed.md#snapshots-repository-backup) for details.
165-
:::
184+
**Do not** configure the repository as an [ECE-managed repository](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md), which is intended for automatic snapshots of deployments. In this case, you need to add a custom repository that already contains snapshots from another cluster.
185+
::::
166186
167-
:::{tab-item} {{ece}}
168187
169-
From the Cloud UI of the **new** {{es}} cluster add the snapshot repository.
188+
### Step 2: Run the snapshot restore [migrate-restore]
170189
171-
For details about configuring snapshot repositories on Amazon Web Services (AWS), Google Cloud Storage (GCS), or Azure Blob Storage, check [manage Snapshot Repositories](../deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md).
190+
After the repository has been registered and verified, you are ready to restore any data from any of its snapshots to your new cluster.
172191
173-
If you’re migrating [searchable snapshots](../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters.
174-
:::
192+
You can run a restore operation using the {{kib}} Management UI, or using the {{es}} API. Refer to [Restore a snapshot](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md) for more details, including API-based examples.
175193
176-
::::
194+
For details about the contents of a snapshot, refer to [](/deploy-manage/tools/snapshot-and-restore.md#snapshot-contents).
177195
178-
4. Start the Restore process.
196+
To start the restore process:
179197
180-
1. Open Kibana and go to **Management** > **Snapshot and Restore**.
181-
2. Under the **Snapshots** tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it.
182-
3. Select **Restore**.
183-
4. Select the indices you wish to restore.
184-
5. Configure any additional index settings.
185-
6. Select **Restore snapshot** to begin the process.
198+
1. Open Kibana and go to **Management** > **Snapshot and Restore**.
199+
2. Under the **Snapshots** tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it.
200+
3. Select **Restore**.
201+
4. Select the index or indices you wish to restore.
202+
5. Optionally, configure additional restore options, such as **Restore aliases**, **Restore global state**, or **Restore feature state**.
203+
6. Select **Restore snapshot** to begin the process.
186204
187-
5. Verify that the new index is restored in your deployment with this query:
205+
7. Verify that each restored index is available in your deployment. You can do this using {{kib}} **Index Management** UI, or by running this query:
188206
189207
```sh
190208
GET INDEX_NAME/_search?pretty
191209
```
192210
211+
If you have restored many indices you can also run `GET _cat/indices?s=index` to list all indices for verification.

0 commit comments

Comments
 (0)