You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Manage data -> Data migration to ECH or ECE updated (#1426)
When working on the issue
#1391 I realized that the
migration guide was a bit outdated and had multiple important issues to
address.
I have refined the content and applied major updates.
Updated docs:
- [Migrate your Elasticsearch
data](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/1426/manage-data/migrate)
- [Migrate system
indices](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/1426/manage-data/migrate/migrate-internal-indices)
Summary of fixes and changes:
- Internal / system indices migration based on snapshot was incorrect ->
feature states should be used.
- Internal / system indices migration based on reindexing was incorrect
-> the procedure was faulty and needed extra love and research. But
considering that based on Elasticsearch docs we state that from 8.0
onwards, `feature states` are the only way to backup and restore system
indices and system data streams, I have removed the reindex option for
this.
- Snapshot repository creation for ECE was invalid, as users shouldn't
create a repository for data migration at ECE-level, but directly at
deployment-level (this is a gap in our docs as we don't have specific
ECE docs to create custom repos only at deployment level) -> The ECH doc
is valid for this, so ECE users can follow the ECH doc to create a
custom repo at deployment level.
- Added a lot of links to other relevant documents of the new IA. The
intention of this document is to feel like a narrative document
explaining what to do and how, but with links to the official snapshots
or reindex documentation for more information.
- For reindex method, I have added explanations and recommendations for
mappings and index templates handling, without specific instructions
(just pointers).
- The previous doc presented the official doc about Backup Snapshot
repositories in a context where it didn't belong. I've included a tip
explaining a use case where that can be useful.
- The previous doc suggested the usage of a `url` type repository in a
way that felt incorrect (together with AWS / GCS / Azure keystore
credentials which in my opinion cannot be related). I've added it as
another possible use case (I'm still not 100% sure about this, and we
have some contradictions in other documents that we have to address
separately).
Closes: #1391
---------
Co-authored-by: shainaraskas <[email protected]>
Copy file name to clipboardExpand all lines: manage-data/migrate.md
+89-70Lines changed: 89 additions & 70 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,26 +6,30 @@ mapped_pages:
6
6
applies_to:
7
7
stack: ga
8
8
deployment:
9
-
eck: unavailable
10
9
ess: ga
11
10
ece: ga
12
-
serverless: unavailable
13
11
products:
14
12
- id: cloud-hosted
15
13
- id: cloud-enterprise
16
14
---
17
15
18
16
# Migrate your {{es}} data
19
17
20
-
You might have switched to {{ech}} or {{ece}} for any number of reasons, and you’re likely wondering how to get your existing {{es}} data into your new infrastructure. Along with easily creating as many new deployments with {{es}} clusters that you need, you have several options for moving your data over. Choose the option that works best for you:
18
+
You might have switched to {{ech}} (ECH) or {{ece}} (ECE) for any number of reasons, and you’re likely wondering how to get your existing {{es}} data into your new infrastructure. Along with easily creating as many new deployments with {{es}} clusters that you need, you have several options for moving your data over. Choose the option that works best for you:
21
19
22
20
* Index your data from the original source, which is the simplest method and provides the greatest flexibility for the {{es}} version and ingestion method.
23
21
* Reindex from a remote cluster, which rebuilds the index from scratch.
24
22
* Restore from a snapshot, which copies the existing indices.
25
23
26
-
### Before you begin [ec_migrate_before_you_begin]
24
+
::::{note}
25
+
Although this guide focuses on migrating data from a self-managed cluster to an {{ech}} or {{ece}} deployment, the steps can also be adapted for other scenarios, such as when the source cluster is managed by {{eck}}, or when migrating from {{ece}} to {{ech}}.
26
+
27
+
If both the source and destination clusters belong to the same {{ech}} or {{ece}} environment, refer to [](/deploy-manage/tools/snapshot-and-restore/ece-restore-across-clusters.md).
28
+
::::
29
+
30
+
## Before you begin [ec_migrate_before_you_begin]
27
31
28
-
Depending on which option that you choose, you might have limitations or need to do some preparation beforehand.
32
+
Depending on which option you choose, you might have limitations or need to do some preparation beforehand.
29
33
30
34
Indexing from the source
31
35
: The new cluster must be the same size as your old one, or larger, to accommodate the data.
@@ -38,33 +42,37 @@ Reindex from a remote cluster
38
42
Restore from a snapshot
39
43
: The new cluster must be the same size as your old one, or larger, to accommodate the data. The new cluster must also be an Elasticsearch version that is compatible with the old cluster (check [Elasticsearch snapshot version compatibility](/deploy-manage/tools/snapshot-and-restore.md#snapshot-restore-version-compatibility) for details). If you have not already done so, you will need to [set up snapshots for your old cluster](/deploy-manage/tools/snapshot-and-restore/self-managed.md) using a repository that can be accessed from the new cluster.
40
44
41
-
Migrating internal {{es}} indices
42
-
: For {{ech}}, if you are migrating internal {{es}} indices from another cluster, specifically the `.kibana` index or the `.security` index, there are two options:
43
-
44
-
* Use the steps on this page to reindex the internal indices from a remote cluster. The steps for reindexing internal indices and regular, data indices are the same.
45
-
* Check [Migrating internal indices](migrate/migrate-internal-indices.md) to restore the internal {{es}} indices from a snapshot.
46
-
47
-
::::{warning}
48
-
Before you migrate your {{es}} data, [define your index mappings](/manage-data/data-store/mapping.md) on the new cluster. Index mappings are unable to migrate during reindex operations.
49
-
::::
45
+
:::{admonition} Migrating system {{es}} indices
46
+
In {{es}} 8.0 and later versions, to back up and restore system indices and system data streams such as `.kibana` or `.security`, you must snapshot and restore the related feature's [feature state](/deploy-manage/tools/snapshot-and-restore.md#feature-state).
47
+
48
+
Refer to [Migrate system indices](./migrate/migrate-internal-indices.md) to learn how to restore the internal {{es}} system indices from a snapshot.
49
+
:::
50
50
51
-
###Index from the source [ec-index-source]
51
+
## Index from the source [ec-index-source]
52
52
53
53
If you still have access to the original data source, outside of your old {{es}} cluster, you can load the data from there. This might be the simplest option, allowing you to choose the {{es}} version and take advantage of the latest features. You have the option to use any ingestion method that you want—Logstash, Beats, the {{es}} clients, or whatever works best for you.
54
54
55
55
If the original source isn’t available or has other issues that make it non-viable, there are still two more migration options, getting the data from a remote cluster or restoring from a snapshot.
56
56
57
-
###Reindex from a remote cluster [ech-reindex-remote]
57
+
## Reindex from a remote cluster [ech-reindex-remote]
58
58
59
-
Through the {{es}} reindex API, you can connect your new {{es}} Service deployment remotely to your old {{es}} cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run.
59
+
Through the {{es}} [reindex API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex), you can connect your new {{es}} deployment remotely to your old {{es}} cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run than a [snapshot restore](#ec-restore-snapshots).
60
+
61
+
::::{warning}
62
+
Reindex operations do not migrate index mappings, settings, or associated index templates from the source cluster.
63
+
64
+
Before migrating your {{es}} data, define the necessary [mappings](/manage-data/data-store/mapping.md) and [templates](/manage-data/data-store/templates.md) on the new cluster. The easiest way to do this is to copy the relevant index templates from the old cluster to the new one before starting reindex operations.
65
+
::::
66
+
67
+
Follow these steps to reindex data remotely:
60
68
61
69
1. Log in to {{ech}} or {{ece}}.
62
70
2. Select a deployment or create one.
63
-
3.If the old {{es}} cluster is on a remote host (any type of host accessible over the internet), you need to make sure that the host can be accessed. Access is determined by the {{es}} `reindex.remote.whitelist` user setting.
71
+
3.Ensure that the new {{es}} cluster can access the remote source cluster to perform the reindex operation. Access is controlled by the {{es}} `reindex.remote.whitelist` user setting.
64
72
65
-
Domains matching the pattern`["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`.
73
+
Domains matching the patterns`["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`.
66
74
67
-
Otherwise, if your remote endpoint is not covered by the default settings, adjust the setting to add the remote {{es}} cluster as an allowed host:
75
+
Otherwise, if your remote endpoint is not covered by the default patterns, adjust the setting to add the remote {{es}} cluster as an allowed host:
68
76
69
77
1. From your deployment menu, go to the **Edit** page.
70
78
2. In the **Elasticsearch** section, select **Manage user settings and extensions**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for each node type instead.
@@ -78,8 +86,9 @@ Through the {{es}} reindex API, you can connect your new {{es}} Service deployme
78
86
79
87
4. Save your changes.
80
88
81
-
4. From the **API Console** or in the Kibana Console app, create the destination index.
82
-
5. Copy the index from the remote cluster:
89
+
4. Using the **API Console** or within {{kib}}, either create the destination index with the appropriate settings and [mappings](/manage-data/data-store/mapping.md), or ensure that the relevant [index templates](/manage-data/data-store/templates.md) are in place.
90
+
91
+
5. Using the **API Console** or [{{kib}} DevTools Console](/explore-analyze/query-filter/tools/console.md), reindex the data remotely from the old cluster:
83
92
84
93
```sh
85
94
POST _reindex
@@ -101,92 +110,102 @@ Through the {{es}} reindex API, you can connect your new {{es}} Service deployme
101
110
}
102
111
```
103
112
113
+
For additional options and details, refer to the [reindex API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex).
114
+
104
115
6. Verify that the new index is present:
105
116
106
117
```sh
107
118
GET INDEX-NAME/_search?pretty
108
119
```
109
120
110
-
7. You can remove the reindex.remote.whitelist user setting that you added previously.
121
+
7. If you are not planning to reindex more data from the remote, you can remove the `reindex.remote.whitelist` user setting that you added previously.
122
+
123
+
## Restore from a snapshot [ec-restore-snapshots]
111
124
125
+
Restoring from a snapshot is often the fastest and most reliable way to migrate data between {{es}} clusters. It preserves mappings, settings, and optionally parts of the cluster state such as index templates, component templates, and system indices.
112
126
113
-
### Restore from a snapshot [ec-restore-snapshots]
127
+
System indices can be restored by including their corresponding [feature states](/deploy-manage/tools/snapshot-and-restore.md#feature-state) in the restore operation, allowing you to retain internal configurations related to security, {{kib}}, or other stack features.
114
128
115
-
If you cannot connect to a remote index forwhatever reason, such as if it’sin a non-working state, you can try restoring from the most recent working snapshot.
129
+
This method is especially useful when you want to fully replicate the source cluster or when remote reindexing is not possible, forexample if the source cluster isin a degraded or unreachable state.
130
+
131
+
To use this method, the new cluster must have access to the snapshot repository that contains the data from the old cluster. Also ensure that both clusters use [compatible versions](/deploy-manage/tools/snapshot-and-restore.md#snapshot-compatibility).
132
+
133
+
For more information, refer to [Restore into a different cluster](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md#restore-different-cluster)
116
134
117
135
::::{note}
118
136
For {{ece}} users, while it is most common to have Amazon S3 buckets, you should be able to restore from any addressable external storage that has your {{es}} snapshots.
119
137
::::
120
138
121
-
1. On your old {{es}} cluster, choose an option to get the name of your snapshot repository bucket:
139
+
The following steps assume you already have a snapshot repository configured inthe old cluster, with at least one valid snapshot containing the data you want to migrate.
122
140
123
-
```sh
124
-
GET /_snapshot
125
-
GET /_snapshot/_all
126
-
```
141
+
### Step 1: Set up the repository in the new cluster [migrate-repo-setup]
127
142
128
-
2. Get the snapshot name:
143
+
In this step, you’ll configure a snapshot repository in the new cluster that points to the storage location used by the old cluster. This allows the new cluster to access and restore snapshots created in the original environment.
129
144
130
-
```sh
131
-
GET /_snapshot/NEW-REPOSITORY-NAME/_all
132
-
```
145
+
::::{tip}
146
+
If your new {{ech}} or {{ece}} deployment cannot connect to the same repository used by your self-managed cluster, for example if it's a private NFS share, consider one of the following alternatives:
147
+
148
+
* [Back up your repository](/deploy-manage/tools/snapshot-and-restore/self-managed.md#snapshots-repository-backup) to a supported storage system such as AWS S3, Google Cloud Storage, or Azure Blob Storage, and then configure your new cluster to use that location for the data migration.
149
+
* Expose the repository contents over `ftp`, `http`, or `https`, and use a [read-only URL repository](/deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md) type in your new deployment to access the snapshots.
150
+
::::
133
151
134
-
The output for each entry provides a `"snapshot":` value which is the snapshot name.
152
+
1. On your old {{es}} cluster, retrieve the snapshot repository configuration:
Take note of the repository name and type (for example, `s3`, `gcs`, or `azure`), its base path, and any additional settings. Authentication credentials are often stored in the [secure settings](/deploy-manage/security/secure-settings.md) on each node. You’ll need to replicate all this configuration when registering the repository in the new ECH or ECE deployment.
148
159
149
-
3. Add the snapshot repository:
160
+
If your old cluster has multiple repositories configured, identify the repository with the snapshots containing the data that you want to migrate.
150
161
151
-
::::{tab-set}
162
+
2. Add the snapshot repository on the new cluster.
152
163
153
-
:::{tab-item} {{ech}}
164
+
Considerations:
154
165
155
-
From the [console](https://cloud.elastic.co?page=docs&placement=docs-body) of the **new** {{es}} cluster, add the snapshot repository.
166
+
* If you’re migrating [searchable snapshots](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters.
167
+
* If the old cluster still has write access to the repository, register the repository as read-only to avoid data corruption. This can be done using the `readonly: true` option.
156
168
157
-
For details, check our guidelines for:
158
-
* [Amazon Web Services (AWS) Storage](../deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md)
To connect the existing snapshot repository to your new deployment, follow the steps for the storage provider where the repository is hosted:
161
170
162
-
If you’re migrating [searchable snapshots](../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters.
171
+
* **Amazon Web Services (AWS) Storage**
172
+
* [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md#ec-snapshot-secrets-keystore)
173
+
* [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md#ec-create-aws-repository)
174
+
* **Google Cloud Storage (GCS)**
175
+
* [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md#ec-configure-gcs-keystore)
176
+
* [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md#ec-create-gcs-repository)
177
+
* **Azure Blob Storage**
178
+
* [Store credentials in the keystore](/deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md#ec-configure-azure-keystore).
179
+
* [Create the repository](/deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md#ec-create-azure-repository).
180
+
181
+
::::{important}
182
+
Although these instructions are focused on {{ech}}, you should follow the same steps for {{ece}} by configuring the repository directly **at the deployment level**.
163
183
164
-
If the source cluster is still writing to the repository, you need to set the destination cluster’s repository connection to `readonly:true` to avoid data corruption. Refer to [backup a repository](../deploy-manage/tools/snapshot-and-restore/self-managed.md#snapshots-repository-backup) fordetails.
165
-
:::
184
+
**Do not** configure the repository as an [ECE-managed repository](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md), which is intended for automatic snapshots of deployments. In this case, you need to add a custom repository that already contains snapshots from another cluster.
185
+
::::
166
186
167
-
:::{tab-item} {{ece}}
168
187
169
-
From the Cloud UI of the **new** {{es}} cluster add the snapshot repository.
188
+
### Step 2: Run the snapshot restore [migrate-restore]
170
189
171
-
For details about configuring snapshot repositories on Amazon Web Services (AWS), Google Cloud Storage (GCS), or Azure Blob Storage, check [manage Snapshot Repositories](../deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md).
190
+
After the repository has been registered and verified, you are ready to restore any data from any of its snapshots to your new cluster.
172
191
173
-
If you’re migrating [searchable snapshots](../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters.
174
-
:::
192
+
You can run a restore operation using the {{kib}} Management UI, or using the {{es}} API. Refer to [Restore a snapshot](/deploy-manage/tools/snapshot-and-restore/restore-snapshot.md) for more details, including API-based examples.
175
193
176
-
::::
194
+
For details about the contents of a snapshot, refer to [](/deploy-manage/tools/snapshot-and-restore.md#snapshot-contents).
177
195
178
-
4. Start the Restore process.
196
+
To start the restore process:
179
197
180
-
1. Open Kibana and go to **Management**>**Snapshot and Restore**.
181
-
2. Under the **Snapshots** tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it.
182
-
3. Select **Restore**.
183
-
4. Select the indices you wish to restore.
184
-
5. Configure any additional index settings.
185
-
6. Select **Restore snapshot** to begin the process.
198
+
1. Open Kibana and go to **Management** > **Snapshot and Restore**.
199
+
2. Under the **Snapshots** tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it.
200
+
3. Select **Restore**.
201
+
4. Select the index or indices you wish to restore.
202
+
5. Optionally, configure additional restore options, such as **Restore aliases**, **Restore global state**, or **Restore feature state**.
203
+
6. Select **Restore snapshot** to begin the process.
186
204
187
-
5. Verify that the new index is restoredin your deployment with this query:
205
+
7. Verify that each restored index is available in your deployment. You can do this using {{kib}} **Index Management** UI, or by running this query:
188
206
189
207
```sh
190
208
GET INDEX_NAME/_search?pretty
191
209
```
192
210
211
+
If you have restored many indices you can also run `GET _cat/indices?s=index` to list all indices for verification.
0 commit comments