From 3a51f84efd51d86a05c1ed8d57327bc1fa190acc Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 12 Feb 2025 14:36:55 -0500 Subject: [PATCH 1/4] Add 'Migrating data' and children --- manage-data/migrate.md | 182 ++++++++++++++++-- ...signed-certificate-using-remote-reindex.md | 4 + .../migrate/migrate-internal-indices.md | 120 +++++++++++- .../cloud-enterprise/ece-migrating-data.md | 156 --------------- .../cloud-heroku/ech-migrate-data-internal.md | 92 --------- .../cloud/cloud-heroku/ech-migrate-data2.md | 159 --------------- .../cloud/cloud/ec-migrate-data-internal.md | 92 --------- .../cloud/cloud/ec-migrating-data.md | 174 ----------------- raw-migrated-files/toc.yml | 5 - 9 files changed, 284 insertions(+), 700 deletions(-) delete mode 100644 raw-migrated-files/cloud/cloud-enterprise/ece-migrating-data.md delete mode 100644 raw-migrated-files/cloud/cloud-heroku/ech-migrate-data-internal.md delete mode 100644 raw-migrated-files/cloud/cloud-heroku/ech-migrate-data2.md delete mode 100644 raw-migrated-files/cloud/cloud/ec-migrate-data-internal.md delete mode 100644 raw-migrated-files/cloud/cloud/ec-migrating-data.md diff --git a/manage-data/migrate.md b/manage-data/migrate.md index e9b2ae1bd5..d776720ae5 100644 --- a/manage-data/migrate.md +++ b/manage-data/migrate.md @@ -3,38 +3,184 @@ mapped_urls: - https://www.elastic.co/guide/en/cloud/current/ec-migrating-data.html - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrating-data.html - https://www.elastic.co/guide/en/cloud-heroku/current/ech-migrate-data2.html +applies: + serverless: unavailable + hosted: all + ece: all --- -# Migrate data between Elasticsearch clusters +# Migrate your {{es}} data -% What needs to be done: Refine +You might have switched to {{ech}}, {{ece}}, or Elasticsearch Add-On for Heroku for any number of reasons, and you’re likely wondering how to get your existing {{es}} data into your new infrastructure. Along with easily creating as many new deployments with {{es}} clusters that you need, you have several options for moving your data over. Choose the option that works best for you: -% GitHub issue: docs-projects#331 +* Index your data from the original source, which is the simplest method and provides the greatest flexibility for the {{es}} version and ingestion method. +* Reindex from a remote cluster, which rebuilds the index from scratch. +* Restore from a snapshot, which copies the existing indices. -% Scope notes: We can combine all three into one page, using tabbed widgets probably for the ECH/ECE/ESS differences. +### Before you begin [ec_migrate_before_you_begin] -% Use migrated content from existing pages that map to this page: +Depending on which option that you choose, you might have limitations or need to do some preparation beforehand. -% - [ ] ./raw-migrated-files/cloud/cloud/ec-migrating-data.md -% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-migrating-data.md -% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-migrate-data2.md +Indexing from the source +: The new cluster must be the same size as your old one, or larger, to accommodate the data. -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): +Reindex from a remote cluster +: The new cluster must be the same size as your old one, or larger, to accommodate the data. Depending on your security settings for your old cluster, you might need to temporarily allow TCP traffic on port 9243 for this procedure. -$$$ech-reindex-remote$$$ + For {{ech}}, if your cluster is self-managed with a self-signed certificate, you can follow this [step-by-step migration guide](migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md). -$$$ec-restore-snapshots$$$ +Restore from a snapshot +: The new cluster must be the same size as your old one, or larger, to accommodate the data. The new cluster must also be an Elasticsearch version that is compatible with the old cluster (check [Elasticsearch snapshot version compatibility](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html#snapshot-restore-version-compatibility) for details). If you have not already done so, you will need to [set up snapshots for your old cluster](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html) using a repository that can be accessed from the new cluster. -$$$ec-index-source$$$ +Migrating internal {{es}} indices +: For {{ech}} and Elasticsearch Add-On for Heroku, if you are migrating internal {{es}} indices from another cluster, specifically the `.kibana` index or the `.security` index, there are two options: -$$$ec-migrate-data2$$$ + * Use the steps on this page to reindex the internal indices from a remote cluster. The steps for reindexing internal indices and regular, data indices are the same. + * Check [Migrating internal indices](../../../manage-data/migrate/migrate-internal-indices.md) to restore the internal {{es}} indices from a snapshot. -$$$ec-reindex-remote$$$ +::::{warning} +Before you migrate your {{es}} data, [define your index mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) on the new cluster. Index mappings are unable to migrate during reindex operations. +:::: -$$$ece-index-source$$$ +### Index from the source [ec-index-source] -$$$ece-migrate-data2$$$ +If you still have access to the original data source, outside of your old {{es}} cluster, you can load the data from there. This might be the simplest option, allowing you to choose the {{es}} version and take advantage of the latest features. You have the option to use any ingestion method that you want—​Logstash, Beats, the {{es}} clients, or whatever works best for you. -$$$ece-reindex-remote$$$ +If the original source isn’t available or has other issues that make it non-viable, there are still two more migration options, getting the data from a remote cluster or restoring from a snapshot. + +### Reindex from a remote cluster [ec-reindex-remote] + +Through the {{es}} reindex API, you can connect your new {{es}} Service deployment remotely to your old {{es}} cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run. + +1. Log in to {{ech}}, {{ece}}, or Elasticsearch Add-On for Heroku. +2. Select a deployment or create one. +3. If the old {{es}} cluster is on a remote host (any type of host accessible over the internet), you need to make sure that the host can be accessed. Access is determined by the {{es}} `reindex.remote.whitelist` user setting. + + Domains matching the pattern `["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`. + + Otherwise, if your remote endpoint is not covered by the default settings, adjust the setting to add the remote {{es}} cluster as an allowed host: + + 1. From your deployment menu, go to the **Edit** page. + 2. In the **Elasticsearch** section, select **Manage user settings and extensions**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for each node type instead. + 3. Add the following `reindex.remote.whitelist: [REMOTE_HOST:PORT]` user setting, where `REMOTE_HOST` is a pattern matching the URL for the remote {{es}} host that you are reindexing from, and PORT is the host port number. Do not include the `https://` prefix. + + Note that if you override the parameter it replaces the defaults: `["*.io:*", "*.com:*"]`. If you still want these patterns to be allowed you need to specify them explicitly in the value. + + For example: + + `reindex.remote.whitelist: ["*.us-east-1.aws.found.io:9243", "*.com:*"]` + + 4. Save your changes. + +4. From the **API Console** or in the Kibana Console app, create the destination index. +5. Copy the index from the remote cluster: + + ```sh + POST _reindex + { + "source": { + "remote": { + "host": "https://REMOTE_ELASTICSEARCH_ENDPOINT:PORT", + "username": "USER", + "password": "PASSWORD" + }, + "index": "INDEX_NAME", + "query": { + "match_all": {} + } + }, + "dest": { + "index": "INDEX_NAME" + } + } + ``` + +6. Verify that the new index is present: + + ```sh + GET INDEX-NAME/_search?pretty + ``` + +7. You can remove the reindex.remote.whitelist user setting that you added previously. + + +### Restore from a snapshot [ec-restore-snapshots] + +If you cannot connect to a remote index for whatever reason, such as if it’s in a non-working state, you can try restoring from the most recent working snapshot. + +::::{note} +For {{ece}} users, while it is most common to have Amazon S3 buckets, you should be able to restore from any addressable external storage that has your {{es}} snapshots. +:::: + +1. On your old {{es}} cluster, choose an option to get the name of your snapshot repository bucket: + + ```sh + GET /_snapshot + GET /_snapshot/_all + ``` + +2. Get the snapshot name: + + ```sh + GET /_snapshot/NEW-REPOSITORY-NAME/_all + ``` + + The output for each entry provides a `"snapshot":` value which is the snapshot name. + + ```json + { + "snapshots": [ + { + "snapshot": "scheduled-1527616008-instance-0000000004", + ... + }, + ... + ] + } + ``` + + +3. Add the snapshot repository: + + ::::{tab-set} + + :::{tab-item} {{ech}} and Elasticsearch Add-On for Heroku + + From the [console](https://cloud.elastic.co?page=docs&placement=docs-body) of the **new** {{es}} cluster, add the snapshot repository. + + For details, check our guidelines for: + * [Amazon Web Services (AWS) Storage](../deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md) + * [Google Cloud Storage (GCS)](../deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md) + * [Azure Blob Storage](../deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md). + + If you’re migrating [searchable snapshots](../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters. + + If the source cluster is still writing to the repository, you need to set the destination cluster’s repository connection to `readonly:true` to avoid data corruption. Refer to [backup a repository](../deploy-manage/tools/snapshot-and-restore/self-managed.md#snapshots-repository-backup) for details. + ::: + + :::{tab-item} {{ece}} + + From the Cloud UI of the **new** {{es}} cluster add the snapshot repository. + + For details about configuring snapshot repositories on Amazon Web Services (AWS), Google Cloud Storage (GCS), or Azure Blob Storage, check [manage Snapshot Repositories](../deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md). + + If you’re migrating [searchable snapshots](../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters. + ::: + + :::: + +4. Start the Restore process. + + 1. Open Kibana and go to **Management** > **Snapshot and Restore**. + 2. Under the **Snapshots** tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it. + 3. Select **Restore**. + 4. Select the indices you wish to restore. + 5. Configure any additional index settings. + 6. Select **Restore snapshot** to begin the process. + +5. Verify that the new index is restored in your deployment with this query: + + ```sh + GET INDEX_NAME/_search?pretty + ``` -$$$ece-restore-snapshots$$$ \ No newline at end of file diff --git a/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md b/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md index d211f27ee4..63808232be 100644 --- a/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md +++ b/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md @@ -1,6 +1,10 @@ --- mapped_pages: - https://www.elastic.co/guide/en/cloud/current/ec-remote-reindex.html +applies: + serverless: unavailable + hosted: all + ece: unavailable --- # Migrate from a self-managed cluster with a self-signed certificate using remote reindex [ec-remote-reindex] diff --git a/manage-data/migrate/migrate-internal-indices.md b/manage-data/migrate/migrate-internal-indices.md index ef1867b897..4c9c09f380 100644 --- a/manage-data/migrate/migrate-internal-indices.md +++ b/manage-data/migrate/migrate-internal-indices.md @@ -2,13 +2,125 @@ mapped_urls: - https://www.elastic.co/guide/en/cloud/current/ec-migrate-data-internal.html - https://www.elastic.co/guide/en/cloud-heroku/current/ech-migrate-data-internal.html +applies: + serverless: unavailable + hosted: all + ece: unavailable --- # Migrate internal indices -% What needs to be done: Lift-and-shift +When you migrate your {{es}} data into a new infrastructure you may also want to migrate your {{es}} internal indices, specifically the `.kibana` index and the `.security` index. -% Use migrated content from existing pages that map to this page: +There are two ways to migrate the internal {{es}} indices: -% - [ ] ./raw-migrated-files/cloud/cloud/ec-migrate-data-internal.md -% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-migrate-data-internal.md \ No newline at end of file +1. Reindex the indices from a remote cluster. +2. Restore the indices from a snapshot. + +To reindex internal indices from a remote cluster, you can follow the same steps that you use to reindex regular indices when you [migrate your {{es}} data indices](../migrate.md#ec-reindex-remote). + +To restore internal indices from a snapshot, the procedure is a bit different from migrating {{es}} data indices. Use these steps to restore internal indices from a snapshot: + +1. On your old {{es}} cluster, choose an option to get the name of your snapshot repository bucket: + + ```sh + GET /_snapshot + GET /_snapshot/_all + ``` + +2. Get the snapshot name: + + ```sh + GET /_snapshot/NEW-REPOSITORY-NAME/_all + ``` + + The output for each entry provides a `"snapshot":` value which is the snapshot name. + + ``` + { + "snapshots": [ + { + "snapshot": "scheduled-1527616008-instance-0000000004", + ``` + + + +3. To restore internal {{es}} indices, you need to register the snapshot repository in `read-only` mode. + + ::::{tab-set} + + :::{tab-item} {{ech}} + First, add the authentication information for the repository to the {{ech}} keystore, following the steps for your cloud provider: + * [AWS S3](../../deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md#ec-snapshot-secrets-keystore) + * [Google Cloud Storage](../../deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md#ec-configure-gcs-keystore) + * [Azure Blog storage](../../deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md#ec-configure-azure-keystore) + + Next, register a read-only repository. Open an {{es}} [API console](../../explore-analyze/query-filter/tools/console.md) and run the [Read-only URL repository](../../deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md) API call. + ::: + + :::{tab-item} Elasticsearch Add-On for Heroku + First, add the authentication information for the repository to the Elasticsearch Add-On for Heroku keystore, following the steps for your cloud provider: + * [AWS S3](../../deploy-manage/tools/snapshot-and-restore/ech-aws-custom-repository.md#ech-snapshot-secrets-keystore) + * [Google Cloud Storage](../../deploy-manage/tools/snapshot-and-restore/ech-gcs-snapshotting.md#ech-configure-gcs-keystore) + * [Azure Blog storage](../../deploy-manage/tools/snapshot-and-restore/ech-azure-snapshotting.md#ech-configure-azure-keystore) + + Next, register a read-only repository. Open an {{es}} [API console](../../explore-analyze/query-filter/tools/console.md) and run the [Read-only URL repository](../../deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md) API call. + + ::: + + :::: + +4. Once the repository has been registered and verified, you are ready to restore the internal indices to your new cluster, either all at once or individually. + + * **Restore all internal indices** + + Run the following API call to restore all internal indices from a snapshot to the cluster: + + ```sh + POST /_snapshot/repo/snapshot/_restore + { + "indices": ".*", + "ignore_unavailable": true, + "include_global_state": false, + "include_aliases": false, + "rename_pattern": ".(.+)", + "rename_replacement": "restored_security_$1" + } + ``` + + * **Restore an individual internal index** + + ::::{warning} + When restoring internal indices, ensure that the `include_aliases` parameter is set to `false`. Not doing so will make Kibana inaccessible. If you do run the restore without `include_aliases`, the restored index can be deleted or the alias reference to it can be removed. This will have to be done from either the API console or a curl command as Kibana will not be accessible. + :::: + + Run the following API call to restore one internal index from a snapshot to the cluster: + + ```sh + POST /_snapshot/repo/snapshot/_restore + { + "indices": ".kibana", + "ignore_unavailable": true, + "include_global_state": false, + "include_aliases": false, + "rename_pattern": ".(.+)", + "rename_replacement": "restored_security_$1" + } + ``` + + Next, the restored index needs to be reindexed into the internal index, as shown: + + ```sh + POST _reindex + { + "source": { + "index": "restored_kibana" + }, + "dest": { + "index": ".kibana" + } + } + ``` + + +Your internal {{es}} index or indices should now be available in your new {{es}} cluster. Once verified, the `restored_*` indices are safe to delete. diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-migrating-data.md b/raw-migrated-files/cloud/cloud-enterprise/ece-migrating-data.md deleted file mode 100644 index 7417fab653..0000000000 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-migrating-data.md +++ /dev/null @@ -1,156 +0,0 @@ -# Migrating data [ece-migrating-data] - -If you want to move your existing Elasticsearch data into your new infrastructure, check out the [migration options](../../../manage-data/migrate.md#ece-migrate-data2). You’ll find instructions to guide you through: - -* Migrating data [from its original source](../../../manage-data/migrate.md#ece-index-source) -* Reindexing data [from a remote Elasticsearch cluster](../../../manage-data/migrate.md#ece-reindex-remote) -* Restoring data [into a new deployment](../../../deploy-manage/tools/snapshot-and-restore/ece-restore-snapshots-into-new-deployment.md) -* Restoring data [from a snapshot](../../../manage-data/migrate.md#ece-restore-snapshots) - - -## Migrate your Elasticsearch data [ece-migrate-data2] - -You might have switched to Elastic Cloud Enterprise for any number of reasons and you’re likely wondering how to get your existing Elasticsearch data into your new infrastructure. Along with easily creating as many new deployments with Elasticsearch clusters that you need, you have several options for moving your data over. Choose the option that works best for you: - -* Index your data from the original source, which is the simplest method and provides the greatest flexibility for the Elasticsearch version and ingestion method. -* Reindex from a remote cluster, which rebuilds the index from scratch. -* Restore from a snapshot, which copies the existing indices. - -One of the many advantages of Elastic Cloud Enterprise is that you can spin up a deployment quickly, try out something, and then delete it if you don’t like it. This flexibility provides the freedom to experiment while your existing production cluster continues to work. - - -### Before you begin [ece_before_you_begin_11] - -Depending on which option that you choose, you might have limitations or need to do some preparation beforehand. - -Indexing from the source -: The new cluster must be the same size as your old one, or larger, to accommodate the data. - -Reindex from a remote cluster -: The new cluster must be the same size as your old one, or larger, to accommodate the data. Depending on your security settings for your old cluster, you might need to temporarily allow TCP traffic on port 9243 for this procedure. - -Restore from a snapshot -: The new cluster must be the same size as your old one, or larger, to accommodate the data. The new cluster must also be an Elasticsearch version that is compatible with the old cluster (check [Elasticsearch snapshot version compatibility](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html#snapshot-restore-version-compatibility) for details). If you have not already done so, you will need to [set up snapshots for your old cluster](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html) using a repository that can be accessed from the new cluster. - -::::{warning} -Before you migrate your Elasticsearch data, [define your index mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) on the new cluster. Index mappings are unable to migrate during reindex operations. -:::: - - - -### Index from the source [ece-index-source] - -If you still have access to the original data source, outside of your old Elasticsearch cluster, you can load the data from there. This might be the simplest option, allowing you to choose the Elasticsearch version and take advantage of the latest features. You have the option to use any ingestion method that you want—​Logstash, Beats, the Elasticsearch clients, or whatever works best for you. - -If the original source isn’t available or has other issues that make it non-viable, there are still two more migration options, getting the data from a remote cluster or restoring from a snapshot. - - -### Reindex from a remote cluster [ece-reindex-remote] - -Through the Elasticsearch reindex API, you can connect your new Elastic Cloud Enterprise deployment remotely to your old Elasticsearch cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run. - -1. [Log into the Cloud UI](../../../deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). -2. Select a deployment or create one. -3. If the old Elasticsearch cluster is on a remote host (any type of host accessible over the internet), you need to make sure that the host can be accessed. Access is determined by the Elasticsearch `reindex.remote.whitelist` user setting. - - Domains matching the pattern `["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`. - - Otherwise, if your remote endpoint is not covered by the default settings, adjust the setting to add the remote Elasticsearch cluster as an allowed host: - - 1. From your deployment menu, go to the **Edit** page. - 2. In the **Elasticsearch** section, select **Manage user settings and extensions**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for each node type instead. - 3. Add the following `reindex.remote.whitelist: [REMOTE_HOST:PORT]` user setting, where `REMOTE_HOST` is a pattern matching the URL for the remote Elasticsearch host that you are reindexing from, and PORT is the host port number. Do not include the `https://` prefix. - - Note that if you override the parameter it replaces the defaults: `["*.io:*", "*.com:*"]`. If you still want these patterns to be allowed you need to specify them explicitly in the value. - - For example: - - `reindex.remote.whitelist: ["*.us-east-1.aws.found.io:9243", "*.com:*"]` - - 4. Save your changes. - -4. From the **API Console** or in the Kibana Console app, create the destination index on Elastic Cloud Enterprise. -5. Copy the index from the remote cluster: - - ```sh - POST _reindex - { - "source": { - "remote": { - "host": "https://REMOTE_ELASTICSEARCH_ENDPOINT:PORT", - "username": "USER", - "password": "PASSWORD" - }, - "index": "INDEX_NAME", - "query": { - "match_all": {} - } - }, - "dest": { - "index": "INDEX_NAME" - } - } - ``` - -6. Verify that the new index is present: - - ```sh - GET INDEX-NAME/_search?pretty - ``` - -7. You can remove the reindex.remote.whitelist user setting that you added previously. - - -### Restore from a snapshot [ece-restore-snapshots] - -If you cannot connect to a remote index for whatever reason, such as if it’s in a non-working state, you can try restoring from the most recent working snapshot. - -1. On your old Elasticsearch cluster, choose an option to get the name of your snapshot repository bucket: - - ```sh - GET /_snapshot - GET /_snapshot/_all - ``` - -2. Get the snapshot name: - - ```sh - GET /_snapshot/NEW-REPOSITORY-NAME/_all - ``` - - The output for each entry provides a `"snapshot":` value which is the snapshot name. - - ```json - { - "snapshots": [ - { - "snapshot": "scheduled-1527616008-instance-0000000004", - ... - }, - ... - ] - } - ``` - -3. From the Cloud UI of the **new** Elasticsearch cluster add the snapshot repository. For details about configuring snapshot repositories on Amazon Web Services (AWS), Google Cloud Storage (GCS), or Azure Blob Storage, check [manage Snapshot Repositories](../../../deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md). - - ::::{important} - If you’re migrating [searchable snapshots](../../../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters. - :::: - -4. Start the Restore process. - - 1. Open Kibana and go to **Management** > **Snapshot and Restore**. - 2. Under the **Snapshots** tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it. - 3. Select **Restore**. - 4. Select the indices you wish to restore. - 5. Configure any additional index settings. - 6. Select **Restore snapshot** to begin the process. - -5. Verify that the new index is restored in your Elastic Cloud Enterprise deployment with this query: - - ```sh - GET INDEX_NAME/_search?pretty - ``` - - diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-migrate-data-internal.md b/raw-migrated-files/cloud/cloud-heroku/ech-migrate-data-internal.md deleted file mode 100644 index b2142a3cde..0000000000 --- a/raw-migrated-files/cloud/cloud-heroku/ech-migrate-data-internal.md +++ /dev/null @@ -1,92 +0,0 @@ -# Migrate internal indices [ech-migrate-data-internal] - -When you migrate your Elasticsearch data into a new infrastructure you may also want to migrate your Elasticsearch internal indices, specifically the `.kibana` index and the `.security` index. - -There are two ways to migrate the internal Elasticsearch indices: - -1. Reindex the indices from a remote cluster. -2. Restore the indices from a snapshot. - -To reindex internal indices from a remote cluster, you can follow the same steps that you use to reindex regular indices when you [migrate your Elasticsearch data indices](../../../manage-data/migrate.md#ech-reindex-remote). - -To restore internal indices from a snapshot, the procedure is a bit different from migrating Elasticsearch data indices. Use these steps to restore internal indices from a snapshot: - -1. On your old Elasticsearch cluster, choose an option to get the name of your snapshot repository bucket: - - ```sh - GET /_snapshot - GET /_snapshot/_all - ``` - -2. Get the snapshot name: - - ```sh - GET /_snapshot/NEW-REPOSITORY-NAME/_all - ``` - - The output for each entry provides a `"snapshot":` value which is the snapshot name. - - ``` - { - "snapshots": [ - { - "snapshot": "scheduled-1527616008-instance-0000000004", - ``` - -3. To restore internal Elasticsearch indices, you need to register the snapshot repository in `read-only` mode. To do so, first add the authentication information for the repository to the Elasticsearch Add-On for Heroku keystore, following the steps for [AWS S3](../../../deploy-manage/tools/snapshot-and-restore/ech-aws-custom-repository.md#ech-snapshot-secrets-keystore), [Google Cloud Storage](../../../deploy-manage/tools/snapshot-and-restore/ech-gcs-snapshotting.md#ech-configure-gcs-keystore), or [Azure Blog storage](../../../deploy-manage/tools/snapshot-and-restore/ech-azure-snapshotting.md#ech-configure-azure-keystore). -4. To register a read-only repository, open the Elasticsearch [API console](../../../deploy-manage/deploy/elastic-cloud/ech-api-console.md) or the Kibana [Dev Tools page](../../../explore-analyze/query-filter/tools.md) and run the [Read-only URL repository](../../../deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md) API call. -5. Once the repository has been registered and verified, you are ready to restore the internal indices to your new cluster, either all at once or individually. - - * **Restore all internal indices** - - Run the following API call to restore all internal indices from a snapshot to the cluster: - - ```sh - POST /_snapshot/repo/snapshot/_restore - { - "indices": ".*", - "ignore_unavailable": true, - "include_global_state": false, - "include_aliases": false, - "rename_pattern": ".(.+)", - "rename_replacement": "restored_security_$1" - } - ``` - - * **Restore an individual internal index** - - ::::{warning} - When restoring internal indices, ensure that the `include_aliases` parameter is set to `false`. Not doing so will make Kibana inaccessible. If you do run the restore without `include_aliases`, the restored index can be deleted or the alias reference to it can be removed. This will have to be done from either the API console or a curl command as Kibana will not be accessible. - :::: - - - Run the following API call to restore one internal index from a snapshot to the cluster: - - ```sh - POST /_snapshot/repo/snapshot/_restore - { - "indices": ".kibana", - "ignore_unavailable": true, - "include_global_state": false, - "include_aliases": false, - "rename_pattern": ".(.+)", - "rename_replacement": "restored_security_$1" - } - ``` - - Next, the restored index needs to be reindexed into the internal index, as shown: - - ```sh - POST _reindex - { - "source": { - "index": "restored_kibana" - }, - "dest": { - "index": ".kibana" - } - } - ``` - - -Your internal Elasticsearch index or indices should now be available in your new Elasticsearch cluster. Once verified, the `restored_*` indices are safe to delete. diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-migrate-data2.md b/raw-migrated-files/cloud/cloud-heroku/ech-migrate-data2.md deleted file mode 100644 index fc75911466..0000000000 --- a/raw-migrated-files/cloud/cloud-heroku/ech-migrate-data2.md +++ /dev/null @@ -1,159 +0,0 @@ -# Migrate your Elasticsearch data [ech-migrate-data2] - -You might have switched to Elasticsearch Add-On for Heroku for any number of reasons and you’re likely wondering how to get your existing Elasticsearch data into your new infrastructure. Along with easily creating as many new deployments with Elasticsearch clusters that you need, you have several options for moving your data over. Choose the option that works best for you: - -* Index your data from the original source, which is the simplest method and provides the greatest flexibility for the Elasticsearch version and ingestion method. -* Reindex from a remote cluster, which rebuilds the index from scratch. -* Restore from a snapshot, which copies the existing indices. - -One of the many advantages of Elasticsearch Add-On for Heroku is that you can spin up a deployment quickly, try out something, and then delete it if you don’t like it. This flexibility provides the freedom to experiment while your existing production cluster continues to work. - - -## Before you begin [echbefore_you_begin_3] - -Depending on which option that you choose, you might have limitations or need to do some preparation beforehand. - -Indexing from the source -: The new cluster must be the same size as your old one, or larger, to accommodate the data. - -Reindex from a remote cluster -: The new cluster must be the same size as your old one, or larger, to accommodate the data. Depending on your security settings for your old cluster, you might need to temporarily allow TCP traffic on port 9243 for this procedure. - -Restore from a snapshot -: The new cluster must be the same size as your old one, or larger, to accommodate the data. The new cluster must also be an Elasticsearch version that is compatible with the old cluster (check [Elasticsearch snapshot version compatibility](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html#snapshot-restore-version-compatibility) for details). If you have not already done so, you will need to [set up snapshots for your old cluster](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html) using a repository that can be accessed from the new cluster. - -Migrating internal Elasticsearch indices -: If you are migrating internal Elasticsearch indices from another cluster, specifically the `.kibana` index or the `.security` index, there are two options: - - * Use the steps on this page to reindex the internal indices from a remote cluster. The steps for reindexing internal indices and regular, data indices are the same. - * Check [Migrating internal indices](../../../manage-data/migrate/migrate-internal-indices.md) to restore the internal Elasticsearch indices from a snapshot. - - -::::{warning} -Before you migrate your Elasticsearch data, [define your index mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) on the new cluster. Index mappings are unable to migrate during reindex operations. -:::: - - - -## Index from the source [ech-index-source] - -If you still have access to the original data source, outside of your old Elasticsearch cluster, you can load the data from there. This might be the simplest option, allowing you to choose the Elasticsearch version and take advantage of the latest features. You have the option to use any ingestion method that you want—​Logstash, Beats, the Elasticsearch clients, or whatever works best for you. - -If the original source isn’t available or has other issues that make it non-viable, there are still two more migration options, getting the data from a remote cluster or restoring from a snapshot. - - -## Reindex from a remote cluster [ech-reindex-remote] - -Through the Elasticsearch reindex API, you can connect your new Elasticsearch Add-On for Heroku deployment remotely to your old Elasticsearch cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run. - -1. Log in to the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. Select a deployment or create one. -3. If the old Elasticsearch cluster is on a remote host (any type of host accessible over the internet), you need to make sure that the host can be accessed. Access is determined by the Elasticsearch `reindex.remote.whitelist` user setting. - - Domains matching the pattern `["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`. - - Otherwise, if your remote endpoint is not covered by the default settings, adjust the setting to add the remote Elasticsearch cluster as an allowed host: - - 1. From your deployment menu, go to the **Edit** page. - 2. In the **Elasticsearch** section, select **Manage user settings and extensions**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for each node type instead. - 3. Add the following `reindex.remote.whitelist: [REMOTE_HOST:PORT]` user setting, where `REMOTE_HOST` is a pattern matching the URL for the remote Elasticsearch host that you are reindexing from, and PORT is the host port number. Do not include the `https://` prefix. - - Note that if you override the parameter it replaces the defaults: `["*.io:*", "*.com:*"]`. If you still want these patterns to be allowed you need to specify them explicitly in the value. - - For example: - - `reindex.remote.whitelist: ["*.us-east-1.aws.found.io:9243", "*.com:*"]` - - 4. Save your changes. - -4. From the **API Console** or in the Kibana Console app, create the destination index on Elasticsearch Add-On for Heroku. -5. Copy the index from the remote cluster: - - ```sh - POST _reindex - { - "source": { - "remote": { - "host": "https://REMOTE_ELASTICSEARCH_ENDPOINT:PORT", - "username": "USER", - "password": "PASSWORD" - }, - "index": "INDEX_NAME", - "query": { - "match_all": {} - } - }, - "dest": { - "index": "INDEX_NAME" - } - } - ``` - -6. Verify that the new index is present: - - ```sh - GET INDEX-NAME/_search?pretty - ``` - -7. You can remove the reindex.remote.whitelist user setting that you added previously. - - -## Restore from a snapshot [ech-restore-snapshots] - -If you cannot connect to a remote index for whatever reason, such as if it’s in a non-working state, you can try restoring from the most recent working snapshot. - -1. On your old Elasticsearch cluster, choose an option to get the name of your snapshot repository bucket: - - ```sh - GET /_snapshot - GET /_snapshot/_all - ``` - -2. Get the snapshot name: - - ```sh - GET /_snapshot/NEW-REPOSITORY-NAME/_all - ``` - - The output for each entry provides a `"snapshot":` value which is the snapshot name. - - ```json - { - "snapshots": [ - { - "snapshot": "scheduled-1527616008-instance-0000000004", - ... - }, - ... - ] - } - ``` - -3. From the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body) of the **new** Elasticsearch cluster, add the snapshot repository. For details, check our guidelines for [Amazon Web Services (AWS) Storage](../../../deploy-manage/tools/snapshot-and-restore/ech-aws-custom-repository.md), [Google Cloud Storage (GCS)](../../../deploy-manage/tools/snapshot-and-restore/ech-gcs-snapshotting.md), or [Azure Blob Storage](../../../deploy-manage/tools/snapshot-and-restore/ech-azure-snapshotting.md). - - ::::{important} - If you’re migrating [searchable snapshots](../../../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters. - :::: - - - ::::{important} - If source cluster is still writing to the repository, you need to set the destination cluster’s repository connection to `readonly:true` to avoid data corruption. Refer to [backup a repository](../../../deploy-manage/tools/snapshot-and-restore/self-managed.md#snapshots-repository-backup) for details. - :::: - -4. Start the Restore process. - - 1. Open Kibana and go to **Management** > **Snapshot and Restore**. - 2. Under the **Snapshots** tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it. - 3. Select **Restore**. - 4. Select the indices you wish to restore. - 5. Configure any additional index settings. - 6. Select **Restore snapshot** to begin the process. - -5. Verify that the new index is restored in your Elasticsearch Add-On for Heroku deployment with this query: - - ```sh - GET INDEX_NAME/_search?pretty - ``` - - - diff --git a/raw-migrated-files/cloud/cloud/ec-migrate-data-internal.md b/raw-migrated-files/cloud/cloud/ec-migrate-data-internal.md deleted file mode 100644 index d1858d6968..0000000000 --- a/raw-migrated-files/cloud/cloud/ec-migrate-data-internal.md +++ /dev/null @@ -1,92 +0,0 @@ -# Migrate internal indices [ec-migrate-data-internal] - -When you migrate your Elasticsearch data into a new infrastructure you may also want to migrate your Elasticsearch internal indices, specifically the `.kibana` index and the `.security` index. - -There are two ways to migrate the internal Elasticsearch indices: - -1. Reindex the indices from a remote cluster. -2. Restore the indices from a snapshot. - -To reindex internal indices from a remote cluster, you can follow the same steps that you use to reindex regular indices when you [migrate your Elasticsearch data indices](../../../manage-data/migrate.md#ec-reindex-remote). - -To restore internal indices from a snapshot, the procedure is a bit different from migrating Elasticsearch data indices. Use these steps to restore internal indices from a snapshot: - -1. On your old Elasticsearch cluster, choose an option to get the name of your snapshot repository bucket: - - ```sh - GET /_snapshot - GET /_snapshot/_all - ``` - -2. Get the snapshot name: - - ```sh - GET /_snapshot/NEW-REPOSITORY-NAME/_all - ``` - - The output for each entry provides a `"snapshot":` value which is the snapshot name. - - ``` - { - "snapshots": [ - { - "snapshot": "scheduled-1527616008-instance-0000000004", - ``` - -3. To restore internal Elasticsearch indices, you need to register the snapshot repository in `read-only` mode. To do so, first add the authentication information for the repository to the Elasticsearch Service keystore, following the steps for [AWS S3](../../../deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md#ec-snapshot-secrets-keystore), [Google Cloud Storage](../../../deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md#ec-configure-gcs-keystore), or [Azure Blog storage](../../../deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md#ec-configure-azure-keystore). -4. To register a read-only repository, open the Elasticsearch [API console](https://www.elastic.co/guide/en/cloud/current/ec-api-console.html) or the Kibana [Dev Tools page](../../../explore-analyze/query-filter/tools.md) and run the [Read-only URL repository](../../../deploy-manage/tools/snapshot-and-restore/read-only-url-repository.md) API call. -5. Once the repository has been registered and verified, you are ready to restore the internal indices to your new cluster, either all at once or individually. - - * **Restore all internal indices** - - Run the following API call to restore all internal indices from a snapshot to the cluster: - - ```sh - POST /_snapshot/repo/snapshot/_restore - { - "indices": ".*", - "ignore_unavailable": true, - "include_global_state": false, - "include_aliases": false, - "rename_pattern": ".(.+)", - "rename_replacement": "restored_security_$1" - } - ``` - - * **Restore an individual internal index** - - ::::{warning} - When restoring internal indices, ensure that the `include_aliases` parameter is set to `false`. Not doing so will make Kibana inaccessible. If you do run the restore without `include_aliases`, the restored index can be deleted or the alias reference to it can be removed. This will have to be done from either the API console or a curl command as Kibana will not be accessible. - :::: - - - Run the following API call to restore one internal index from a snapshot to the cluster: - - ```sh - POST /_snapshot/repo/snapshot/_restore - { - "indices": ".kibana", - "ignore_unavailable": true, - "include_global_state": false, - "include_aliases": false, - "rename_pattern": ".(.+)", - "rename_replacement": "restored_security_$1" - } - ``` - - Next, the restored index needs to be reindexed into the internal index, as shown: - - ```sh - POST _reindex - { - "source": { - "index": "restored_kibana" - }, - "dest": { - "index": ".kibana" - } - } - ``` - - -Your internal Elasticsearch index or indices should now be available in your new Elasticsearch cluster. Once verified, the `restored_*` indices are safe to delete. diff --git a/raw-migrated-files/cloud/cloud/ec-migrating-data.md b/raw-migrated-files/cloud/cloud/ec-migrating-data.md deleted file mode 100644 index 569481afff..0000000000 --- a/raw-migrated-files/cloud/cloud/ec-migrating-data.md +++ /dev/null @@ -1,174 +0,0 @@ -# Migrating data [ec-migrating-data] - -If you want to move your existing Elasticsearch data into your new infrastructure, check out the [migration options](../../../manage-data/migrate.md#ec-migrate-data2). You’ll find instructions to guide you through: - -* Migrating data [from its original source](../../../manage-data/migrate.md#ec-index-source) -* Reindexing data [from a remote Elasticsearch cluster](../../../manage-data/migrate.md#ec-reindex-remote) -* Restoring data [into a new deployment](../../../deploy-manage/tools/snapshot-and-restore/ece-restore-snapshots-into-new-deployment.md) -* Restoring data [from a snapshot](../../../manage-data/migrate.md#ec-restore-snapshots) -* Migrating [internal Elasticsearch indices](../../../manage-data/migrate/migrate-internal-indices.md) -* Migrating [through remote reindex](../../../manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md) - - -## Migrate your Elasticsearch data [ec-migrate-data2] - -You might have switched to Elasticsearch Service for any number of reasons and you’re likely wondering how to get your existing Elasticsearch data into your new infrastructure. Along with easily creating as many new deployments with Elasticsearch clusters that you need, you have several options for moving your data over. Choose the option that works best for you: - -* Index your data from the original source, which is the simplest method and provides the greatest flexibility for the Elasticsearch version and ingestion method. -* Reindex from a remote cluster, which rebuilds the index from scratch. -* Restore from a snapshot, which copies the existing indices. - -One of the many advantages of Elasticsearch Service is that you can spin up a deployment quickly, try out something, and then delete it if you don’t like it. This flexibility provides the freedom to experiment while your existing production cluster continues to work. - - -### Before you begin [ec_before_you_begin_2] - -Depending on which option that you choose, you might have limitations or need to do some preparation beforehand. - -Indexing from the source -: The new cluster must be the same size as your old one, or larger, to accommodate the data. - -Reindex from a remote cluster -: The new cluster must be the same size as your old one, or larger, to accommodate the data. Depending on your security settings for your old cluster, you might need to temporarily allow TCP traffic on port 9243 for this procedure. - -If your cluster is self-managed with a self-signed certificate, you can follow this [step-by-step migration guide](../../../manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md). - -Restore from a snapshot -: The new cluster must be the same size as your old one, or larger, to accommodate the data. The new cluster must also be an Elasticsearch version that is compatible with the old cluster (check [Elasticsearch snapshot version compatibility](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html#snapshot-restore-version-compatibility) for details). If you have not already done so, you will need to [set up snapshots for your old cluster](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html) using a repository that can be accessed from the new cluster. - -Migrating internal Elasticsearch indices -: If you are migrating internal Elasticsearch indices from another cluster, specifically the `.kibana` index or the `.security` index, there are two options: - - * Use the steps on this page to reindex the internal indices from a remote cluster. The steps for reindexing internal indices and regular, data indices are the same. - * Check [Migrating internal indices](../../../manage-data/migrate/migrate-internal-indices.md) to restore the internal Elasticsearch indices from a snapshot. - - -::::{warning} -Before you migrate your Elasticsearch data, [define your index mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) on the new cluster. Index mappings are unable to migrate during reindex operations. -:::: - - - -### Index from the source [ec-index-source] - -If you still have access to the original data source, outside of your old Elasticsearch cluster, you can load the data from there. This might be the simplest option, allowing you to choose the Elasticsearch version and take advantage of the latest features. You have the option to use any ingestion method that you want—​Logstash, Beats, the Elasticsearch clients, or whatever works best for you. - -If the original source isn’t available or has other issues that make it non-viable, there are still two more migration options, getting the data from a remote cluster or restoring from a snapshot. - - -### Reindex from a remote cluster [ec-reindex-remote] - -Through the Elasticsearch reindex API, you can connect your new Elasticsearch Service deployment remotely to your old Elasticsearch cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run. - -1. Log in to the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. Select a deployment or create one. -3. If the old Elasticsearch cluster is on a remote host (any type of host accessible over the internet), you need to make sure that the host can be accessed. Access is determined by the Elasticsearch `reindex.remote.whitelist` user setting. - - Domains matching the pattern `["*.io:*", "*.com:*"]` are allowed by default, so if your remote host URL matches that pattern you do not need to explicitly define `reindex.remote.whitelist`. - - Otherwise, if your remote endpoint is not covered by the default settings, adjust the setting to add the remote Elasticsearch cluster as an allowed host: - - 1. From your deployment menu, go to the **Edit** page. - 2. In the **Elasticsearch** section, select **Manage user settings and extensions**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for each node type instead. - 3. Add the following `reindex.remote.whitelist: [REMOTE_HOST:PORT]` user setting, where `REMOTE_HOST` is a pattern matching the URL for the remote Elasticsearch host that you are reindexing from, and PORT is the host port number. Do not include the `https://` prefix. - - Note that if you override the parameter it replaces the defaults: `["*.io:*", "*.com:*"]`. If you still want these patterns to be allowed you need to specify them explicitly in the value. - - For example: - - `reindex.remote.whitelist: ["*.us-east-1.aws.found.io:9243", "*.com:*"]` - - 4. Save your changes. - -4. From the **API Console** or in the Kibana Console app, create the destination index on Elasticsearch Service. -5. Copy the index from the remote cluster: - - ```sh - POST _reindex - { - "source": { - "remote": { - "host": "https://REMOTE_ELASTICSEARCH_ENDPOINT:PORT", - "username": "USER", - "password": "PASSWORD" - }, - "index": "INDEX_NAME", - "query": { - "match_all": {} - } - }, - "dest": { - "index": "INDEX_NAME" - } - } - ``` - -6. Verify that the new index is present: - - ```sh - GET INDEX-NAME/_search?pretty - ``` - -7. You can remove the reindex.remote.whitelist user setting that you added previously. - - -### Restore from a snapshot [ec-restore-snapshots] - -If you cannot connect to a remote index for whatever reason, such as if it’s in a non-working state, you can try restoring from the most recent working snapshot. - -1. On your old Elasticsearch cluster, choose an option to get the name of your snapshot repository bucket: - - ```sh - GET /_snapshot - GET /_snapshot/_all - ``` - -2. Get the snapshot name: - - ```sh - GET /_snapshot/NEW-REPOSITORY-NAME/_all - ``` - - The output for each entry provides a `"snapshot":` value which is the snapshot name. - - ```json - { - "snapshots": [ - { - "snapshot": "scheduled-1527616008-instance-0000000004", - ... - }, - ... - ] - } - ``` - -3. From the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body) of the **new** Elasticsearch cluster, add the snapshot repository. For details, check our guidelines for [Amazon Web Services (AWS) Storage](../../../deploy-manage/tools/snapshot-and-restore/ec-aws-custom-repository.md), [Google Cloud Storage (GCS)](../../../deploy-manage/tools/snapshot-and-restore/ec-gcs-snapshotting.md), or [Azure Blob Storage](../../../deploy-manage/tools/snapshot-and-restore/ec-azure-snapshotting.md). - - ::::{important} - If you’re migrating [searchable snapshots](../../../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md), the repository name must be identical in the source and destination clusters. - :::: - - - ::::{important} - If source cluster is still writing to the repository, you need to set the destination cluster’s repository connection to `readonly:true` to avoid data corruption. Refer to [backup a repository](../../../deploy-manage/tools/snapshot-and-restore/self-managed.md#snapshots-repository-backup) for details. - :::: - -4. Start the Restore process. - - 1. Open Kibana and go to **Management** > **Snapshot and Restore**. - 2. Under the **Snapshots** tab, you can find the available snapshots from your newly added snapshot repository. Select any snapshot to view its details, and from there you can choose to restore it. - 3. Select **Restore**. - 4. Select the indices you wish to restore. - 5. Configure any additional index settings. - 6. Select **Restore snapshot** to begin the process. - -5. Verify that the new index is restored in your Elasticsearch Service deployment with this query: - - ```sh - GET INDEX_NAME/_search?pretty - ``` - - - - diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index 60707228e9..08db4316ed 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -75,7 +75,6 @@ toc: - file: cloud/cloud-enterprise/ece-manage-kibana-settings.md - file: cloud/cloud-enterprise/ece-manage-kibana.md - file: cloud/cloud-enterprise/ece-migrate-index-management.md - - file: cloud/cloud-enterprise/ece-migrating-data.md - file: cloud/cloud-enterprise/ece-monitoring-deployments.md - file: cloud/cloud-enterprise/ece-password-reset-elastic.md - file: cloud/cloud-enterprise/ece-restful-api-examples-configuring-keystore.md @@ -130,8 +129,6 @@ toc: - file: cloud/cloud-heroku/ech-manage-apm-settings.md - file: cloud/cloud-heroku/ech-manage-kibana-settings.md - file: cloud/cloud-heroku/ech-metrics-memory-pressure.md - - file: cloud/cloud-heroku/ech-migrate-data-internal.md - - file: cloud/cloud-heroku/ech-migrate-data2.md - file: cloud/cloud-heroku/ech-monitoring-setup.md - file: cloud/cloud-heroku/ech-monitoring.md - file: cloud/cloud-heroku/ech-multiple-node-deployment-disk-used.md @@ -206,9 +203,7 @@ toc: - file: cloud/cloud/ec-manage-enterprise-search-settings.md - file: cloud/cloud/ec-manage-kibana-settings.md - file: cloud/cloud/ec-metrics-memory-pressure.md - - file: cloud/cloud/ec-migrate-data-internal.md - file: cloud/cloud/ec-migrate-index-management.md - - file: cloud/cloud/ec-migrating-data.md - file: cloud/cloud/ec-monitoring-setup.md - file: cloud/cloud/ec-monitoring.md - file: cloud/cloud/ec-password-reset.md From 5f43cafe9f3a67f8d3e2792818e873b101a748d0 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 12 Feb 2025 15:12:53 -0500 Subject: [PATCH 2/4] fix broken links --- manage-data/migrate.md | 4 ++-- ...ter-with-a-self-signed-certificate-using-remote-reindex.md | 1 - .../cloud/cloud-enterprise/ece-create-deployment.md | 2 +- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/manage-data/migrate.md b/manage-data/migrate.md index d776720ae5..a1f2f41557 100644 --- a/manage-data/migrate.md +++ b/manage-data/migrate.md @@ -36,7 +36,7 @@ Migrating internal {{es}} indices : For {{ech}} and Elasticsearch Add-On for Heroku, if you are migrating internal {{es}} indices from another cluster, specifically the `.kibana` index or the `.security` index, there are two options: * Use the steps on this page to reindex the internal indices from a remote cluster. The steps for reindexing internal indices and regular, data indices are the same. - * Check [Migrating internal indices](../../../manage-data/migrate/migrate-internal-indices.md) to restore the internal {{es}} indices from a snapshot. + * Check [Migrating internal indices](migrate/migrate-internal-indices.md) to restore the internal {{es}} indices from a snapshot. ::::{warning} Before you migrate your {{es}} data, [define your index mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) on the new cluster. Index mappings are unable to migrate during reindex operations. @@ -48,7 +48,7 @@ If you still have access to the original data source, outside of your old {{es}} If the original source isn’t available or has other issues that make it non-viable, there are still two more migration options, getting the data from a remote cluster or restoring from a snapshot. -### Reindex from a remote cluster [ec-reindex-remote] +### Reindex from a remote cluster [ec-reindex-remote] [ech-reindex-remote] Through the {{es}} reindex API, you can connect your new {{es}} Service deployment remotely to your old {{es}} cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run. diff --git a/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md b/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md index 63808232be..46cf897460 100644 --- a/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md +++ b/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md @@ -50,7 +50,6 @@ The `Destination` cluster should be the same or newer version as the `Source` cl :::: - ## Step 4: Enable bundle and add `reindex` settings on the `Desination` cluster. [ec-remote-reindex-step4] 1. From your deployment page, go to the **Edit** page, click **Manage user settings and extensions**, select tab **Extensions** then enable `my_source_ca`. diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-create-deployment.md b/raw-migrated-files/cloud/cloud-enterprise/ece-create-deployment.md index df50435001..b5c805804b 100644 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-create-deployment.md +++ b/raw-migrated-files/cloud/cloud-enterprise/ece-create-deployment.md @@ -45,7 +45,7 @@ If these system templates are not suitable for your use case, you can [create yo ::: -After a deployment is spun up, you can scale the size and add other features; however, the instance configuration and computing ratios cannot be changed. If you need to change an existing deployment to another template, we recommend [migrating your data](../../../manage-data/migrate.md#ece-migrate-data2). +After a deployment is spun up, you can scale the size and add other features; however, the instance configuration and computing ratios cannot be changed. If you need to change an existing deployment to another template, we recommend [migrating your data](../../../manage-data/migrate.md). From 42af4f984016d3e84a4ede3773189119cb7cf914 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 12 Feb 2025 15:37:20 -0500 Subject: [PATCH 3/4] fix more broken links --- manage-data/migrate.md | 2 +- manage-data/migrate/migrate-internal-indices.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/manage-data/migrate.md b/manage-data/migrate.md index a1f2f41557..fb6fffbb3c 100644 --- a/manage-data/migrate.md +++ b/manage-data/migrate.md @@ -48,7 +48,7 @@ If you still have access to the original data source, outside of your old {{es}} If the original source isn’t available or has other issues that make it non-viable, there are still two more migration options, getting the data from a remote cluster or restoring from a snapshot. -### Reindex from a remote cluster [ec-reindex-remote] [ech-reindex-remote] +### Reindex from a remote cluster [ech-reindex-remote] Through the {{es}} reindex API, you can connect your new {{es}} Service deployment remotely to your old {{es}} cluster. This pulls the data from your old cluster and indexes it into your new one. Reindexing essentially rebuilds the index from scratch and it can be more resource intensive to run. diff --git a/manage-data/migrate/migrate-internal-indices.md b/manage-data/migrate/migrate-internal-indices.md index 4c9c09f380..944c64978d 100644 --- a/manage-data/migrate/migrate-internal-indices.md +++ b/manage-data/migrate/migrate-internal-indices.md @@ -17,7 +17,7 @@ There are two ways to migrate the internal {{es}} indices: 1. Reindex the indices from a remote cluster. 2. Restore the indices from a snapshot. -To reindex internal indices from a remote cluster, you can follow the same steps that you use to reindex regular indices when you [migrate your {{es}} data indices](../migrate.md#ec-reindex-remote). +To reindex internal indices from a remote cluster, you can follow the same steps that you use to reindex regular indices when you [migrate your {{es}} data indices](../migrate.md#ech-reindex-remote). To restore internal indices from a snapshot, the procedure is a bit different from migrating {{es}} data indices. Use these steps to restore internal indices from a snapshot: From a786f56b9ad27b19c2a2ee0b815d16fa1a7cbb1a Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Thu, 13 Feb 2025 11:22:52 -0500 Subject: [PATCH 4/4] Shorten the navtitle for self-managed reindex --- ...luster-with-a-self-signed-certificate-using-remote-reindex.md | 1 + 1 file changed, 1 insertion(+) diff --git a/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md b/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md index 46cf897460..27ccec7519 100644 --- a/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md +++ b/manage-data/migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md @@ -5,6 +5,7 @@ applies: serverless: unavailable hosted: all ece: unavailable +navigation_title: Reindex from a self-managed cluster --- # Migrate from a self-managed cluster with a self-signed certificate using remote reindex [ec-remote-reindex]