From 1bbd571203966d1bdb493f0da93d94324ba4de96 Mon Sep 17 00:00:00 2001 From: Brendan Cully Date: Wed, 13 Nov 2024 12:07:28 -0800 Subject: [PATCH 1/2] Add warning about restart migration We have gotten more than one SDH due to customers not understanding why restarts involving fully-mounted indices can pull a lot of data from the snapshot tier, so it may help to be more explicit about why this happens and how it can be avoided. --- docs/reference/searchable-snapshots/index.asciidoc | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/reference/searchable-snapshots/index.asciidoc b/docs/reference/searchable-snapshots/index.asciidoc index a38971a0bae6a..2c5083b546037 100644 --- a/docs/reference/searchable-snapshots/index.asciidoc +++ b/docs/reference/searchable-snapshots/index.asciidoc @@ -176,10 +176,10 @@ nodes that have a shared cache. ==== Manually mounting snapshots captured by an Index Lifecycle Management ({ilm-init}) policy can interfere with {ilm-init}'s automatic management. This may lead to issues such as data loss -or complications with snapshot handling. +or complications with snapshot handling. For optimal results, allow {ilm-init} to manage -snapshots automatically. +snapshots automatically. <>. ==== @@ -293,6 +293,14 @@ repository. If you wish to search data across multiple regions, configure multiple clusters and use <> or <> instead of {search-snaps}. +It's worth noting that if a searchable snapshot index has no replicas (as is the default +in the cold tier), then when the node hosting it is shut down, allocation will immediately +try to relocate the index to a new node in order to maximize availability. For fully mounted +indices this will result in the new node downloading the entire index snapshot from +the cloud repository, which might be expensive especially during rolling restarts. Temporarily +disabling allocation during planned node restart will prevent this, as described in +the <>. + [discrete] [[back-up-restore-searchable-snapshots]] === Back up and restore {search-snaps} From d1950c593c86ffd765aad61598f8f5ec15ec2776 Mon Sep 17 00:00:00 2001 From: Brendan Cully Date: Thu, 14 Nov 2024 08:45:34 -0800 Subject: [PATCH 2/2] PR feedback on doc update --- docs/reference/searchable-snapshots/index.asciidoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/reference/searchable-snapshots/index.asciidoc b/docs/reference/searchable-snapshots/index.asciidoc index 2c5083b546037..8b0b3dc57686e 100644 --- a/docs/reference/searchable-snapshots/index.asciidoc +++ b/docs/reference/searchable-snapshots/index.asciidoc @@ -293,11 +293,11 @@ repository. If you wish to search data across multiple regions, configure multiple clusters and use <> or <> instead of {search-snaps}. -It's worth noting that if a searchable snapshot index has no replicas (as is the default -in the cold tier), then when the node hosting it is shut down, allocation will immediately -try to relocate the index to a new node in order to maximize availability. For fully mounted -indices this will result in the new node downloading the entire index snapshot from -the cloud repository, which might be expensive especially during rolling restarts. Temporarily +It's worth noting that if a searchable snapshot index has no replicas, then when the node +hosting it is shut down, allocation will immediately try to relocate the index to a new node +in order to maximize availability. For fully mounted indices this will result in the new node +downloading the entire index snapshot from the cloud repository. Under a rolling cluster restart, +this may happen multiple times for each searchable snapshot index. Temporarily disabling allocation during planned node restart will prevent this, as described in the <>.