From 3b5e68891004695da35dc31038d8a5136d7696fd Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Mon, 24 Mar 2025 11:30:07 -0400 Subject: [PATCH 01/13] First draft --- .../detection-engine-intro.asciidoc | 22 ++++++++++++------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 5607e31dab..43b452ea70 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -53,18 +53,24 @@ To make sure you can access Detections and manage rules, see [[cold-tier-detections]] == Compatibility with cold and frozen tier nodes -Cold and frozen {ref}/data-tiers.html[data tiers] hold time series data that is only accessed occasionally. In {stack} version >=7.11.0, {elastic-sec} supports cold but not frozen tier data for the following {es} indices: +Cold data tiers store time series data that is accessed infrequently and rarely updated, while frozen data tiers hold time series data that is accessed even less frequently and never updated. If you are automating searches across different {ref}/data-tiers.html[data tiers] using rules, consider the following best practices and limitations. -* Index patterns specified in `securitySolution:defaultIndex` -* Index patterns specified in the definitions of detection rules, except for indicator match rules -* Index patterns specified in the data sources selector on various {security-app} pages +[float] +[[best-practices-data-tiers]] +=== Best practices + +* **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. ILM policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. +* **Replicas for Mission-Critical Data**: Data that must always be available should have replicas. Since frozen tiers do not support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be encountered during or after upgrades. If this happens, you can re-run the rule over the affected time period using <>. -{elastic-sec} does *NOT* support either cold or frozen tier data for the following {es} indices: +[float] +[[limitations-data-tiers]] +=== Limitations -* Index patterns controlled by {elastic-sec}, including alerts and list indices -* Index patterns specified in the definition of indicator match rules +Data tiers are a powerful and useful tool. It is important to keep in mind some important limitations: -Using either cold or frozen tier data for unsupported indices may result in detection rule timeouts and overall performance degradation. +* Index lifecycle management policies for indices controlled by {elastic-sec}, including alerts and list indices, must not be modified. +* Indicator match rule performance can be severely impacted by querying data within frozen indices. +* Cold and frozen source data must have an index lifecycle management policy that keeps it in hot or warm for at least one day. [float] [[support-indicator-rules]] From 5ce789ed3182fce92b0635b6c30d3e0711564565 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Tue, 25 Mar 2025 14:01:03 -0400 Subject: [PATCH 02/13] Revisions --- docs/detections/detection-engine-intro.asciidoc | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 43b452ea70..239f01ea55 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -51,7 +51,7 @@ To make sure you can access Detections and manage rules, see [float] [[cold-tier-detections]] -== Compatibility with cold and frozen tier nodes +== Manage data in cold and frozen tiers Cold data tiers store time series data that is accessed infrequently and rarely updated, while frozen data tiers hold time series data that is accessed even less frequently and never updated. If you are automating searches across different {ref}/data-tiers.html[data tiers] using rules, consider the following best practices and limitations. @@ -59,18 +59,18 @@ Cold data tiers store time series data that is accessed infrequently and rarely [[best-practices-data-tiers]] === Best practices -* **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. ILM policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. -* **Replicas for Mission-Critical Data**: Data that must always be available should have replicas. Since frozen tiers do not support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be encountered during or after upgrades. If this happens, you can re-run the rule over the affected time period using <>. +* **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. {ilm-cap} policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. +* **Replicas for Mission-Critical Data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can re-run the rule over the affected time period using <>. [float] [[limitations-data-tiers]] === Limitations -Data tiers are a powerful and useful tool. It is important to keep in mind some important limitations: +Data tiers are a powerful and useful tool. When using them, keep the following limitations in mind: -* Index lifecycle management policies for indices controlled by {elastic-sec}, including alerts and list indices, must not be modified. -* Indicator match rule performance can be severely impacted by querying data within frozen indices. -* Cold and frozen source data must have an index lifecycle management policy that keeps it in hot or warm for at least one day. +* {ilm-cap} policies for indices controlled by {elastic-sec}, including alerts and list indices, must not be modified. +* Indicator match rule performance can be severely impacted by querying data in frozen tiers. +* Cold and frozen source data must have an {ilm} policy that keeps it in the hot or warm tiers for at least one day. [float] [[support-indicator-rules]] From c7870a6474739a77ad729d7f3ed78608911f05fb Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Tue, 25 Mar 2025 14:48:31 -0400 Subject: [PATCH 03/13] Update docs/detections/detection-engine-intro.asciidoc --- docs/detections/detection-engine-intro.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 239f01ea55..017bbc25bf 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -60,7 +60,7 @@ Cold data tiers store time series data that is accessed infrequently and rarely === Best practices * **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. {ilm-cap} policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. -* **Replicas for Mission-Critical Data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can re-run the rule over the affected time period using <>. +* **Replicas for Mission-Critical Data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can <> the rule over the affected time period. [float] [[limitations-data-tiers]] From b4b0084427c982f00761562f1c9a79bb909ce632 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Tue, 25 Mar 2025 14:56:07 -0400 Subject: [PATCH 04/13] lowercase --- docs/detections/detection-engine-intro.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 017bbc25bf..46c9c0beff 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -60,7 +60,7 @@ Cold data tiers store time series data that is accessed infrequently and rarely === Best practices * **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. {ilm-cap} policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. -* **Replicas for Mission-Critical Data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can <> the rule over the affected time period. +* **Replicas for mission-critical data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can <> the rule over the affected time period. [float] [[limitations-data-tiers]] From e5f887268709198f1bd500a4537d7fefd81b67cd Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Wed, 2 Apr 2025 13:11:04 -0400 Subject: [PATCH 05/13] Update docs/detections/detection-engine-intro.asciidoc Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> --- docs/detections/detection-engine-intro.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 46c9c0beff..4835933a1b 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -70,7 +70,7 @@ Data tiers are a powerful and useful tool. When using them, keep the following l * {ilm-cap} policies for indices controlled by {elastic-sec}, including alerts and list indices, must not be modified. * Indicator match rule performance can be severely impacted by querying data in frozen tiers. -* Cold and frozen source data must have an {ilm} policy that keeps it in the hot or warm tiers for at least one day. +* Source data must have an {ilm} policy that keeps it in the hot or warm tiers for at least one day before moving to cold or frozen tiers. [float] [[support-indicator-rules]] From 3c2b98d0fb46ebc946a6bb1175001d86d238dd9e Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Wed, 2 Apr 2025 13:13:10 -0400 Subject: [PATCH 06/13] Update docs/detections/detection-engine-intro.asciidoc Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> --- docs/detections/detection-engine-intro.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 4835933a1b..02cfd3d125 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -60,7 +60,7 @@ Cold data tiers store time series data that is accessed infrequently and rarely === Best practices * **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. {ilm-cap} policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. -* **Replicas for mission-critical data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can <> the rule over the affected time period. +* **Replicas for mission-critical data**: Your data should have replicas if it must be highly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can <> rules over the affected time period once the shards are available. [float] [[limitations-data-tiers]] From 480a65a75b3153285f1198814e1376579d297441 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Fri, 4 Apr 2025 13:40:32 -0400 Subject: [PATCH 07/13] Update docs/detections/detection-engine-intro.asciidoc --- docs/detections/detection-engine-intro.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 02cfd3d125..4d0c4ed1de 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -59,7 +59,7 @@ Cold data tiers store time series data that is accessed infrequently and rarely [[best-practices-data-tiers]] === Best practices -* **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. {ilm-cap} policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. +* **Retention in hot tier**: We recommend keeping data in the hot data tier ({ilm-cap} hot phase) for at least 24 hours. {ilm-cap} policies that move ingested data from the hot phase to another phase (for example, cold or frozen) in less than 24 hours may cause performance issues and/or rule execution errors. * **Replicas for mission-critical data**: Your data should have replicas if it must be highly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can <> rules over the affected time period once the shards are available. [float] From 2f2bda18c817df83482b29840b7de533bbc8715e Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Fri, 4 Apr 2025 14:22:15 -0400 Subject: [PATCH 08/13] contractions --- docs/detections/detection-engine-intro.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index b11691a028..413c0dabd9 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -53,7 +53,7 @@ To make sure you can access Detections and manage rules, see [[cold-tier-detections]] == Manage data in cold and frozen tiers -Cold data tiers store time series data that is accessed infrequently and rarely updated, while frozen data tiers hold time series data that is accessed even less frequently and never updated. If you are automating searches across different {ref}/data-tiers.html[data tiers] using rules, consider the following best practices and limitations. +Cold {ref}/data-tiers.html[data tiers] store time series data that's accessed infrequently and rarely updated, while frozen data tiers hold time series data that's accessed even less frequently and never updated. If you're automating searches across different data tiers using rules, consider the following best practices and limitations. [float] [[best-practices-data-tiers]] From 9c8598e45b5b112129a7f222bc46059e2546f2e7 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 7 Apr 2025 10:59:52 -0400 Subject: [PATCH 09/13] Update docs/detections/detection-engine-intro.asciidoc --- docs/detections/detection-engine-intro.asciidoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 413c0dabd9..b0ae1df125 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -60,7 +60,8 @@ Cold {ref}/data-tiers.html[data tiers] store time series data that's accessed in === Best practices * **Retention in hot tier**: We recommend keeping data in the hot data tier ({ilm-cap} hot phase) for at least 24 hours. {ilm-cap} policies that move ingested data from the hot phase to another phase (for example, cold or frozen) in less than 24 hours may cause performance issues and/or rule execution errors. -* **Replicas for mission-critical data**: Your data should have replicas if it must be highly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can <> rules over the affected time period once the shards are available. +* **Replicas for mission-critical data**: Your data should have replicas if it must be highly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures if rules query frozen tiers. Shard unavailability may also occur during or after {stack} upgrades. If this happens, you can <> rules over the affected time period once the shards are available. + [float] [[limitations-data-tiers]] From 5fc105a9c9511f0fd38cf3796f4d2974a3bbe73e Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Mon, 7 Apr 2025 15:18:25 -0400 Subject: [PATCH 10/13] Update docs/detections/detection-engine-intro.asciidoc --- docs/detections/detection-engine-intro.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index b0ae1df125..90be8afb0b 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -71,7 +71,7 @@ Data tiers are a powerful and useful tool. When using them, keep the following l * {ilm-cap} policies for indices controlled by {elastic-sec}, including alerts and list indices, must not be modified. * Indicator match rule performance can be severely impacted by querying data in frozen tiers. -* Source data must have an {ilm} policy that keeps it in the hot or warm tiers for at least one day before moving to cold or frozen tiers. +* Source data must have an {ilm} policy that keeps it in the hot or warm tiers for at least 24 hours before moving to cold or frozen tiers. [float] [[support-indicator-rules]] From 13616aaa6f7cd46af41b5e7537673006ec84e987 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Wed, 9 Apr 2025 17:12:15 -0400 Subject: [PATCH 11/13] Update docs/detections/detection-engine-intro.asciidoc --- docs/detections/detection-engine-intro.asciidoc | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 90be8afb0b..67be0e5fb4 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -70,7 +70,6 @@ Cold {ref}/data-tiers.html[data tiers] store time series data that's accessed in Data tiers are a powerful and useful tool. When using them, keep the following limitations in mind: * {ilm-cap} policies for indices controlled by {elastic-sec}, including alerts and list indices, must not be modified. -* Indicator match rule performance can be severely impacted by querying data in frozen tiers. * Source data must have an {ilm} policy that keeps it in the hot or warm tiers for at least 24 hours before moving to cold or frozen tiers. [float] From edea5c068390a867015cda7695619cf28ed8f0f3 Mon Sep 17 00:00:00 2001 From: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com> Date: Wed, 9 Apr 2025 21:52:35 -0400 Subject: [PATCH 12/13] Update docs/detections/detection-engine-intro.asciidoc --- docs/detections/detection-engine-intro.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index 67be0e5fb4..cdf2ef2c67 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -69,7 +69,7 @@ Cold {ref}/data-tiers.html[data tiers] store time series data that's accessed in Data tiers are a powerful and useful tool. When using them, keep the following limitations in mind: -* {ilm-cap} policies for indices controlled by {elastic-sec}, including alerts and list indices, must not be modified. +* To avoid rule failures, do not modify {ilm-cap} policies for {elastic-sec}-controlled indices, such as alert and list indices. * Source data must have an {ilm} policy that keeps it in the hot or warm tiers for at least 24 hours before moving to cold or frozen tiers. [float] From 495f0deb32e269ff9ebcefdf9ade9513d2ce2a36 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Thu, 10 Apr 2025 16:05:08 -0400 Subject: [PATCH 13/13] fixes --- docs/detections/detection-engine-intro.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc index cdf2ef2c67..5327037580 100644 --- a/docs/detections/detection-engine-intro.asciidoc +++ b/docs/detections/detection-engine-intro.asciidoc @@ -67,9 +67,9 @@ Cold {ref}/data-tiers.html[data tiers] store time series data that's accessed in [[limitations-data-tiers]] === Limitations -Data tiers are a powerful and useful tool. When using them, keep the following limitations in mind: +Data tiers are a powerful and useful tool. When using them, keep the following in mind: -* To avoid rule failures, do not modify {ilm-cap} policies for {elastic-sec}-controlled indices, such as alert and list indices. +* To avoid rule failures, do not modify {ilm} policies for {elastic-sec}-controlled indices, such as alert and list indices. * Source data must have an {ilm} policy that keeps it in the hot or warm tiers for at least 24 hours before moving to cold or frozen tiers. [float]