[8.16] Updates docs for and related to the excludedDataTiersForRuleExecution advanced setting (#5962)

nastasha-solomon · marshallmain · benironside · mergify[bot] · commit ac4b3f2d12a3 · 2024-11-14T20:37:56.000Z
* First draft * Updating IM rules * disclaimer about certain rule types and shards * Minor tweak to dsl query docs * Update docs/detections/detection-engine-intro.asciidoc Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> * Update docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> * Update docs/getting-started/advanced-setting.asciidoc Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> * Update docs/release-notes/8.16.asciidoc Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> * Marshall's suggestion * Update docs/detections/detection-engine-intro.asciidoc * Removes note that's no longer needed * Moves file back to remove this change from the PR * Update docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc * Update docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc * Updates what's new * Fixed title * Update docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc * Update docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc * Update docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc * Update docs/getting-started/advanced-setting.asciidoc * Update docs/getting-started/advanced-setting.asciidoc * Update docs/whats-new.asciidoc * Update docs/whats-new.asciidoc * Update docs/release-notes/8.16.asciidoc * Fixes a typo * Minor wording adjustments * Update docs/whats-new.asciidoc Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> * Update docs/release-notes/8.16.asciidoc Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> * Update docs/getting-started/advanced-setting.asciidoc * Update docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> * Update docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> * Update docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> * Update docs/detections/detection-engine-intro.asciidoc Co-authored-by: Yara Tercero <yctercero@users.noreply.github.com> --------- Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com> Co-authored-by: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Co-authored-by: Yara Tercero <yctercero@users.noreply.github.com> (cherry picked from commit cd4f12b)
diff --git a/docs/detections/detection-engine-intro.asciidoc b/docs/detections/detection-engine-intro.asciidoc
@@ -86,7 +86,7 @@ Indicator match rules provide a powerful capability to search your security data
 
 In addition, the following support restrictions are in place:
 
-* {elastic-sec} does not support the use of either cold or frozen {ref}/data-tiers.html[tier data] with indicator match rules.
+* Indicator match rules don't support cold or frozen data. Cold or frozen data in indices queried by indicator match rules must be older than the time range queried by the rule. If your data's timestamps are unreliable, you can exclude cold and frozen tier data using a <<exclude-cold-frozen-data-individual-rules,Query DSL filter>>. 
 * Indicator match rules with an additional look-back time value greater than 24 hours are not supported.
 
 [float]
diff --git a/docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc b/docs/detections/detections-exclude-cold-frozen-data-tiers.asciidoc
@@ -1,18 +1,25 @@
 [[exclude-cold-frozen-data-individual-rules]]
-== Exclude cold and frozen data from a rule
+== Exclude cold and frozen data from individual rules
 
 :frontmatter-description: Configure a rule to ignore cold and frozen data during execution. 
 :frontmatter-tags-products: [security]
 :frontmatter-tags-content-type: [how-to]
 :frontmatter-tags-user-goals: [manage]
 
-Rules that query cold and frozen data might perform more slowly. To exclude cold and frozen data, add a Query DSL filter that ignores cold and frozen {ref}/data-tiers.html[data tiers] when executing. You can add the filter when creating a new rule or updating an existing one. 
+Your rule might perform slower or fail if it queries data from cold or frozen {ref}/data-tiers.html[data tiers]. To help Elasticsearch exclude cold and frozen data more efficiently, apply a Query DSL filter that ignores cold and frozen documents when your rule executes. You can add the filter when creating a new rule or updating an existing one.
 
-NOTE: This method is not supported for {esql} and {ml} rules.
+TIP: To ensure that rules in your {kib} space exclude cold and frozen documents when executing, configure the `excludedDataTiersForRuleExecution` <<exclude-cold-frozen-data-rule-executions,advanced setting>>. This setting does not apply to {ml} rules. 
 
-TIP: To ensure that _all_ rules in a {kib} space exclude cold and frozen data when executing, configure the `excludedDataTiersForRuleExecution` <<exclude-cold-frozen-data-rule-executions,advanced setting>>.
+[IMPORTANT]
+====
 
-Here is a sample Query DSL filter that excludes frozen tier data from a rule's execution:
+* This method is not supported for {esql} and {ml} rules.
+* Even when applying this filter, indicator match and event correlation rules may still fail if a frozen or cold shard that matches the rule's specified index pattern is unavailable during rule executions. If failures occur, we recommend modifying the rule's index patterns to only match indices containing hot tier data.
+
+
+====
+
+Here is a sample Query DSL filter that excludes frozen tier documents during rule execution:
 
 [source,console]
 ----
@@ -29,7 +36,7 @@ Here is a sample Query DSL filter that excludes frozen tier data from a rule's e
 }
 ----
 
-Here is another sample Query DSL filter that excludes cold and frozen tier data from a rule's execution:
+Here is another sample Query DSL filter that excludes cold and frozen tier documents during rule execution:
 
 [source,console]
 ----
diff --git a/docs/getting-started/advanced-setting.asciidoc b/docs/getting-started/advanced-setting.asciidoc
@@ -187,15 +187,17 @@ The `securitySolution:maxUnassociatedNotes` field determines the maximum number
 
 [discrete]
 [[exclude-cold-frozen-data-rule-executions]]
-== Exclude cold and frozen data from rule executions 
+== Exclude cold and frozen data from rules 
 
-To ensure rules don't search cold and frozen data when executing, specify cold and frozen {ref}/data-tiers.html[data tiers] in the `excludedDataTiersForRuleExecution` field. Multiple data tiers must be separated by commas, for example: `data_frozen`, `data_cold`. This setting is turned off by default; turning it on can improve rule performance and reduce execution time. 
+To ensure the rules in your {kib} space exclude query results from cold and frozen tiers when executing, specify cold and frozen {ref}/data-tiers.html[data tiers] in the `excludedDataTiersForRuleExecution` field. Multiple data tiers must be separated by commas, for example: `data_frozen`, `data_cold`. This setting is turned off by default; turning it on can improve rule performance and reduce execution time. 
 
-This setting does not apply to {ml} rules. 
+This setting does not apply to {ml} rules because {ml} anomalies are not stored in cold or frozen data tiers.
 
 [TIP]
 ====
 
-This setting applies to all rules in a {kib} space. To only exclude cold and frozen data from specific rules, add a <<exclude-cold-frozen-data-individual-rules,Query DSL filter>> to the rules you want affected. 
+To only exclude cold and frozen data from specific rules, add a <<exclude-cold-frozen-data-individual-rules,Query DSL filter>> to the rules you want affected. 
 
-====
+====
+
+IMPORTANT: Even when the `excludedDataTiersForRuleExecution` advanced setting is enabled, indicator match, event correlation, and {esql} rules may still fail if a frozen or cold shard that matches the rule's specified index pattern is unavailable during rule executions. If failures occur, we recommend modifying the rule's index patterns to only match indices containing hot tier data.
diff --git a/docs/release-notes/8.16.asciidoc b/docs/release-notes/8.16.asciidoc
@@ -104,7 +104,9 @@ NOTE: These instructions only apply to the Google Chrome browser. Modify the ste
 * Allows you to view {es} queries that run during rule execution. This option is provided for {esql} and EQL rules only ({kibana-pull}191107[#191107]).
 * Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]).
 * Introduces a new advanced setting, `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]).
-* Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). 
+* Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule executions ({kibana-pull}186908[#186908]). 
++
+IMPORTANT: Even when the `excludedDataTiersForRuleExecution` advanced setting is enabled, indicator match, event correlation, and {esql} rules may still fail if a frozen or cold shard that matches the rule's specified index pattern is unavailable during rule executions. If failures occur, we recommend modifying the rule's index patterns to only match indices containing hot tier data.
 * Enhances the Insights section of the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]).
 * Turns off the host field size reduction setting on {elastic-defend}'s integration policy by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <<adv-policy-settings,advanced policy setting>>.
 * Allows you to reduce CPU usage, I/O, and event sizes by turning on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <<adv-policy-settings,advanced policy setting>>.
diff --git a/docs/whats-new.asciidoc b/docs/whats-new.asciidoc
@@ -85,9 +85,9 @@ image::whats-new/images/8.16/install-enable-rules.png[Install and enable rules,
 image::whats-new/images/8.16/manual-rule-run-table.png[Manual rule run table]
 
 [float]
-=== Exclude cold and frozen data from rule execution
+=== Exclude cold and frozen data from rules
 
-Rules that query cold and frozen data tiers might perform more slowly. To {security-guide}/exclude-cold-frozen-data-individual-rules.html[exclude query results from cold and frozen tiers], add a Query DSL filter that ignores cold and frozen documents when executing. This can help {es} exclude cold and frozen data more efficiently.
+Rules that query cold and frozen data tiers might perform more slowly or fail. To ensure that the rules in your {kib} space exclude query results from cold and frozen tiers when executing, configure the `excludedDataTiersForRuleExecution` <<exclude-cold-frozen-data-rule-executions,advanced setting>>.
 
 [float]
 === View {es} queries that run during rule execution
