Skip to content
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 15 additions & 9 deletions docs/detections/detection-engine-intro.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -51,20 +51,26 @@ To make sure you can access Detections and manage rules, see

[float]
[[cold-tier-detections]]
== Compatibility with cold and frozen tier nodes
== Manage data in cold and frozen tiers

Cold and frozen {ref}/data-tiers.html[data tiers] hold time series data that is only accessed occasionally. In {stack} version >=7.11.0, {elastic-sec} supports cold but not frozen tier data for the following {es} indices:
Cold data tiers store time series data that is accessed infrequently and rarely updated, while frozen data tiers hold time series data that is accessed even less frequently and never updated. If you are automating searches across different {ref}/data-tiers.html[data tiers] using rules, consider the following best practices and limitations.

* Index patterns specified in `securitySolution:defaultIndex`
* Index patterns specified in the definitions of detection rules, except for indicator match rules
* Index patterns specified in the data sources selector on various {security-app} pages
[float]
[[best-practices-data-tiers]]
=== Best practices

* **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. {ilm-cap} policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues.
* **Replicas for Mission-Critical Data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can re-run the rule over the affected time period using <<manually-run-rules,manual runs>>.

{elastic-sec} does *NOT* support either cold or frozen tier data for the following {es} indices:
[float]
[[limitations-data-tiers]]
=== Limitations

* Index patterns controlled by {elastic-sec}, including alerts and list indices
* Index patterns specified in the definition of indicator match rules
Data tiers are a powerful and useful tool. When using them, keep the following limitations in mind:

Using either cold or frozen tier data for unsupported indices may result in detection rule timeouts and overall performance degradation.
* {ilm-cap} policies for indices controlled by {elastic-sec}, including alerts and list indices, must not be modified.
* Indicator match rule performance can be severely impacted by querying data in frozen tiers.
* Cold and frozen source data must have an {ilm} policy that keeps it in the hot or warm tiers for at least one day.

[float]
[[support-indicator-rules]]
Expand Down