add logs data retention

Author: mdbirnstiehl

solutions/observability/logs/logs-data-retention.md

@@ -0,0 +1,82 @@
+---
+applies_to:
+  stack: all
+products:
+  - id: observability
+---
+
+# Configure logs data retention
+
+This page explains how to manage log data retention using index lifecycle management (ILM). You’ll learn about customizing the built-in `logs@lifecycle` policy, automating rollover, and inspecting and managing ILM policies.
+
+## Store logs in data streams
+
+A data stream lets you store append-only time series data across multiple indices while giving you a single named resource for requests. Data streams also provide the following benefits:
+
+- ILM out of the box to automate the management of the backing indices.
+- Automatic rollover to ensure backing indices stay within optimal size and performance limits.
+- Tiered storage (hot, warm, and cold phases) to optimize storage and performance.
+
+Refer to the [data stream](/manage-data/data-store/data-streams.md) docs for more information.
+
+## Customize the built-in `logs@lifecycle` policy
+
+The `logs@lifecycle` ILM policy is preconfigured for common logging use cases. View or duplicate the policy at **Stack Management** → **Index Lifecycle Policies** or find `Index Lifecycle Policies` in the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
+
+The logs ILM policy provides a good foundation for your logs data streams, but you may need to tailor it to fit your specific use case. Common modifications include:
+
+- Adjust phase transitions.
+- Set retention durations for different phases.
+- Update rollover conditions.
+
+Refer to the **[Customize built-in policies tutorial](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md)** for more on modifying the logs ILM policy.
+
+% might need some more common modifications or reasons why you might want to make these modifications.
+
+## Automate rollover based on log volume
+
+When continuously indexing timestamped documents, you'll want to periodically roll over to a new index to ensure that backing indices stay within optimal size and performance limits.
+
+% any logs specific limits the user might want to set and why?
+
+Refer to the [Automate rollover tutorial](/manage-data/lifecycle/index-lifecycle-management/tutorial-automate-rollover.md) for more information.
+
+## Use the ILM API
+
+You can also review ILM policies using the `Get lifecycle policies` API.
+
+For example, running `GET /_ilm/policy/logs@lifecycle` pulls up the default logs ILM policy, and you'll see something like the following:
+
+```json
+{
+  "logs@lifecycle": {
+    "version": 1,
+    "modified_date": "2025-05-19T16:45:58.754Z",
+    "policy": {
+      "phases": {
+        "hot": {
+          "min_age": "0ms",
+          "actions": {
+            "rollover": {
+              "max_age": "30d",
+              "max_primary_shard_size": "50gb"
+            }
+          }
+        }
+      },
+      "_meta": {
+        "description": "default policy for the logs index template installed by x-pack",
+        "managed": true
+      },
+      "deprecated": false
+    },
+    "in_use_by": {
+      "indices": [],
+      "data_streams": [],
+      "composable_templates": []
+    }
+  }
+}
+```
+
+Refer to the [ILM API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-ilm) for more information.

solutions/toc.yml

@@ -404,6 +404,7 @@ toc:
           - file: observability/logs/run-pattern-analysis-on-log-data.md
           - file: observability/logs/add-service-name-to-logs.md
           - file: observability/logs/logs-index-template-reference.md
+          - file: observability/logs/logs-data-retention.md
           - file: observability/logs/streams/streams.md
             children:
             - file: observability/logs/streams/management/extract.md