Skip to content

Commit abd13c8

Browse files
Add: 'Manage the lifecycle for integrations data' (#2344)
Integrations are our primary ingest mechanism but they currently aren't mentioned in the ILM docs. We should explain how, starting from integrations content in the UI (i.e., from dashboard visualizations, the Integrations page, and the Fleet UI), to find the underlying data stream that our users need in order to customize the data lifecycle for incoming Integrations data. This PR adds a new [Manage the lifecycle for integrations data](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/2344/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data) page to help connect the dots. Closes: #1450 --------- Co-authored-by: Vlada Chirmicci <[email protected]>
1 parent 4a7f923 commit abd13c8

File tree

9 files changed

+69
-4
lines changed

9 files changed

+69
-4
lines changed
562 KB
Loading
107 KB
Loading
389 KB
Loading
277 KB
Loading
170 KB
Loading

manage-data/lifecycle/index-lifecycle-management.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ For example, if you are indexing metrics data from a fleet of ATMs into Elastics
6868
You can create and manage index lifecycle policies through {{kib}}'s [Index Management](/manage-data/data-store/index-basics.md#index-management) UI or the [{{ilm-init}} APIs](https://www.elastic.co/docs/api/doc/elasticsearch/v8/group/endpoint-ilm). For more details on creating and managing index lifecycle policies refer to:
6969

7070
* [Configure a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md)
71-
* [View the lifecycle status of an index](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md)
71+
* [View the lifecycle status of an index or datastream](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md)
7272
* [Update or switch a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md)
7373
* [Start and stop index lifecycle management](/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md)
7474
* [Restore a managed data stream or index](/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md)
Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
---
2+
navigation_title: Manage the lifecycle policy for integrations data
3+
applies_to:
4+
stack: ga
5+
products:
6+
- id: elasticsearch
7+
---
8+
9+
# Manage the lifecycle policy for integrations data [ilm-manage-lifecycle-policy-integrations-data]
10+
11+
An Elastic integration is a pre-packaged collection of assets that provides an effective, simplified way to monitor a product, system, or service, with minimal required setup. Most integrations rely on {{agent}} as an ingest mechanism, and the policies used to govern installed integrations are managed in {{fleet}}.
12+
13+
You can find installation and configuration details for all integrations in the [Elastic integrations](https://docs.elastic.co/en/integrations) chapter of the **Reference** section. To learn about managing your installed integrations, refer to [Manage Elastic Agent integrations](/reference/fleet/manage-integrations.md) in the {{fleet}} and {{agent}} chapter of the **Reference** section.
14+
15+
When you install an integration, an [{{ilm}}](/manage-data/lifecycle/index-lifecycle-management.md) ({{ilm-init}}) policy is configured automatically to manage the integration's component [data streams](/manage-data/data-store/data-streams.md) and their backing indices. To view or adjust how your integration data is managed, a first step is to find the data streams that you're interested in. There are a few ways to do this:
16+
17+
::::{dropdown} Find the data stream for a {{kib}} visualation
18+
To find the data stream associated with a visualization in a {{kib}}:
19+
20+
1. Open **Dashboards** from the {{kib}} menu and select a dashboard to view. For example, with the [System integration](integration-docs://reference/system.md) installed, you can open the `[Metrics System] Host overview` dashboard to find visualizations about the host system being monitored.
21+
22+
1. Hover over any visualization and click the **Explore in Discover** icon.
23+
24+
![Explore in discover](/manage-data/images/ilm-explore-in-discover.png "")
25+
26+
1. In **Discover**, the list of documents shows the columns for fields applicable to the visualzation that you selected. Select any document that has data for those fields and click the **Toggle dialog with details** icon.
27+
28+
![Discover documents list](/manage-data/images/ilm-toggle-document-details.png "")
29+
30+
1. In the document details, note that there are three `data_stream` fields. The full [data stream name](/reference/fleet/data-streams.md#data-streams-naming-scheme) is a composite of `data_stream.type`, `data_stream.dataset` and `data_stream.namespace`, separated by a hyphen. For example, in the System integration, the **CPU usage over time** visualization is associated with the `metrics-system.cpu-default` data stream.
31+
32+
You can also see the data stream's current backing index, as well as other information such as the document timestamp and details about the agent that ingested the data.
33+
34+
![Document details](/manage-data/images/ilm-document-data-stream.png "")
35+
::::
36+
37+
::::{dropdown} Find the data streams for an integration
38+
To find the data streams associated with an installed integration:
39+
40+
1. In {{kib}} go to **Management > Integrations > Installed integrations** and select any integration.
41+
42+
1. Open the **Assets** tab and expand the **Index templates** list.
43+
44+
In the list, the name of each [index template](/manage-data/data-store/templates.md) matches an associated data stream. For example, the `metrics-system.cpu` template matches the `metrics-system.cpu-default` data stream that is set up when the System integration is installed.
45+
46+
![Integration assets](/manage-data/images/ilm-integration-assets.png "")
47+
::::
48+
49+
::::{dropdown} Find the data streams managed in {{fleet}}
50+
To find all of the data streams that are managed in {{fleet}}:
51+
52+
1. In {{kib}} go to **Management > {{fleet}}** and select the **Data streams** tab.
53+
54+
1. Use the search field and dropdown menus to filter the list. You can filter by the data stream type, dataset, namespace, or by the integration that the data stream belongs to.
55+
56+
![Integration assets](/manage-data/images/ilm-fleet-data-streams.png "")
57+
::::
58+
59+
For any data stream that you're interested in, you can [view its current lifecycle status](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md), including details about its associated ILM policy.
60+
61+
After you've identified one or more data streams for which you'd like to customize how the data is managed over time, refer to our tutorials:
62+
63+
* For a general guide about configuring a custom ILM policy for any managed data stream, try out our [Customize built-in policies](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) tutorial in the data lifecycle documentation.
64+
* For the steps to customize an ILM policy for a set of data streams, such as all logs or metrics data streams across all namespaces, across only a selected namespace, and others, check the set of tutorials in [Customize data retention policies](/reference/fleet/data-streams-ilm-tutorial.md) in the {{fleet}} and {{agent}} reference documentation.

manage-data/lifecycle/index-lifecycle-management/policy-view-status.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
navigation_title: View the lifecycle status of an index
2+
navigation_title: View the lifecycle status of an index or data stream
33
mapped_pages:
44
- https://www.elastic.co/guide/en/elasticsearch/reference/current/update-lifecycle-policy.html
55
applies_to:
@@ -8,9 +8,9 @@ products:
88
- id: elasticsearch
99
---
1010

11-
# View the lifecycle status of an index [view-lifecycle-status]
11+
# View the lifecycle status of an index or data stream [view-lifecycle-status]
1212

13-
For any existing managed index in your cluster, you can access the ILM policy applied to it and details such as its current phase.
13+
For any existing managed index or data stream in your cluster, you can access the ILM policy applied to it and its current status.
1414

1515
:::::{tab-set}
1616
:group: kibana-api

manage-data/toc.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,7 @@ toc:
124124
- file: lifecycle/index-lifecycle-management/configure-lifecycle-policy.md
125125
- file: lifecycle/index-lifecycle-management/policy-apply.md
126126
- file: lifecycle/index-lifecycle-management/policy-view-status.md
127+
- file: lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md
127128
- file: lifecycle/index-lifecycle-management/policy-updates.md
128129
- file: lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md
129130
- file: lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md

0 commit comments

Comments
 (0)