Skip to content

Commit 23478ac

Browse files
abdonpijpelinkklacabane
abdonpijpelink
and
klacabane
authored
[DOCS] Clarify configuring indices/data streams used by stack monitoring (#95285) (#95549)
* [DOCS] Clarify indices/data streams using by stack monitoring * Incorporate review feedback * Update docs/reference/monitoring/indices.asciidoc --------- Co-authored-by: Kevin Lacabane <[email protected]>
1 parent 9bff987 commit 23478ac

File tree

1 file changed

+124
-56
lines changed

1 file changed

+124
-56
lines changed

‎docs/reference/monitoring/indices.asciidoc‎

Lines changed: 124 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -1,79 +1,147 @@
11
[role="xpack"]
22
[[config-monitoring-indices]]
3-
== Configuring indices for monitoring
3+
== Configuring data streams/indices for monitoring
44

5-
Sometimes the default index settings might not work for your situation. For
6-
example, you might want to change index lifecycle management (ILM) settings,
7-
add custom mappings, or set the number of shards and replicas
8-
for the monitoring indices.
5+
Monitoring data is stored in data streams or indices in {es}. The default data
6+
stream or index settings may not work for your situation. For example, you might
7+
want to change index lifecycle management (ILM) settings, add custom mappings,
8+
or change the number of shards and replicas. The steps to change these
9+
settings depend on the monitoring method:
910

10-
You can change the default behavior. The steps you follow depend on which data
11-
collection method is used.
11+
* <<config-monitoring-data-streams-elastic-agent>>
12+
* <<config-monitoring-data-streams-metricbeat-8>>
13+
* <<config-monitoring-indices-metricbeat-7-internal-collection>> (the default
14+
for the {ess} on {ecloud})
1215

13-
[float]
16+
IMPORTANT: Changing mappings or settings can cause your monitoring dashboards to
17+
stop working correctly.
18+
19+
[[config-monitoring-data-streams-elastic-agent]]
1420
=== Configuring data streams created by {agent}
1521

16-
{agent} uses data streams to store time series data across multiple indices
17-
while giving you a single named resource for requests. You can alter the
18-
settings of each dataset by configuring an `@custom` component template.
19-
For more information, refer to {fleet-guide}/data-streams.html[Data streams].
22+
When <<configuring-elastic-agent,monitoring using {agent}>>, data is stored in a
23+
set of data streams named
24+
`metrics-{product}.stack_monitoring.{dataset}-{namespace}`. For example:
25+
`metrics-elasticsearch.stack_monitoring.shard-default`.
26+
27+
The settings and mappings for these data streams are determined by an index
28+
template named `metrics-{product}.stack_monitoring.{dataset}`. For example:
29+
`metrics-elasticsearch.stack_monitoring.shard`.
30+
31+
To change the settings of each data stream, edit the
32+
`metrics-{product}.stack_monitoring.{dataset}@custom` component template that
33+
already exists. You can do this in {kib}:
34+
35+
* Navigate to *Stack Management* > *Index Management* > *Component Templates*.
36+
* Search for the component template.
37+
* Select the *Edit* action.
38+
39+
You can also use the {es} API:
40+
41+
* Retrieve the component template using the <<getting-component-templates,get
42+
component template API>>.
43+
* Edit the component template.
44+
* Store the updated component template using the <<indices-component-template,
45+
update component template API>>.
46+
47+
After changing the component template, the updated settings are only applied
48+
to the data stream's new backing indices.
49+
<<manually-roll-over-a-data-stream,Roll over the data stream>> to immediately
50+
apply the updated settings to the data stream’s write index.
51+
52+
[[config-monitoring-data-streams-metricbeat-8]]
53+
=== Configuring data streams created by {metricbeat} 8
54+
55+
When <<configuring-metricbeat,monitoring using {metricbeat} 8>>, data is stored
56+
in a set of data streams called `.monitoring-{product}-8-mb`. For example:
57+
`.monitoring-es-8-mb`.
58+
59+
The settings and mappings for these data streams are determined by an index
60+
template named `.monitoring-{product}-mb`. For example: `.monitoring-es-mb`. You
61+
can alter the settings of each data stream by cloning this index template and
62+
editing it.
63+
64+
WARNING: You need to repeat this procedure when upgrading the {stack} to get the
65+
latest updates to the default monitoring index templates.
2066

21-
[float]
22-
=== Configuring indices created by {metricbeat}
67+
You can clone index templates in {kib}:
2368

24-
<<indices-templates-v1,Index templates>> are used to configure the indices that
25-
store the monitoring data that {metricbeat} collects from a cluster.
69+
* Navigate to *Stack Management* > *Index Management* > *Index Templates*.
70+
* From the *View* dropdown, select *System templates*.
71+
* Search for the index template.
72+
* Select the *Clone* action.
73+
* Change the name, for example into `custom_monitoring`.
74+
* Set the priority to `500`, to ensure it overrides the default index template.
75+
* Specify the settings you want to change in the `settings` section.
76+
* Save the cloned template.
2677

27-
You can retrieve the templates through the `_template` API:
78+
You can also use the {es} API:
79+
80+
* Retrieve the index template using the <<indices-get-template,get index
81+
template API>>.
82+
* Edit the index template: set the template `priority` to `500`, and specify the
83+
settings you want to change in the `settings` section.
84+
* Store the updated index template under a different name, for example
85+
`custom_monitoring`, using the
86+
<<indices-put-template,create index template API>>.
87+
88+
NOTE: {metricbeat} 8 uses <<index-templates,composable templates>>, rather than
89+
legacy templates.
90+
91+
After changing the index template, the updated settings are only applied to the
92+
data stream's new backing indices.
93+
<<manually-roll-over-a-data-stream,Roll over the data stream>> to immediately
94+
apply the updated settings to the data stream’s write index.
95+
96+
[[config-monitoring-indices-metricbeat-7-internal-collection]]
97+
=== Configuring indices created by {metricbeat} 7 or internal collection
98+
99+
When monitoring <<configuring-metricbeat,using {metricbeat} 7>> or
100+
{filebeat-ref}/monitoring-internal-collection.html[internal collection], data is
101+
stored in a set of indices called either:
102+
103+
* `.monitoring-{product}-7-mb-{date}`, when using {metricbeat} 7.
104+
* `.monitoring-{product}-7-{date}`, when using internal collection.
105+
106+
The settings and mappings for these indices are determined by
107+
<<indices-templates-v1,legacy index templates>> named `.monitoring-{product}`.
108+
You can retrieve these templates in {kib} by navigating to *Stack Management* >
109+
*Index Management* > *Index Templates*, or by using the {es} `_template` API:
28110

29111
[source,console]
30-
----------------------------------
31-
GET /_index_template/.monitoring-es-mb*
32-
----------------------------------
112+
----
113+
GET /_template/.monitoring-*
114+
----
33115

34-
By default, the template configures one shard and one replica for the
35-
monitoring indices. To override the default settings, add your own template:
116+
To change the settings of the indices, add a custom index template. You can do
117+
that in {kib}, or using the {es} API:
36118

37-
. Set `index_patterns` to match existing `.monitoring-{product}-8-*` indices.
38-
. Set the template `priority` to `1`. This ensures your template is
39-
applied after the default template, which has an priority of 0.
40-
. Specify the `number_of_shards` and/or `number_of_replicas` in the `settings`
119+
* Set `index_patterns` to match the `.monitoring-{product}-7-*` indices.
120+
* Set the template `order` to `1`. This ensures your template is
121+
applied after the default template, which has an order of 0.
122+
* Specify the `number_of_shards` and/or `number_of_replicas` in the `settings`
41123
section.
42124

43-
For example, the following template increases the number of shards to five
44-
and the number of replicas to two.
45-
46125
[source,console]
47-
----------------------------------
48-
PUT /_index_template/custom_monitoring
126+
----
127+
PUT /_template/custom_monitoring
49128
{
50-
"index_patterns": [
51-
".monitoring-beats-8-*",
52-
".monitoring-es-8-*",
53-
".monitoring-kibana-8-*",
54-
".monitoring-logstash-8-*",
55-
".monitoring-ent-search-8-*"
56-
],
57-
"priority": 1,
58-
"template": {
59-
"settings": {
60-
"number_of_shards": 5,
61-
"number_of_replicas": 2
62-
}
63-
}
129+
"index_patterns": [".monitoring-beats-7-*", ".monitoring-es-7-*", ".monitoring-kibana-7-*", ".monitoring-logstash-7-*"],
130+
"order": 1,
131+
"settings": {
132+
"number_of_shards": 5,
133+
"number_of_replicas": 2
134+
}
64135
}
65-
----------------------------------
136+
----
66137

67-
//////////////////////////
138+
After changing the index template, the updated settings are only applied to new
139+
indices.
68140

141+
////
69142
[source,console]
70-
--------------------------------------------------
71-
DELETE /_index_template/custom_monitoring
72-
--------------------------------------------------
143+
----
144+
DELETE /_template/custom_monitoring
145+
----
73146
// TEST[continued]
74-
75-
//////////////////////////
76-
77-
IMPORTANT: Only set the `number_of_shards` and `number_of_replicas` in the
78-
settings section. Overriding other monitoring template settings could cause
79-
your monitoring dashboards to stop working correctly.
147+
////

0 commit comments

Comments
 (0)