[DOCS] Describe how to use Elastic Agent to monitor Elasticsearch (#94348) (#94754)

* [DOCS] Describe how to use Elastic Agent to monitor Elasticsearch

* Temporarily fix doc build

* Add question about showing Elastic Agent metrics in the monitoring UI

* Apply changes from review

* Activate link to Kibana docs

* Fix broken link

* Update docs/reference/monitoring/indices.asciidoc

(cherry picked from commit a6c7c1f)

Author: dedemorton

docs/reference/monitoring/collecting-monitoring-data.asciidoc

@@ -8,7 +8,7 @@
 include::{es-ref-dir}/settings/monitoring-settings.asciidoc[tag=monitoring-deprecation-notice]
 
 This method for collecting metrics about {es} involves sending the metrics to
-the monitoring cluster by using exporters. For the recommended method, see <<configuring-metricbeat>>.
+the monitoring cluster by using exporters.
 
 Advanced monitoring settings enable you to control how frequently data is
 collected, configure timeouts, and set the retention period for locally-stored

docs/reference/monitoring/collectors.asciidoc

@@ -2,17 +2,7 @@
 [[es-monitoring-collectors]]
 == Collectors
 
-[IMPORTANT]
-=========================
-{metricbeat} is the recommended method for collecting and shipping monitoring
-data to a monitoring cluster.
-
-If you have previously configured legacy collection methods, you should migrate
-to using {metricbeat} collection methods. Use either {metricbeat} collection or
-legacy collection methods; do not use both.
-
-Learn more about <<configuring-metricbeat>>.
-=========================
+include::production.asciidoc[tag=monitoring-rec]
 
 Collectors, as their name implies, collect things. Each collector runs once for
 each collection interval to obtain data from the public APIs in {es} and {xpack}

docs/reference/monitoring/configuring-elastic-agent.asciidoc

@@ -0,0 +1,73 @@
+[[configuring-elastic-agent]]
+== Collecting {es} monitoring data with {agent}
+
+[subs="attributes"]
+++++
+<titleabbrev>Collecting monitoring data with {agent}</titleabbrev>
+++++
+
+In 8.5 and later, you can use {agent} to collect data about {es} and ship it to
+the monitoring cluster, rather than <<configuring-metricbeat,using {metricbeat}>>
+or routing it through exporters as described in <<collecting-monitoring-data>>.
+
+[discrete]
+=== Prerequisites
+
+* (Optional) Create a monitoring cluster as described in <<monitoring-production>>.
+
+* Create a user on the production cluster that has the
+`remote_monitoring_collector` {ref}/built-in-roles.html[built-in role].
+
+[discrete]
+=== Add {es} monitoring data
+
+To collect {es} monitoring data, add an {es} integration to an {agent} and
+deploy it to the host where {es} is running.
+
+. Go to the {kib} home page and click **Add integrations**.
+. In the query bar, search for and select the **{es}** integration for
+{agent}.
+. Read the overview to make sure you understand integration requirements and
+other considerations.
+. Click **Add Elasticsearch**.
++
+TIP: If you're installing an integration for the first time, you may be prompted
+to install {agent}. Click **Add integration only (skip agent installation)**.
+
+. Configure the integration name and optionally add a description. Make sure you
+configure all required settings:
+.. Under **Collect Elasticsearch logs**, modify the log paths to match your {es}
+environment.
+.. Under **Collect Elasticsearch metrics**, make sure the hosts setting points to
+your {es} host URLs. By default, the integration collects {es} monitoring
+metrics from `localhost:9200`. If that host and port number are not correct,
+update the `hosts` setting. If you configured {es} to use encrypted
+communications, you must access it via HTTPS. For example, use a `hosts` setting
+like `https://localhost:9200`.
+.. Expand **Advanced options**. If the Elastic {security-features} are enabled,
+enter the username and password of a user that has the
+`remote_monitoring_collector` role.
+.. Specify the scope:
+** Specify `cluster` if each entry in the hosts list indicates a single
+endpoint for a distinct {es} cluster (for example, a load-balancing proxy
+fronting the cluster that directs requests to the master-ineligible nodes in the
+cluster).
+** Otherwise, accept the default scope, `node`. If this scope is set, you
+will need to install {agent} on each {es} node to collect all metrics. {agent}
+will collect most of the metrics from the elected master of the cluster, so you
+must scale up all your master-eligible nodes to account for this extra load. Do
+not use this `node` if you have dedicated master nodes.
+. Choose where to add the integration policy. Click **New hosts** to add it to
+new agent policy or **Existing hosts** to add it to an existing agent policy.
+. Click **Save and continue**. This step takes a minute or two to complete. When
+it's done, you'll have an agent policy that contains an integration for
+collecting monitoring data from {es}.
+. If an {agent} is already assigned to the policy and deployed to the host where
+{es} is running, you're done. Otherwise, you need to deploy an {agent}. To
+deploy an {agent}:
+.. Go to **{fleet} -> Agents**, then click **Add agent**.
+.. Follow the steps in the **Add agent** flyout to download, install,
+and enroll the {agent}. Make sure you choose the agent policy you created
+earlier.
+. Wait a minute or two until incoming data is confirmed.
+. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}]. 

docs/reference/monitoring/configuring-filebeat.asciidoc

@@ -11,6 +11,9 @@ You can use {filebeat} to monitor the {es} log files, collect log events, and
 ship them to the monitoring cluster. Your recent logs are visible on the
 *Monitoring* page in {kib}.
 
+IMPORTANT: If you're using {agent}, do not deploy {filebeat} for log collection.
+Instead, configure the {es} integration to collect logs.
+
 //NOTE: The tagged regions are re-used in the Stack Overview.
 
 . Verify that {es} is running and that the monitoring cluster is ready to

docs/reference/monitoring/configuring-metricbeat.asciidoc

@@ -9,7 +9,9 @@
 
 In 6.5 and later, you can use {metricbeat} to collect data about {es} 
 and ship it to the monitoring cluster, rather than routing it through exporters 
-as described in <<collecting-monitoring-data>>. 
+as described in <<collecting-monitoring-data>>.
+
+Want to use {agent} instead? Refer to <<configuring-elastic-agent>>.
 
 image::monitoring/images/metricbeat.png[Example monitoring architecture]
 

docs/reference/monitoring/exporters.asciidoc

@@ -2,17 +2,7 @@
 [[es-monitoring-exporters]]
 == Exporters
 
-[IMPORTANT]
-=========================
-{metricbeat} is the recommended method for collecting and shipping monitoring
-data to a monitoring cluster.
-
-If you have previously configured legacy collection methods, you should migrate
-to using {metricbeat} collection methods. Use either {metricbeat} collection or
-legacy collection methods; do not use both.
-
-Learn more about <<configuring-metricbeat>>.
-=========================
+include::production.asciidoc[tag=monitoring-rec]
 
 The purpose of exporters is to take data collected from any Elastic Stack
 source and route it to the monitoring cluster. It is possible to configure

docs/reference/monitoring/how-monitoring-works.asciidoc

@@ -5,28 +5,41 @@
 <titleabbrev>How it works</titleabbrev>
 ++++
 
-Each {es} node, {ls} node, {kib} instance, and Beat instance is considered
-unique in the cluster based on its persistent UUID, which is written to the
-<<path-settings,`path.data`>> directory when the node or instance starts.
+Each monitored {stack} component is considered unique in the cluster based on
+its persistent UUID, which is written to the <<path-settings,`path.data`>>
+directory when the node or instance starts.
 
 Monitoring documents are just ordinary JSON documents built by monitoring each
-{stack} component at a specified collection interval. If you want to alter the
-templates for these indices, see <<config-monitoring-indices>>.
+{stack} component at a specified collection interval. If you want to alter how
+these documents are structured or stored, refer to <<config-monitoring-indices>>.
 
-{metricbeat} is used to collect monitoring data and to ship it directly to the
-monitoring cluster.
+You can use {agent} or {metricbeat} to collect monitoring data and to ship it
+directly to the monitoring cluster.
 
-To learn how to collect monitoring data, see:
+To learn how to collect monitoring data, refer to:
 
-* <<collecting-monitoring-data>>
-* <<configuring-metricbeat>>
+* One of the following topics depending on how you want to collect monitoring
+data from {es}:
+** <<configuring-elastic-agent>>: Uses a single agent to
+gather logs and metrics. Can be managed from a central location in {fleet}.
+** <<configuring-metricbeat>>: Uses a lightweight {beats}
+shipper to gather metrics. May be preferred if you have an existing investment
+in {beats} or are not yet ready to use {agent}.
+** <<collecting-monitoring-data>>: Uses internal exporters to
+gather metrics. Not recommended. If you have previously configured legacy
+collection methods, you should migrate to using {agent} or {metricbeat}.
 * {kibana-ref}/xpack-monitoring.html[Monitoring {kib}]
 * {logstash-ref}/configuring-logstash.html[Monitoring {ls}]
-* Monitoring Beats:
+* {enterprise-search-ref}/monitoring.html[Monitoring {ents}]
+* Monitoring {beats}:
 ** {auditbeat-ref}/monitoring.html[{auditbeat}]
 ** {filebeat-ref}/monitoring.html[{filebeat}]
 ** {functionbeat-ref}/monitoring.html[{functionbeat}]
 ** {heartbeat-ref}/monitoring.html[{heartbeat}]
 ** {metricbeat-ref}/monitoring.html[{metricbeat}]
 ** {packetbeat-ref}/monitoring.html[{packetbeat}]
 ** {winlogbeat-ref}/monitoring.html[{winlogbeat}]
+* {apm-guide-ref}/monitor-apm.html[Monitoring APM Server]
+* {fleet-guide}/monitor-elastic-agent.html[Monitoring {agent}s]
+{fleet}-managed agents) or
+{fleet-guide}/elastic-agent-monitoring-configuration.html[Configure monitoring for standalone {agent}s]

docs/reference/monitoring/http-export.asciidoc

@@ -2,17 +2,7 @@
 [[http-exporter]]
 === HTTP exporters
 
-[IMPORTANT]
-=========================
-{metricbeat} is the recommended method for collecting and shipping monitoring
-data to a monitoring cluster.
-
-If you have previously configured legacy collection methods, you should migrate
-to using {metricbeat} collection methods. Use either {metricbeat} collection or
-legacy collection methods; do not use both.
-
-Learn more about <<configuring-metricbeat>>.
-=========================
+include::production.asciidoc[tag=monitoring-rec]
 
 The `http` exporter is the preferred exporter in the {es} {monitor-features}
 because it enables the use of a separate monitoring cluster. As a secondary

docs/reference/monitoring/index.asciidoc

@@ -10,6 +10,7 @@ performance of your {es} cluster.
 * <<monitoring-overview>>
 * <<how-monitoring-works>>
 * <<monitoring-production>>
+* <<configuring-elastic-agent>>
 * <<configuring-metricbeat>>
 * <<configuring-filebeat>>
 * <<config-monitoring-indices>>
@@ -24,6 +25,8 @@ include::how-monitoring-works.asciidoc[]
 
 include::production.asciidoc[]
 
+include::configuring-elastic-agent.asciidoc[]
+
 include::configuring-metricbeat.asciidoc[]
 
 include::configuring-filebeat.asciidoc[]

docs/reference/monitoring/indices.asciidoc

@@ -2,8 +2,27 @@
 [[config-monitoring-indices]]
 == Configuring indices for monitoring
 
-<<indices-templates-v1,Index templates>> are used to configure the indices
-that store the monitoring data collected from a cluster.
+Sometimes the default index settings might not work for your situation. For
+example, you might want to change index lifecycle management (ILM) settings,
+add custom mappings, or set the number of shards and replicas
+for the monitoring indices.
+
+You can change the default behavior. The steps you follow depend on which data
+collection method is used.
+
+[float]
+=== Configuring data streams created by {agent}
+
+{agent} uses data streams to store time series data across multiple indices
+while giving you a single named resource for requests. You can alter the
+settings of each dataset by configuring an `@custom` component template.
+For more information, refer to {fleet-guide}/data-streams.html[Data streams].
+
+[float]
+=== Configuring indices created by {metricbeat}
+
+<<indices-templates-v1,Index templates>> are used to configure the indices that
+store the monitoring data that {metricbeat} collects from a cluster.
 
 You can retrieve the templates through the `_template` API:
 
@@ -32,7 +51,8 @@ PUT /_index_template/custom_monitoring
   ".monitoring-beats-8-*",
   ".monitoring-es-8-*",
   ".monitoring-kibana-8-*",
-  ".monitoring-logstash-8-*"
+  ".monitoring-logstash-8-*",
+  ".monitoring-ent-search-8-*"
   ],
   "priority": 1,
   "template": {