[D+M] Monitoring: orchestrators, kib task manager, logging config

deploy-manage/maintenance/start-stop-services/start-stop-elasticsearch.md

@@ -1,7 +1,7 @@
 ---
 mapped_urls:
-  - https://www.elastic.co/guide/en/{{es}}/reference/current/starting-elasticsearch.html
-  - https://www.elastic.co/guide/en/{{es}}/reference/current/stopping-elasticsearch.html
+  - https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html
+  - https://www.elastic.co/guide/en/elasticsearch/reference/current/stopping-elasticsearch.html
 applies_to:
   deployment:
      self:

deploy-manage/monitor.md

@@ -74,18 +74,31 @@ Out of the box logs and metrics tools, including ECH preconfigured logs and metr
 
 To learn more about the health and performance tools in {{ecloud}}, refer to [](/deploy-manage/monitor/cloud-health-perf.md).
 
+## {{kib}} task manager monitoring
+
+```{applies_to}
+stack: preview
+```
+The {{kib}} [task manager](/deploy-manage/distributed-architecture/kibana-tasks-management.md) has an internal monitoring mechanism to keep track of a variety of metrics, which can be consumed with either the health monitoring API or the {{kib}} server log. [Learn how to configure thresholds and consume related to {{kib}} task manager](/deploy-manage/monitor/kibana-task-manager-health-monitoring.md).
+
 ## Monitoring your orchestrator
 ```{applies_to}
 deployment:
   ece:
   eck:
 ```
 
-TODO
+In addition to monitoring your cluster or deployment health and performance, you need to monitor your orchestrator. Monitoring is especially important for orchestrators hosted on infrastructure that you control.
 
-## Logging
+Learn how to enable monitoring of your orchestrator:
+
+* [ECK operator metrics](/deploy-manage/monitor/orchestrators/eck-metrics-configuration.md): Open and secure a metrics endpoint that can be used to monitor the operator’s performance and health. This endpoint can be scraped by third-party Kubernetes monitoring tools.
+* [ECK platform monitoring](/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md): Learn about how ECE collects monitoring data for your installation in the `logging-and-metrics` deployment, and how to access monitoring data.
 
-TODO
+:::{admonition} Monitoring {{ecloud}}
+Elastic monitors [{{ecloud}}](/deploy-manage/deploy/elastic-cloud.md) service metrics and performance as part of [our shared responsibility](https://www.elastic.co/cloud/shared-responsibility). We provide service availability information on our [service status page](/deploy-manage/cloud-organization/service-status.md).
+:::
 
-% * [*Elasticsearch application logging*](../../../deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md)
+## Logging
 
+You can configure several types of logs in {{stack}} that can help you to gain insight into {{stack}} operations, diagnose issues, and track certain types of events. [Learn about the types of logs available, where to find them, and how to configure them](/deploy-manage/monitor/logging-configuration.md).

deploy-manage/monitor/kibana-task-manager-health-monitoring.md

@@ -3,13 +3,9 @@ navigation_title: "Kibana task manager monitoring"
 mapped_pages:
   - https://www.elastic.co/guide/en/kibana/current/task-manager-health-monitoring.html
 applies_to:
-  deployment:
-    self: preview
+  stack: preview
 ---
 
-
-
-
 # Kibana task manager health monitoring [task-manager-health-monitoring]
 
 
@@ -18,7 +14,7 @@ This functionality is in technical preview and may be changed or removed in a fu
 ::::
 
 
-The Task Manager has an internal monitoring mechanism to keep track of a variety of metrics, which can be consumed with either the health monitoring API or the {{kib}} server log.
+The {{kib}} [Task Manager](/deploy-manage/distributed-architecture/kibana-tasks-management.md) has an internal monitoring mechanism to keep track of a variety of metrics, which can be consumed with either the health monitoring API or the {{kib}} server log.
 
 The health monitoring API provides a reliable endpoint that can be monitored. Consuming this endpoint doesn’t cause additional load, but rather returns the latest health checks made by the system. This design enables consumption by external monitoring services at a regular cadence without additional load to the system.
 
@@ -59,13 +55,19 @@ xpack.task_manager.monitored_task_execution_thresholds:
 
 ## Consuming health stats [task-manager-consuming-health-stats]
 
-The health API is best consumed by via the `/api/task_manager/_health` endpoint.
+The health API is best consumed using the `/api/task_manager/_health` endpoint.
 
 Additionally, there are two ways to consume these metrics:
 
-**Debug logging**
+### Debug logging
+```{applies_to}
+deployment:
+  self:
+  ece:
+  eck:
+```
 
-The metrics are logged in the {{kib}} `DEBUG` logger at a regular cadence. To enable Task Manager debug logging in your {{kib}} instance, add the following to your `kibana.yml`:
+In self-managed deployments, you can configure health stats to be logged in the {{kib}} `DEBUG` logger at a regular cadence. To enable Task Manager debug logging in your {{kib}} instance, add the following to your `kibana.yml`:
 
 ```yaml
 logging:
@@ -77,7 +79,7 @@ logging:
 
 These stats are logged based on the number of milliseconds set in your [`xpack.task_manager.poll_interval`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which could add substantial noise to your logs. Only enable this level of logging temporarily.
 
-**Automatic logging**
+### Automatic logging
 
 By default, the health API runs at a regular cadence, and each time it runs, it attempts to self evaluate its performance. If this self evaluation yields a potential problem, a message will log to the {{kib}} server log. In addition, the health API will look at how long tasks have waited to start (from when they were scheduled to start). If this number exceeds a configurable threshold ([`xpack.task_manager.monitored_stats_health_verbose_log.warn_delayed_task_start_in_seconds`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings)), the same message as above will log to the {{kib}} server log.
 
@@ -92,9 +94,9 @@ If this message appears, set [`xpack.task_manager.monitored_stats_health_verbose
 
 ## Making sense of Task Manager health stats [making-sense-of-task-manager-health-stats]
 
-The health monitoring API exposes three sections: `configuration`, `workload` and `runtime`:
+The health monitoring API exposes the following sections:
 
-|     |     |
+| Section | Description |
 | --- | --- |
 | Configuration | This section summarizes the current configuration of Task Manager.  This includes dynamic configurations that change over time, such as `poll_interval` and `max_workers`, which can adjust in reaction to changing load on the system. |
 | Workload | This section summarizes the work load across the cluster, including the tasks in the system, their types, and current status. |

deploy-manage/monitor/logging-configuration.md

@@ -6,49 +6,120 @@ applies_to:
     eck: all
     self: all
 ---
-# Logging configuration
+# Logging
 
-% What needs to be done: Write from scratch
+You can configure several types of logs in {{stack}} that can help you to gain insight into {{stack}} operations, diagnose issues, and track certain types of events.
 
-% GitHub issue: https://github.com/elastic/docs-projects/issues/350
+The following logging features are available: 
 
-⚠️ **This page is a work in progress.** ⚠️
+## For {{es}} [extra-logging-features-elasticsearch]
 
+* **Application and component logging**: Logs messages related to running {{es}}. 
+
+  You can [configure the log level for {{es}}](/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md), and, in self-managed clusters, [configure underlying Log4j settings](/deploy-manage/monitor/logging-configuration/elasticsearch-log4j-configuration-self-managed.md) to customize logging behavior.
+* [Deprecation logging](/deploy-manage/monitor/logging-configuration/elasticsearch-deprecation-logs.md): Deprecation logs record a message to the {{es}} log directory when you use deprecated {{es}} functionality. You can use the deprecation logs to update your application before upgrading {{es}} to a new major version.
+* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md): Logs security-related events on your deployment.
+* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md): Helps find and debug slow queries and indexing.
 
-## Logging features [ECE/ECH] [extra-logging-features]
+## For {{kib}} [extra-logging-features-kibana]
 
-When shipping logs to a monitoring deployment there are more logging features available to you. These features include:
+* **Application and component logging**: Logs messages related to running {{kib}}. 
+
+  You can [configure the log level for {{kib}}](/deploy-manage/monitor/logging-configuration/kibana-log-levels.md), and, in self-managed, ECE, or ECK deployments, [configure advanced settings](/deploy-manage/monitor/logging-configuration/kib-advanced-logging.md) to customize logging behavior.
 
+* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md): Logs security-related events on your deployment.
 
-### For {{es}} [extra-logging-features-elasticsearch]
+## Access {{kib}} and {{es}} logs
 
-* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment
-* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing
-* Verbose logging - helps debug stack issues by increasing component logs
+The way that you access your logs differs depending on your deployment method.
 
-After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings.md) to enable these features.
+### Orchestrated deployments
 
+Access your logs using one of the following options: 
 
-### For {{kib}} [extra-logging-features-kibana]
+* All orchestrated deployments: [](/deploy-manage/monitor/stack-monitoring.md)
+* {{ech}}: [Preconfigured logs and metrics](/deploy-manage/monitor/cloud-health-perf.md#ec-es-health-preconfigured)
+* {{ece}}: [Platform monitoring](/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md)
 
-* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment
+### Self-managed deployments
 
-After you’ve enabled log delivery on your deployment, you can [add the {{kib}} user settings](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings.md) to enable this feature.
+#### {{kib}}
 
+If you run {{kib}} as a service, the default location of the logs varies based on your platform and installation method:
 
-### Other components [extra-logging-features-enterprise-search]
+:::::::{tab-set}
 
-Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments:
+::::::{tab-item} Docker
+On [Docker](/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md), log messages go to the console and are handled by the configured Docker logging driver. To access logs, run `docker logs`.
+::::::
 
-**APM**
+::::::{tab-item} Debian (APT) and RPM
+For [Debian](/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package.md) and [RPM](/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm.md) installations, {{es}} writes logs to `/var/log/kibana`.
+::::::
+
+::::::{tab-item} macOS and Linux
+For [macOS and Linux `.tar.gz`](/deploy-manage/deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md) installations, {{es}} writes logs to `$KIBANA_HOME/logs`.
+
+Files in `$KIBANA_HOME` risk deletion during an upgrade. In production, you should configure a [different location for your logs](/deploy-manage/monitor/logging-configuration/kib-advanced-logging.md).
+::::::
+
+::::::{tab-item} Windows .zip
+For [Windows `.zip`](/deploy-manage/deploy/self-managed/install-elasticsearch-with-zip-on-windows.md) installations, {{es}} writes logs to `%KIBANA_HOME%\logs`.
+
+Files in `%KIBANA_HOME%` risk deletion during an upgrade. In production, you should configure a [different location for your logs](/deploy-manage/monitor/logging-configuration/kib-advanced-logging.md).
+::::::
+
+:::::::
+
+If you run {{kib}} from the command line, {{kib}} prints logs to the standard output (`stdout`).
+
+You can also consume logs using [stack monitoring](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-self-managed.md).
+
+#### {{es}}
+
+If you run {{es}} as a service, the default location of the logs varies based on your platform and installation method:
+
+:::::::{tab-set}
+
+::::::{tab-item} Docker
+On [Docker](/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md), log messages go to the console and are handled by the configured Docker logging driver. To access logs, run `docker logs`.
+::::::
+
+::::::{tab-item} Debian (APT) and RPM
+For [Debian](/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package.md) and [RPM](/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm.md) installations, {{es}} writes logs to `/var/log/elasticsearch`.
+::::::
+
+::::::{tab-item} macOS and Linux
+For [macOS and Linux `.tar.gz`](/deploy-manage/deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md) installations, {{es}} writes logs to `$ES_HOME/logs`.
+
+Files in `$ES_HOME` risk deletion during an upgrade. In production, we strongly recommend you set `path.logs` to a location outside of `$ES_HOME`. See [Path settings](/deploy-manage/deploy/self-managed/important-settings-configuration.md#path-settings).
+::::::
+
+::::::{tab-item} Windows .zip
+For [Windows `.zip`](/deploy-manage/deploy/self-managed/install-elasticsearch-with-zip-on-windows.md) installations, {{es}} writes logs to `%ES_HOME%\logs`.
+
+Files in `%ES_HOME%` risk deletion during an upgrade. In production, we strongly recommend you set `path.logs` to a location outside of `%ES_HOME%``. See [Path settings](/deploy-manage/deploy/self-managed/important-settings-configuration.md#path-settings).
+::::::
+
+:::::::
+
+If you run {{es}} from the command line, {{es}} prints logs to the standard output (`stdout`).
+
+You can also consume logs using [stack monitoring](/deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md).
+
+## Other components [extra-logging-features-enterprise-search]
+
+You can also collect and index the following types of logs from other components in your deployments:
+
+[**APM**](/solutions/observability/apps/configure-logging.md)
 
 * `apm*.log*`
 
-**Fleet and Elastic Agent**
+[**Fleet and Elastic Agent**](/reference/ingestion-tools/fleet/monitor-elastic-agent.md)
 
 * `fleet-server-json.log-*`
 * `elastic-agent-json.log-*`
 
 The `*` indicates that we also index the archived files of each type of log.
 
-Check the respective product documentation for more information about the logging capabilities of each product.
+In {{ech}} and {{ece}}, these types of logs are automatically ingested when [stack monitoring](/deploy-manage/monitor/stack-monitoring.md) is enabled.