Skip to content

Commit 8151cb0

Browse files
authored
[D+M] Monitoring: orchestrators, kib task manager, logging config (#833)
1 parent eb1c531 commit 8151cb0

26 files changed

+1021
-1432
lines changed

deploy-manage/maintenance/start-stop-services/start-stop-elasticsearch.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
22
mapped_urls:
3-
- https://www.elastic.co/guide/en/{{es}}/reference/current/starting-elasticsearch.html
4-
- https://www.elastic.co/guide/en/{{es}}/reference/current/stopping-elasticsearch.html
3+
- https://www.elastic.co/guide/en/elasticsearch/reference/current/starting-elasticsearch.html
4+
- https://www.elastic.co/guide/en/elasticsearch/reference/current/stopping-elasticsearch.html
55
applies_to:
66
deployment:
77
self:

deploy-manage/monitor.md

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -74,18 +74,31 @@ Out of the box logs and metrics tools, including ECH preconfigured logs and metr
7474

7575
To learn more about the health and performance tools in {{ecloud}}, refer to [](/deploy-manage/monitor/cloud-health-perf.md).
7676

77+
## {{kib}} task manager monitoring
78+
79+
```{applies_to}
80+
stack: preview
81+
```
82+
The {{kib}} [task manager](/deploy-manage/distributed-architecture/kibana-tasks-management.md) has an internal monitoring mechanism to keep track of a variety of metrics, which can be consumed with either the health monitoring API or the {{kib}} server log. [Learn how to configure thresholds and consume related to {{kib}} task manager](/deploy-manage/monitor/kibana-task-manager-health-monitoring.md).
83+
7784
## Monitoring your orchestrator
7885
```{applies_to}
7986
deployment:
8087
ece:
8188
eck:
8289
```
8390

84-
TODO
91+
In addition to monitoring your cluster or deployment health and performance, you need to monitor your orchestrator. Monitoring is especially important for orchestrators hosted on infrastructure that you control.
8592

86-
## Logging
93+
Learn how to enable monitoring of your orchestrator:
94+
95+
* [ECK operator metrics](/deploy-manage/monitor/orchestrators/eck-metrics-configuration.md): Open and secure a metrics endpoint that can be used to monitor the operator’s performance and health. This endpoint can be scraped by third-party Kubernetes monitoring tools.
96+
* [ECK platform monitoring](/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md): Learn about how ECE collects monitoring data for your installation in the `logging-and-metrics` deployment, and how to access monitoring data.
8797

88-
TODO
98+
:::{admonition} Monitoring {{ecloud}}
99+
Elastic monitors [{{ecloud}}](/deploy-manage/deploy/elastic-cloud.md) service metrics and performance as part of [our shared responsibility](https://www.elastic.co/cloud/shared-responsibility). We provide service availability information on our [service status page](/deploy-manage/cloud-organization/service-status.md).
100+
:::
89101

90-
% * [*Elasticsearch application logging*](../../../deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md)
102+
## Logging
91103

104+
You can configure several types of logs in {{stack}} that can help you to gain insight into {{stack}} operations, diagnose issues, and track certain types of events. [Learn about the types of logs available, where to find them, and how to configure them](/deploy-manage/monitor/logging-configuration.md).

deploy-manage/monitor/kibana-task-manager-health-monitoring.md

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,9 @@ navigation_title: "Kibana task manager monitoring"
33
mapped_pages:
44
- https://www.elastic.co/guide/en/kibana/current/task-manager-health-monitoring.html
55
applies_to:
6-
deployment:
7-
self: preview
6+
stack: preview
87
---
98

10-
11-
12-
139
# Kibana task manager health monitoring [task-manager-health-monitoring]
1410

1511

@@ -18,7 +14,7 @@ This functionality is in technical preview and may be changed or removed in a fu
1814
::::
1915

2016

21-
The Task Manager has an internal monitoring mechanism to keep track of a variety of metrics, which can be consumed with either the health monitoring API or the {{kib}} server log.
17+
The {{kib}} [Task Manager](/deploy-manage/distributed-architecture/kibana-tasks-management.md) has an internal monitoring mechanism to keep track of a variety of metrics, which can be consumed with either the health monitoring API or the {{kib}} server log.
2218

2319
The health monitoring API provides a reliable endpoint that can be monitored. Consuming this endpoint doesn’t cause additional load, but rather returns the latest health checks made by the system. This design enables consumption by external monitoring services at a regular cadence without additional load to the system.
2420

@@ -59,13 +55,19 @@ xpack.task_manager.monitored_task_execution_thresholds:
5955

6056
## Consuming health stats [task-manager-consuming-health-stats]
6157

62-
The health API is best consumed by via the `/api/task_manager/_health` endpoint.
58+
The health API is best consumed using the `/api/task_manager/_health` endpoint.
6359

6460
Additionally, there are two ways to consume these metrics:
6561

66-
**Debug logging**
62+
### Debug logging
63+
```{applies_to}
64+
deployment:
65+
self:
66+
ece:
67+
eck:
68+
```
6769

68-
The metrics are logged in the {{kib}} `DEBUG` logger at a regular cadence. To enable Task Manager debug logging in your {{kib}} instance, add the following to your `kibana.yml`:
70+
In self-managed deployments, you can configure health stats to be logged in the {{kib}} `DEBUG` logger at a regular cadence. To enable Task Manager debug logging in your {{kib}} instance, add the following to your `kibana.yml`:
6971

7072
```yaml
7173
logging:
@@ -77,7 +79,7 @@ logging:
7779

7880
These stats are logged based on the number of milliseconds set in your [`xpack.task_manager.poll_interval`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which could add substantial noise to your logs. Only enable this level of logging temporarily.
7981

80-
**Automatic logging**
82+
### Automatic logging
8183

8284
By default, the health API runs at a regular cadence, and each time it runs, it attempts to self evaluate its performance. If this self evaluation yields a potential problem, a message will log to the {{kib}} server log. In addition, the health API will look at how long tasks have waited to start (from when they were scheduled to start). If this number exceeds a configurable threshold ([`xpack.task_manager.monitored_stats_health_verbose_log.warn_delayed_task_start_in_seconds`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings)), the same message as above will log to the {{kib}} server log.
8385

@@ -92,9 +94,9 @@ If this message appears, set [`xpack.task_manager.monitored_stats_health_verbose
9294

9395
## Making sense of Task Manager health stats [making-sense-of-task-manager-health-stats]
9496

95-
The health monitoring API exposes three sections: `configuration`, `workload` and `runtime`:
97+
The health monitoring API exposes the following sections:
9698

97-
| | |
99+
| Section | Description |
98100
| --- | --- |
99101
| Configuration | This section summarizes the current configuration of Task Manager. This includes dynamic configurations that change over time, such as `poll_interval` and `max_workers`, which can adjust in reaction to changing load on the system. |
100102
| Workload | This section summarizes the work load across the cluster, including the tasks in the system, their types, and current status. |

deploy-manage/monitor/logging-configuration.md

Lines changed: 90 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -6,49 +6,120 @@ applies_to:
66
eck: all
77
self: all
88
---
9-
# Logging configuration
9+
# Logging
1010

11-
% What needs to be done: Write from scratch
11+
You can configure several types of logs in {{stack}} that can help you to gain insight into {{stack}} operations, diagnose issues, and track certain types of events.
1212

13-
% GitHub issue: https://github.com/elastic/docs-projects/issues/350
13+
The following logging features are available:
1414

15-
⚠️ **This page is a work in progress.** ⚠️
15+
## For {{es}} [extra-logging-features-elasticsearch]
1616

17+
* **Application and component logging**: Logs messages related to running {{es}}.
18+
19+
You can [configure the log level for {{es}}](/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md), and, in self-managed clusters, [configure underlying Log4j settings](/deploy-manage/monitor/logging-configuration/elasticsearch-log4j-configuration-self-managed.md) to customize logging behavior.
20+
* [Deprecation logging](/deploy-manage/monitor/logging-configuration/elasticsearch-deprecation-logs.md): Deprecation logs record a message to the {{es}} log directory when you use deprecated {{es}} functionality. You can use the deprecation logs to update your application before upgrading {{es}} to a new major version.
21+
* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md): Logs security-related events on your deployment.
22+
* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md): Helps find and debug slow queries and indexing.
1723

18-
## Logging features [ECE/ECH] [extra-logging-features]
24+
## For {{kib}} [extra-logging-features-kibana]
1925

20-
When shipping logs to a monitoring deployment there are more logging features available to you. These features include:
26+
* **Application and component logging**: Logs messages related to running {{kib}}.
27+
28+
You can [configure the log level for {{kib}}](/deploy-manage/monitor/logging-configuration/kibana-log-levels.md), and, in self-managed, ECE, or ECK deployments, [configure advanced settings](/deploy-manage/monitor/logging-configuration/kib-advanced-logging.md) to customize logging behavior.
2129

30+
* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md): Logs security-related events on your deployment.
2231

23-
### For {{es}} [extra-logging-features-elasticsearch]
32+
## Access {{kib}} and {{es}} logs
2433

25-
* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment
26-
* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing
27-
* Verbose logging - helps debug stack issues by increasing component logs
34+
The way that you access your logs differs depending on your deployment method.
2835

29-
After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings.md) to enable these features.
36+
### Orchestrated deployments
3037

38+
Access your logs using one of the following options:
3139

32-
### For {{kib}} [extra-logging-features-kibana]
40+
* All orchestrated deployments: [](/deploy-manage/monitor/stack-monitoring.md)
41+
* {{ech}}: [Preconfigured logs and metrics](/deploy-manage/monitor/cloud-health-perf.md#ec-es-health-preconfigured)
42+
* {{ece}}: [Platform monitoring](/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md)
3343

34-
* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment
44+
### Self-managed deployments
3545

36-
After you’ve enabled log delivery on your deployment, you can [add the {{kib}} user settings](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings.md) to enable this feature.
46+
#### {{kib}}
3747

48+
If you run {{kib}} as a service, the default location of the logs varies based on your platform and installation method:
3849

39-
### Other components [extra-logging-features-enterprise-search]
50+
:::::::{tab-set}
4051

41-
Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments:
52+
::::::{tab-item} Docker
53+
On [Docker](/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md), log messages go to the console and are handled by the configured Docker logging driver. To access logs, run `docker logs`.
54+
::::::
4255

43-
**APM**
56+
::::::{tab-item} Debian (APT) and RPM
57+
For [Debian](/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package.md) and [RPM](/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm.md) installations, {{es}} writes logs to `/var/log/kibana`.
58+
::::::
59+
60+
::::::{tab-item} macOS and Linux
61+
For [macOS and Linux `.tar.gz`](/deploy-manage/deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md) installations, {{es}} writes logs to `$KIBANA_HOME/logs`.
62+
63+
Files in `$KIBANA_HOME` risk deletion during an upgrade. In production, you should configure a [different location for your logs](/deploy-manage/monitor/logging-configuration/kib-advanced-logging.md).
64+
::::::
65+
66+
::::::{tab-item} Windows .zip
67+
For [Windows `.zip`](/deploy-manage/deploy/self-managed/install-elasticsearch-with-zip-on-windows.md) installations, {{es}} writes logs to `%KIBANA_HOME%\logs`.
68+
69+
Files in `%KIBANA_HOME%` risk deletion during an upgrade. In production, you should configure a [different location for your logs](/deploy-manage/monitor/logging-configuration/kib-advanced-logging.md).
70+
::::::
71+
72+
:::::::
73+
74+
If you run {{kib}} from the command line, {{kib}} prints logs to the standard output (`stdout`).
75+
76+
You can also consume logs using [stack monitoring](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-self-managed.md).
77+
78+
#### {{es}}
79+
80+
If you run {{es}} as a service, the default location of the logs varies based on your platform and installation method:
81+
82+
:::::::{tab-set}
83+
84+
::::::{tab-item} Docker
85+
On [Docker](/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md), log messages go to the console and are handled by the configured Docker logging driver. To access logs, run `docker logs`.
86+
::::::
87+
88+
::::::{tab-item} Debian (APT) and RPM
89+
For [Debian](/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package.md) and [RPM](/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm.md) installations, {{es}} writes logs to `/var/log/elasticsearch`.
90+
::::::
91+
92+
::::::{tab-item} macOS and Linux
93+
For [macOS and Linux `.tar.gz`](/deploy-manage/deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md) installations, {{es}} writes logs to `$ES_HOME/logs`.
94+
95+
Files in `$ES_HOME` risk deletion during an upgrade. In production, we strongly recommend you set `path.logs` to a location outside of `$ES_HOME`. See [Path settings](/deploy-manage/deploy/self-managed/important-settings-configuration.md#path-settings).
96+
::::::
97+
98+
::::::{tab-item} Windows .zip
99+
For [Windows `.zip`](/deploy-manage/deploy/self-managed/install-elasticsearch-with-zip-on-windows.md) installations, {{es}} writes logs to `%ES_HOME%\logs`.
100+
101+
Files in `%ES_HOME%` risk deletion during an upgrade. In production, we strongly recommend you set `path.logs` to a location outside of `%ES_HOME%``. See [Path settings](/deploy-manage/deploy/self-managed/important-settings-configuration.md#path-settings).
102+
::::::
103+
104+
:::::::
105+
106+
If you run {{es}} from the command line, {{es}} prints logs to the standard output (`stdout`).
107+
108+
You can also consume logs using [stack monitoring](/deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md).
109+
110+
## Other components [extra-logging-features-enterprise-search]
111+
112+
You can also collect and index the following types of logs from other components in your deployments:
113+
114+
[**APM**](/solutions/observability/apps/configure-logging.md)
44115

45116
* `apm*.log*`
46117

47-
**Fleet and Elastic Agent**
118+
[**Fleet and Elastic Agent**](/reference/ingestion-tools/fleet/monitor-elastic-agent.md)
48119

49120
* `fleet-server-json.log-*`
50121
* `elastic-agent-json.log-*`
51122

52123
The `*` indicates that we also index the archived files of each type of log.
53124

54-
Check the respective product documentation for more information about the logging capabilities of each product.
125+
In {{ech}} and {{ece}}, these types of logs are automatically ingested when [stack monitoring](/deploy-manage/monitor/stack-monitoring.md) is enabled.

0 commit comments

Comments
 (0)