You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: deploy-manage/monitor.md
+17-4Lines changed: 17 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -74,18 +74,31 @@ Out of the box logs and metrics tools, including ECH preconfigured logs and metr
74
74
75
75
To learn more about the health and performance tools in {{ecloud}}, refer to [](/deploy-manage/monitor/cloud-health-perf.md).
76
76
77
+
## {{kib}} task manager monitoring
78
+
79
+
```{applies_to}
80
+
stack: preview
81
+
```
82
+
The {{kib}} [task manager](/deploy-manage/distributed-architecture/kibana-tasks-management.md) has an internal monitoring mechanism to keep track of a variety of metrics, which can be consumed with either the health monitoring API or the {{kib}} server log. [Learn how to configure thresholds and consume related to {{kib}} task manager](/deploy-manage/monitor/kibana-task-manager-health-monitoring.md).
83
+
77
84
## Monitoring your orchestrator
78
85
```{applies_to}
79
86
deployment:
80
87
ece:
81
88
eck:
82
89
```
83
90
84
-
TODO
91
+
In addition to monitoring your cluster or deployment health and performance, you need to monitor your orchestrator. Monitoring is especially important for orchestrators hosted on infrastructure that you control.
85
92
86
-
## Logging
93
+
Learn how to enable monitoring of your orchestrator:
94
+
95
+
*[ECK operator metrics](/deploy-manage/monitor/orchestrators/eck-metrics-configuration.md): Open and secure a metrics endpoint that can be used to monitor the operator’s performance and health. This endpoint can be scraped by third-party Kubernetes monitoring tools.
96
+
*[ECK platform monitoring](/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md): Learn about how ECE collects monitoring data for your installation in the `logging-and-metrics` deployment, and how to access monitoring data.
87
97
88
-
TODO
98
+
:::{admonition} Monitoring {{ecloud}}
99
+
Elastic monitors [{{ecloud}}](/deploy-manage/deploy/elastic-cloud.md) service metrics and performance as part of [our shared responsibility](https://www.elastic.co/cloud/shared-responsibility). We provide service availability information on our [service status page](/deploy-manage/cloud-organization/service-status.md).
You can configure several types of logs in {{stack}} that can help you to gain insight into {{stack}} operations, diagnose issues, and track certain types of events. [Learn about the types of logs available, where to find them, and how to configure them](/deploy-manage/monitor/logging-configuration.md).
# Kibana task manager health monitoring [task-manager-health-monitoring]
14
10
15
11
@@ -18,7 +14,7 @@ This functionality is in technical preview and may be changed or removed in a fu
18
14
::::
19
15
20
16
21
-
The Task Manager has an internal monitoring mechanism to keep track of a variety of metrics, which can be consumed with either the health monitoring API or the {{kib}} server log.
17
+
The {{kib}} [Task Manager](/deploy-manage/distributed-architecture/kibana-tasks-management.md) has an internal monitoring mechanism to keep track of a variety of metrics, which can be consumed with either the health monitoring API or the {{kib}} server log.
22
18
23
19
The health monitoring API provides a reliable endpoint that can be monitored. Consuming this endpoint doesn’t cause additional load, but rather returns the latest health checks made by the system. This design enables consumption by external monitoring services at a regular cadence without additional load to the system.
## Consuming health stats [task-manager-consuming-health-stats]
61
57
62
-
The health API is best consumed by via the `/api/task_manager/_health` endpoint.
58
+
The health API is best consumed using the `/api/task_manager/_health` endpoint.
63
59
64
60
Additionally, there are two ways to consume these metrics:
65
61
66
-
**Debug logging**
62
+
### Debug logging
63
+
```{applies_to}
64
+
deployment:
65
+
self:
66
+
ece:
67
+
eck:
68
+
```
67
69
68
-
The metrics are logged in the {{kib}} `DEBUG` logger at a regular cadence. To enable Task Manager debug logging in your {{kib}} instance, add the following to your `kibana.yml`:
70
+
In self-managed deployments, you can configure health stats to be logged in the {{kib}} `DEBUG` logger at a regular cadence. To enable Task Manager debug logging in your {{kib}} instance, add the following to your `kibana.yml`:
69
71
70
72
```yaml
71
73
logging:
@@ -77,7 +79,7 @@ logging:
77
79
78
80
These stats are logged based on the number of milliseconds set in your [`xpack.task_manager.poll_interval`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings) setting, which could add substantial noise to your logs. Only enable this level of logging temporarily.
79
81
80
-
**Automatic logging**
82
+
### Automatic logging
81
83
82
84
By default, the health API runs at a regular cadence, and each time it runs, it attempts to self evaluate its performance. If this self evaluation yields a potential problem, a message will log to the {{kib}} server log. In addition, the health API will look at how long tasks have waited to start (from when they were scheduled to start). If this number exceeds a configurable threshold ([`xpack.task_manager.monitored_stats_health_verbose_log.warn_delayed_task_start_in_seconds`](kibana://reference/configuration-reference/task-manager-settings.md#task-manager-settings)), the same message as above will log to the {{kib}} server log.
83
85
@@ -92,9 +94,9 @@ If this message appears, set [`xpack.task_manager.monitored_stats_health_verbose
92
94
93
95
## Making sense of Task Manager health stats [making-sense-of-task-manager-health-stats]
94
96
95
-
The health monitoring API exposes three sections: `configuration`, `workload` and `runtime`:
97
+
The health monitoring API exposes the following sections:
96
98
97
-
|||
99
+
|Section | Description|
98
100
| --- | --- |
99
101
| Configuration | This section summarizes the current configuration of Task Manager. This includes dynamic configurations that change over time, such as `poll_interval` and `max_workers`, which can adjust in reaction to changing load on the system. |
100
102
| Workload | This section summarizes the work load across the cluster, including the tasks in the system, their types, and current status. |
Copy file name to clipboardExpand all lines: deploy-manage/monitor/logging-configuration.md
+90-19Lines changed: 90 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,49 +6,120 @@ applies_to:
6
6
eck: all
7
7
self: all
8
8
---
9
-
# Logging configuration
9
+
# Logging
10
10
11
-
% What needs to be done: Write from scratch
11
+
You can configure several types of logs in {{stack}} that can help you to gain insight into {{stack}} operations, diagnose issues, and track certain types of events.
## For {{es}} [extra-logging-features-elasticsearch]
16
16
17
+
***Application and component logging**: Logs messages related to running {{es}}.
18
+
19
+
You can [configure the log level for {{es}}](/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md), and, in self-managed clusters, [configure underlying Log4j settings](/deploy-manage/monitor/logging-configuration/elasticsearch-log4j-configuration-self-managed.md) to customize logging behavior.
20
+
*[Deprecation logging](/deploy-manage/monitor/logging-configuration/elasticsearch-deprecation-logs.md): Deprecation logs record a message to the {{es}} log directory when you use deprecated {{es}} functionality. You can use the deprecation logs to update your application before upgrading {{es}} to a new major version.
21
+
*[Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md): Logs security-related events on your deployment.
22
+
*[Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md): Helps find and debug slow queries and indexing.
17
23
18
-
## Logging features [ECE/ECH][extra-logging-features]
24
+
## For {{kib}} [extra-logging-features-kibana]
19
25
20
-
When shipping logs to a monitoring deployment there are more logging features available to you. These features include:
26
+
***Application and component logging**: Logs messages related to running {{kib}}.
27
+
28
+
You can [configure the log level for {{kib}}](/deploy-manage/monitor/logging-configuration/kibana-log-levels.md), and, in self-managed, ECE, or ECK deployments, [configure advanced settings](/deploy-manage/monitor/logging-configuration/kib-advanced-logging.md) to customize logging behavior.
21
29
30
+
*[Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md): Logs security-related events on your deployment.
22
31
23
-
### For {{es}} [extra-logging-features-elasticsearch]
32
+
##Access {{kib}} and {{es}} logs
24
33
25
-
*[Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment
26
-
*[Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing
The way that you access your logs differs depending on your deployment method.
28
35
29
-
After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings.md) to enable these features.
36
+
### Orchestrated deployments
30
37
38
+
Access your logs using one of the following options:
31
39
32
-
### For {{kib}} [extra-logging-features-kibana]
40
+
* All orchestrated deployments: [](/deploy-manage/monitor/stack-monitoring.md)
41
+
* {{ech}}: [Preconfigured logs and metrics](/deploy-manage/monitor/cloud-health-perf.md#ec-es-health-preconfigured)
*[Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment
44
+
### Self-managed deployments
35
45
36
-
After you’ve enabled log delivery on your deployment, you can [add the {{kib}} user settings](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings.md) to enable this feature.
46
+
#### {{kib}}
37
47
48
+
If you run {{kib}} as a service, the default location of the logs varies based on your platform and installation method:
38
49
39
-
### Other components [extra-logging-features-enterprise-search]
50
+
:::::::{tab-set}
40
51
41
-
Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments:
52
+
::::::{tab-item} Docker
53
+
On [Docker](/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md), log messages go to the console and are handled by the configured Docker logging driver. To access logs, run `docker logs`.
54
+
::::::
42
55
43
-
**APM**
56
+
::::::{tab-item} Debian (APT) and RPM
57
+
For [Debian](/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package.md) and [RPM](/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm.md) installations, {{es}} writes logs to `/var/log/kibana`.
58
+
::::::
59
+
60
+
::::::{tab-item} macOS and Linux
61
+
For [macOS and Linux `.tar.gz`](/deploy-manage/deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md) installations, {{es}} writes logs to `$KIBANA_HOME/logs`.
62
+
63
+
Files in `$KIBANA_HOME` risk deletion during an upgrade. In production, you should configure a [different location for your logs](/deploy-manage/monitor/logging-configuration/kib-advanced-logging.md).
64
+
::::::
65
+
66
+
::::::{tab-item} Windows .zip
67
+
For [Windows `.zip`](/deploy-manage/deploy/self-managed/install-elasticsearch-with-zip-on-windows.md) installations, {{es}} writes logs to `%KIBANA_HOME%\logs`.
68
+
69
+
Files in `%KIBANA_HOME%` risk deletion during an upgrade. In production, you should configure a [different location for your logs](/deploy-manage/monitor/logging-configuration/kib-advanced-logging.md).
70
+
::::::
71
+
72
+
:::::::
73
+
74
+
If you run {{kib}} from the command line, {{kib}} prints logs to the standard output (`stdout`).
75
+
76
+
You can also consume logs using [stack monitoring](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-self-managed.md).
77
+
78
+
#### {{es}}
79
+
80
+
If you run {{es}} as a service, the default location of the logs varies based on your platform and installation method:
81
+
82
+
:::::::{tab-set}
83
+
84
+
::::::{tab-item} Docker
85
+
On [Docker](/deploy-manage/deploy/self-managed/install-elasticsearch-with-docker.md), log messages go to the console and are handled by the configured Docker logging driver. To access logs, run `docker logs`.
86
+
::::::
87
+
88
+
::::::{tab-item} Debian (APT) and RPM
89
+
For [Debian](/deploy-manage/deploy/self-managed/install-elasticsearch-with-debian-package.md) and [RPM](/deploy-manage/deploy/self-managed/install-elasticsearch-with-rpm.md) installations, {{es}} writes logs to `/var/log/elasticsearch`.
90
+
::::::
91
+
92
+
::::::{tab-item} macOS and Linux
93
+
For [macOS and Linux `.tar.gz`](/deploy-manage/deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md) installations, {{es}} writes logs to `$ES_HOME/logs`.
94
+
95
+
Files in `$ES_HOME` risk deletion during an upgrade. In production, we strongly recommend you set `path.logs` to a location outside of `$ES_HOME`. See [Path settings](/deploy-manage/deploy/self-managed/important-settings-configuration.md#path-settings).
96
+
::::::
97
+
98
+
::::::{tab-item} Windows .zip
99
+
For [Windows `.zip`](/deploy-manage/deploy/self-managed/install-elasticsearch-with-zip-on-windows.md) installations, {{es}} writes logs to `%ES_HOME%\logs`.
100
+
101
+
Files in `%ES_HOME%` risk deletion during an upgrade. In production, we strongly recommend you set `path.logs` to a location outside of `%ES_HOME%``. See [Path settings](/deploy-manage/deploy/self-managed/important-settings-configuration.md#path-settings).
102
+
::::::
103
+
104
+
:::::::
105
+
106
+
If you run {{es}} from the command line, {{es}} prints logs to the standard output (`stdout`).
107
+
108
+
You can also consume logs using [stack monitoring](/deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md).
109
+
110
+
## Other components [extra-logging-features-enterprise-search]
111
+
112
+
You can also collect and index the following types of logs from other components in your deployments:
0 commit comments