Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ For more information, see [{{dfeeds-cap}}](ml-ad-run-jobs.md#ml-ad-datafeeds), [

::::

If you want to see all of the configuration details for your jobs and {{dfeeds}}, you can do so on the **Machine Learning** > **Anomaly Detection** > **Jobs** page. Alternatively, you can see the configuration files in [GitHub ](https://github.com/elastic/kibana/tree/master/x-pack/plugins/ml/server/models/data_recognizer/modules/sample_data_weblogs). For the purposes of this tutorial, however, here’s a quick overview of the goal of each job:
If you want to see all of the configuration details for your jobs and {{dfeeds}}, you can do so on the **Machine Learning** > **Anomaly Detection** > **Jobs** page. Alternatively, you can see the configuration files in [GitHub ](https://github.com/elastic/kibana/tree/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/sample_data_weblogs). For the purposes of this tutorial, however, here’s a quick overview of the goal of each job:

* `low_request_rate` uses the `low_count` function to find unusually low request rates
* `response_code_rates` uses the `count` function and partitions the analysis by `response.keyword` values to find unusual event rates by HTTP response code
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,15 @@ For more details, see the {{dfeed}} and job definitions in [GitHub](https://gith

These legacy {{anomaly-jobs}} find unusual activity in HTTP access logs. For the latest versions, install the Apache integration in {{fleet}}; see [Apache access logs](ootb-ml-jobs-apache.md#apache-access-logs).

For more details, see the {{dfeed}} and job definitions in [GitHub](https://github.com/elastic/kibana/tree/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml).
For more details, see the {{dfeed}} and job definitions in [GitHub](https://github.com/elastic/kibana/tree/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml).

These configurations are only available if data exists that matches the recognizer query specified in the [manifest file](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/manifest.json#L8).
These configurations are only available if data exists that matches the recognizer query specified in the [manifest file](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/manifest.json#L8).

| Name | Description | Job (JSON) | Datafeed |
| --- | --- | --- | --- |
| low_request_rate_ecs | Detects low request rates (ECS). | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml/low_request_rate_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml/datafeed_low_request_rate_ecs.json) |
| source_ip_request_rate_ecs | Detects unusual source IPs - high request rates (ECS). | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml/source_ip_request_rate_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml/datafeed_source_ip_request_rate_ecs.json) |
| source_ip_url_count_ecs | Detect unusual source IPs - high distinct count of URLs (ECS). | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml/source_ip_url_count_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml/datafeed_source_ip_url_count_ecs.json) |
| status_code_rate_ecs | Detects unusual status code rates (ECS). | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml/status_code_rate_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml/datafeed_status_code_rate_ecs.json) |
| visitor_rate_ecs | Detects unusual visitor rates (ECS). | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml/visitor_rate_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apache_ecs/ml/datafeed_visitor_rate_ecs.json) |
| low_request_rate_ecs | Detects low request rates (ECS). | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml/low_request_rate_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml/datafeed_low_request_rate_ecs.json) |
| source_ip_request_rate_ecs | Detects unusual source IPs - high request rates (ECS). | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml/source_ip_request_rate_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml/datafeed_source_ip_request_rate_ecs.json) |
| source_ip_url_count_ecs | Detect unusual source IPs - high distinct count of URLs (ECS). | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml/source_ip_url_count_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml/datafeed_source_ip_url_count_ecs.json) |
| status_code_rate_ecs | Detects unusual status code rates (ECS). | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml/status_code_rate_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml/datafeed_status_code_rate_ecs.json) |
| visitor_rate_ecs | Detects unusual visitor rates (ECS). | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml/visitor_rate_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml/datafeed_visitor_rate_ecs.json) |

Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ mapped_pages:

# APM {{anomaly-detect}} configurations [ootb-ml-jobs-apm]

This {{anomaly-job}} appears in the {{apm-app}} and the {{ml-app}} app when you have data from APM Agents or an APM Server in your cluster. It is available only if data exists that matches the query specified in the [manifest file](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/apm_transaction/manifest.json).
This {{anomaly-job}} appears in the {{apm-app}} and the {{ml-app}} app when you have data from APM Agents or an APM Server in your cluster. It is available only if data exists that matches the query specified in the [manifest file](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apm_transaction/manifest.json).

For more information about {{anomaly-detect}} in the {{apm-app}}, refer to [{{ml-cap}} integration](/solutions/observability/apps/integrate-with-machine-learning.md).

Expand All @@ -14,5 +14,5 @@ For more information about {{anomaly-detect}} in the {{apm-app}}, refer to [{{ml

| Name | Description | Job (JSON) | Datafeed |
| --- | --- | --- | --- |
| apm_tx_metrics | Detects anomalies in transaction latency, throughput and error percentage for metric data. | [code](https://github.com/elastic/kibana/blob/main/x-pack/plugins/ml/server/models/data_recognizer/modules/apm_transaction/ml/apm_tx_metrics.json) | [code](https://github.com/elastic/kibana/blob/main/x-pack/plugins/ml/server/models/data_recognizer/modules/apm_transaction/ml/datafeed_apm_tx_metrics.json) |
| apm_tx_metrics | Detects anomalies in transaction latency, throughput and error percentage for metric data. | [code](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apm_transaction/ml/apm_tx_metrics.json) | [code](https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apm_transaction/ml/datafeed_apm_tx_metrics.json) |

Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ These {{anomaly-job}} wizards appear in {{kib}} if you use [{{auditbeat}}](beats

Detect unusual processes in docker containers from auditd data (ECS).

These configurations are only available if data exists that matches the recognizer query specified in the [manifest file](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/auditbeat_process_docker_ecs/manifest.json#L8).
These configurations are only available if data exists that matches the recognizer query specified in the [manifest file](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/auditbeat_process_docker_ecs/manifest.json#L8).

| Name | Description | Job (JSON)| Datafeed |
| --- | --- | --- | --- |
| docker_high_count_process_events_ecs | Detect unusual increases in process execution rates in docker containers (ECS) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/auditbeat_process_docker_ecs/ml/docker_high_count_process_events_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/auditbeat_process_docker_ecs/ml/datafeed_docker_high_count_process_events_ecs.json) |
| docker_rare_process_activity_ecs | Detect rare process executions in docker containers (ECS) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/auditbeat_process_docker_ecs/ml/docker_rare_process_activity_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/auditbeat_process_docker_ecs/ml/datafeed_docker_rare_process_activity_ecs.json) |
| docker_high_count_process_events_ecs | Detect unusual increases in process execution rates in docker containers (ECS) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/auditbeat_process_docker_ecs/ml/docker_high_count_process_events_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/auditbeat_process_docker_ecs/ml/datafeed_docker_high_count_process_events_ecs.json) |
| docker_rare_process_activity_ecs | Detect rare process executions in docker containers (ECS) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/auditbeat_process_docker_ecs/ml/docker_rare_process_activity_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/auditbeat_process_docker_ecs/ml/datafeed_docker_rare_process_activity_ecs.json) |

Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Detect anomalies in log entries via the Logs UI.

| Name | Description | Job (JSON) | Datafeed |
| --- | --- | --- | --- |
| log_entry_rate | Detects anomalies in the log entry ingestion rate | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/logs_ui_analysis/ml/log_entry_rate.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/logs_ui_analysis/ml/datafeed_log_entry_rate.json) |
| log_entry_rate | Detects anomalies in the log entry ingestion rate | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/logs_ui_analysis/ml/log_entry_rate.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/logs_ui_analysis/ml/datafeed_log_entry_rate.json) |


## Log entry categories [logs-ui-categories]
Expand All @@ -23,5 +23,5 @@ Detect anomalies in count of log entries by category.

| Name | Description | Job (JSON) | Datafeed |
| --- | --- | --- | --- |
| log_entry_categories_count | Detects anomalies in count of log entries by category | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/logs_ui_categories/ml/log_entry_categories_count.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/logs_ui_categories/ml/datafeed_log_entry_categories_count.json) |
| log_entry_categories_count | Detects anomalies in count of log entries by category | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/logs_ui_categories/ml/log_entry_categories_count.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/logs_ui_categories/ml/datafeed_log_entry_categories_count.json) |

Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,11 @@ These {{anomaly-job}} wizards appear in {{kib}} if you use the [{{metricbeat}} s

Detect anomalies in {{metricbeat}} System data (ECS).

These configurations are only available if data exists that matches the recognizer query specified in the [manifest file](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/manifest.json#L8).
These configurations are only available if data exists that matches the recognizer query specified in the [manifest file](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/manifest.json#L8).

| Name | Description | Job (JSON) | Datafeed |
| --- | --- | --- | --- |
| high_mean_cpu_iowait_ecs | Detect unusual increases in cpu time spent in iowait (ECS) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/high_mean_cpu_iowait_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/datafeed_high_mean_cpu_iowait_ecs.json) |
| max_disk_utilization_ecs | Detect unusual increases in disk utilization (ECS) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/max_disk_utilization_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/datafeed_max_disk_utilization_ecs.json) |
| metricbeat_outages_ecs | Detect unusual decreases in metricbeat documents (ECS) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/metricbeat_outages_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/datafeed_metricbeat_outages_ecs.json) |
| high_mean_cpu_iowait_ecs | Detect unusual increases in cpu time spent in iowait (ECS) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/high_mean_cpu_iowait_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/datafeed_high_mean_cpu_iowait_ecs.json) |
| max_disk_utilization_ecs | Detect unusual increases in disk utilization (ECS) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/max_disk_utilization_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/datafeed_max_disk_utilization_ecs.json) |
| metricbeat_outages_ecs | Detect unusual decreases in metricbeat documents (ECS) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/metricbeat_outages_ecs.json) | [code](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/metricbeat_system_ecs/ml/datafeed_metricbeat_outages_ecs.json) |

Loading
Loading