Monitoring audit logging

deploy-manage/monitor/logging-configuration/auditing-search-queries.md

@@ -8,23 +8,24 @@ applies:
   stack: all
 ---
 
-# Auditing search queries [auditing-search-queries]
+# Auditing Elasticsearch search queries [auditing-search-queries]
 
 There is no [audit event type](elasticsearch-audit-events.md) specifically dedicated to search queries. Search queries are analyzed and then processed; the processing triggers authorization actions that are audited. However, the original raw query, as submitted by the client, is not accessible downstream when authorization auditing occurs.
 
 Search queries are contained inside HTTP request bodies, however, and some audit events that are generated by the REST layer, on the coordinating node, can be toggled to output the request body to the audit log. Therefore, one must audit request bodies in order to audit search queries.
 
-To make certain audit events include the request body, edit the following setting in the `elasticsearch.yml` file:
+To make certain audit events include the request body, configure the following setting in {{es}}:
 
 ```yaml
 xpack.security.audit.logfile.events.emit_request_body: true
 ```
 
+In self-managed deployments, add the to the `elasticsearch.yml` file, while in orchestrated environments, apply the setting through the appropriate [configuration mechanism](/deploy-manage/deploy.md).
+
 ::::{important} 
-No filtering is performed when auditing, so sensitive data might be audited in plain text when audit events include the request body. Also, the request body can contain malicious content that can break a parser consuming the audit logs.
+No filtering is performed when auditing, so **sensitive data might be audited in plain text when audit events include the request body**. Also, the request body can contain malicious content that can break a parser consuming the audit logs.
 ::::
 
-
 The request body is printed as an escaped JSON string value (RFC 4627) to the `request.body` event attribute.
 
 Not all events contain the `request.body` attribute, even when the above setting is toggled. The ones that do are: `authentication_success`, `authentication_failed`, `realm_authentication_failed`, `tampered_request`, `run_as_denied`, and `anonymous_access_denied`. The `request.body` attribute is printed on the coordinating node only (the node that handles the REST request). Most of these event types are [not included by default](https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html#xpack-sa-lf-events-include).

deploy-manage/monitor/logging-configuration/elasticsearch-audit-events.md

@@ -8,16 +8,19 @@ applies:
   stack: all
 ---
 
-# Elasticsearch audit events [audit-event-types]
+% Pending to move this page to REFERENCE
 
-When you are [auditing security events](enabling-elasticsearch-audit-logs.md), a single client request might generate multiple audit events, across multiple cluster nodes. The common `request.id` attribute can be used to correlate the associated events.
+# Elasticsearch audit events [elasticsearch-audit-events]
 
-Use the [`xpack.security.audit.logfile.events.include`](https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html#xpack-sa-lf-events-include) setting in `elasticsearch.yml` to specify the kind of events you want to include in the auditing output.
+When you are [auditing security events](/deploy-manage/monitor/logging-configuration/enabling-audit-logs.md), a single client request might generate multiple audit events across multiple cluster nodes. The common `request.id` attribute can be used to correlate the associated events.
+
+This document provides a reference for all audit event [types](#audit-event-types) and their associated [attributes](#audit-event-attributes) in {{es}}. Types and attributes can be used to control what gets logged through [audit event settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html#event-audit-settings) and [ignore policies](/deploy-manage/monitor/logging-configuration/logfile-audit-events-ignore-policies.md).
 
 ::::{note}
-Certain audit events require the `security_config_change` event type to audit the related event action. The description of impacted audit events indicate whether that event type is required.
+Certain audit events require the `security_config_change` event type to log the related event action. The event descriptions in this document indicate whether this requirement is applicable.
 ::::
 
+## Audit event types [audit-event-types]
 
 $$$event-access-denied$$$
 

deploy-manage/monitor/logging-configuration/enabling-audit-logs.md

@@ -0,0 +1,218 @@
+---
+mapped_pages:
+  - https://www.elastic.co/guide/en/elasticsearch/reference/current/enable-audit-logging.html
+  - https://www.elastic.co/guide/en/kibana/current/xpack-security-audit-logging.html
+  - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-enable-auditing.html
+  - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s_audit_logging.html
+  - https://www.elastic.co/guide/en/cloud/current/ec-enable-logging-and-monitoring.html#ec-enable-audit-logs
+applies:
+  hosted: all
+  ece: all
+  eck: all
+  stack: all
+  serverless: unavailable
+---
+
+# Enable audit logging [enable-audit-logging]
+
+You can log security-related events such as authentication failures and refused connections to monitor your cluster for suspicious activity (including data access authorization and user security configuration changes). Audit logging can be enabled independently for {{es}} and {{kib}}.
+
+::::{important}
+Audit logs are only available on certain [subscription levels](https://www.elastic.co/subscriptions).
+::::
+
+::::{note}
+Audit logs are **disabled** by default and must be explicitly enabled.
+::::
+
+This section describes how to enable and configure audit logging in both {{es}} and {{kib}} for all supported deployment types, including self-managed clusters, Elastic Cloud Hosted, Elastic Cloud Enterprise (ECE), and Elastic Cloud on Kubernetes (ECK).
+
+% consider removing this sentence as maybe it doesn't add any benefit
+The process of enabling and configuring audit logging is consistent across all supported deployment types, whether self-managed, Elastic Cloud, Elastic Cloud Enterprise (ECE), or Elastic Cloud on Kubernetes (ECK). The same settings apply regardless of the deployment type, ensuring a unified approach to audit logging configuration.
+
+% consider removing this sentence as maybe it doesn't add any benefit
+The only difference lies in how the configuration is applied:
+  * In self-managed clusters, settings are added directly to the `elasticsearch.yml` and `kibana.yml` configuration files, and you will have to perform a [rolling restart of the cluster](/deploy-manage/maintenance/start-stop-services/full-cluster-restart-rolling-restart-procedures.md) when applying changes to `static` settings (such as `xpack.security.audit.enabled`).
+  * In orchestrated deployments (Elastic Cloud Hosted, ECE or ECK), the configuration is applied using the appropriate mechanisms provided by the orchestrator, and the restart will be applied automatically. Refer to [](/deploy-manage/deploy.md) for more information about the different orchestrators and deployment configuration mechanisms.
+
+## Enabling procedure [enable-audit-logging-procedure]
+
+To enable {{es}} or {{kib}} audit logs, configure `xpack.security.audit.enabled` to `true` in **all {{es}} or {{kib}} nodes**, then restart the nodes to apply the changes. The following provide detailed steps for all deployment types:
+
+::::::{tab-set}
+
+:::::{tab-item} Self-managed
+
+**To enable audit logging in {{es}}**:
+
+1. Set `xpack.security.audit.enabled` to `true` in `elasticsearch.yml`.
+2. Restart {{es}}.
+
+When audit logging is enabled, security events are persisted to a dedicated `<clustername>_audit.json` file on the host’s file system, on every cluster node. For more information, refer to [{{es}} logfile audit output](logfile-audit-output.md).
+
+You can configure additional options to control what events are logged and what information is included in the audit log. For more information, refer to [](#audit-logging-configuration).
+
+**To enable audit logging in {{kib}}**:
+
+1. Set `xpack.security.audit.enabled` to `true` in `kibana.yml`
+2. Restart {{kib}}
+
+You can optionally configure audit logs location, file/rolling file appenders and ignore filters using [{{kib}} audit logging settings](https://www.elastic.co/guide/en/kibana/current/security-settings-kb.html#audit-logging-settings).
+
+
+:::::
+
+:::::{tab-item} Elastic Cloud Hosted
+
+::::{important}
+In orchestrated deployments, audit logs must be shipped to a monitoring deployment; otherwise, they remain at container level and won't be accessible to users. For details on configuring log forwarding in orchestrated environments, refer to [logging configuration](../logging-configuration.md).
+::::
+
+To enable audit logging in an Elastic Cloud Hosted deployment:
+
+1. Log in to the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body).
+
+2. Find your deployment on the home page in the Elasticsearch Service card and select **Manage** to access it directly. Or, select **Hosted deployments** to go to the deployments page to view all of your deployments.
+
+3. From your deployment menu, go to the **Edit** page.
+
+4. To enable auditing for Elasticsearch:
+    * In the **Elasticsearch** section select **Manage user settings and extensions**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for each node instead.
+    * Add the setting `xpack.security.audit.enabled: true`.
+
+5. To enable auditing for Kibana:
+    * In the **Kibana** section, select **Edit user settings**. For deployments with existing user settings, you may have to expand the **Edit kibana.yml** caret instead.
+    * Add the setting `xpack.security.audit.enabled: true`.
+
+6. Select **Save changes**.
+    * A plan change will run on your deployment. When it finishes, **audit logs will be delivered to your monitoring deployment**.
+:::::
+
+:::::{tab-item} ECE
+
+::::{important}
+In orchestrated deployments, audit logs must be shipped to a monitoring deployment; otherwise, they remain at container level and won't be accessible to users. For details on configuring log forwarding in orchestrated environments, refer to [logging configuration](../logging-configuration.md).
+::::
+
+To enable audit logging in an ECE deployment:
+
+1. [Log into the Cloud UI](../../deploy/cloud-enterprise/log-into-cloud-ui.md).
+
+2. On the **Deployments** page, select your deployment.
+
+3. From your deployment menu, go to the **Edit** page.
+
+4. To enable auditing for {{es}}:
+    * In the **Elasticsearch** section, select **Edit user settings and plugins**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for the first node instead.
+    * Add the setting `xpack.security.audit.enabled: true`.
+
+5. To enable auditing for {{kib}}:
+    * In the **Kibana** section, select **Edit user settings**. For deployments with existing user settings, you may have to expand the **Edit kibana.yml** caret instead.
+    * Add the setting `xpack.security.audit.enabled: true`.
+    * If your Elastic Stack version is below 7.6.0, add the setting `logging.quiet: false`.
+
+6. Select **Save**.
+    * A plan change will run on your deployment. When it finishes, **audit logs will be delivered to your monitoring deployment**.
+:::::
+
+:::::{tab-item} ECK
+
+::::{important}
+In orchestrated deployments, audit logs must be shipped to a monitoring deployment; otherwise, they remain at container level and won't be accessible to users. For details on configuring log forwarding in orchestrated environments, refer to [logging configuration](../logging-configuration.md).
+::::
+
+To enable audit logging in an ECK-managed cluster, add `xpack.security.audit.enabled: true` to the `config` section of each {{es}} `nodeSet` and to the `config` section of the {{kib}} object's specification. The following example shows this configuration together with logs and metrics delivery towards a remote cluster:
+
+```yaml
+apiVersion: elasticsearch.k8s.elastic.co/v1
+kind: Elasticsearch
+spec:
+  monitoring:
+    metrics:
+      elasticsearchRefs:
+      - name: monitoring
+        namespace: observability
+    logs:
+      elasticsearchRefs:
+      - name: monitoring
+        namespace: observability
+  nodeSets:
+  - name: default
+    config:
+      # https://www.elastic.co/guide/en/elasticsearch/reference/current/enable-audit-logging.html
+      xpack.security.audit.enabled: true
+---
+apiVersion: kibana.k8s.elastic.co/v1
+kind: Kibana
+spec:
+  monitoring:
+    metrics:
+      elasticsearchRefs:
+      - name: monitoring
+        namespace: observability
+    logs:
+      elasticsearchRefs:
+      - name: monitoring
+        namespace: observability
+  config:
+    # https://www.elastic.co/guide/en/kibana/current/xpack-security-audit-logging.html
+    xpack.security.audit.enabled: true
+```
+
+When enabled, audit logs are collected and shipped to the monitoring cluster referenced in the `monitoring.logs` section. If monitoring is not enabled audit logs will only be visible at container level.
+:::::
+
+::::::
+
+## Advanced Configuration [audit-logging-configuration]
+
+{{es}} and {{kib}} audit logging frameworks offer several configuration mechanisms to control what events are logged and what information is included in the audit log. For more information, refer to:
+
+### Elasticsearch auditing configuration
+
+**{{es}} configuration** options include:
+
+  * [{{es}} audited events settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html#event-audit-settings)
+  * [{{es}} node information settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html#node-audit-settings)
+  * [{{es}} ignore policies settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html#audit-event-ignore-policies)
+
+    ::::{tip}
+    In {{es}}, all auditing settings except `xpack.security.audit.enabled` are **dynamic**, which means you can configure them using the [cluster update settings API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-update-settings.html) as a faster and more convenient method that applies changes without requiring a restart.
+    ::::
+
+For more information about event details and format, refer to:
+  * [{{es}} audit events details and schema](/deploy-manage/monitor/logging-configuration/elasticsearch-audit-events.md).
+  * [{{es}} logentry output format](/deploy-manage/monitor/logging-configuration/logfile-audit-output.md#audit-log-entry-format)
+
+* Elastic Cloud Hosted deployments provide its own subset of supported settings for auditing configuration:
+  * [Elasticsearch audit settings on Elastic Cloud](https://www.elastic.co/guide/en/cloud/current/ec-add-user-settings.html#ec_audit_settings)
+
+### Kibana auditing configuration
+
+**{{kib}} configuration** options include:
+
+  * [{{kib}} ignore filters](https://www.elastic.co/guide/en/kibana/current/security-settings-kb.html#audit-logging-ignore-filters)
+
+    ::::{tip}
+    To configure {{kib}} settings, follow the same [procedure](#enable-audit-logging-procedure) as when enabling {{kib}} audit logs, but apply the relevant settings instead.
+    ::::
+
+For a complete description of auditing event details, such as `category`, `type`, or `action`, refer to:
+  * [{{kib}} audit events](https://www.elastic.co/guide/en/kibana/current/xpack-security-audit-logging.html#xpack-security-ecs-audit-logging)
+
+* Elastic Cloud Hosted deployments provide its own subset of supported settings for auditing configuration:
+  * [Kibana audit settings on Elastic Cloud](https://www.elastic.co/guide/en/cloud/current/ec-manage-kibana-settings.html#ec_logging_and_audit_settings)
+
+### Configuration suggestions
+
+* Consider starting with [`xpack.security.audit.logfile.events.include`](https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html#xpack-sa-lf-events-include) or the corresponding `exclude` setting to specify the type of events you want to include or exclude in the {{es}} auditing output.
+
+* If you need a more granular control, refer to [{{es}} audit events ignore policies](./logfile-audit-events-ignore-policies.md) for a better understanding how ignore policies work and when they are beneficial.
+
+* Refer to [auditing search queries](./auditing-search-queries.md) for details on logging request bodies in the {{es}} audit logs.
+
+  ::::{important}
+  Be advised that **sensitive data may be audited in plain text** when including the request body in audit events, even though all the security APIs, such as those that change the user’s password, have the credentials filtered out when audited.
+  ::::
+
+* Use {{kib}} [ignore filters](https://www.elastic.co/guide/en/kibana/current/security-settings-kb.html#audit-logging-ignore-filters) if you want to filter out certain events from the {{kib}} audit log.