diff --git a/solutions/observability/logs/logs-index-template-defaults.md b/solutions/observability/logs/logs-index-template-defaults.md new file mode 100644 index 0000000000..5f13a23aa5 --- /dev/null +++ b/solutions/observability/logs/logs-index-template-defaults.md @@ -0,0 +1,80 @@ +--- +navigation_title: Default `logs` index template +description: Learn what the default component templates and ingest pipeline hooks for logs do. +applies_to: + stack: all + serverless: all +products: + - id: observability +--- + +# Default `logs` index template + +Elastic applies a managed index template to data streams that follow the `logs-*-*` index pattern. This index template references a set of reusable component templates and an optional `@custom` ingest pipeline that standardizes how log data streams are mapped, indexed, and preprocessed. + +The managed `logs` index template has a priority of `100` and is automatically applied to any data stream following the `logs-*-*` index pattern, unless you override it with your own higher-priority index template. + +By default, this index template also enables [LogsDB index mode](../../..//manage-data/data-store/data-streams/logs-data-stream.md), which optimizes storage and query performance for log data streams. + +Depending on your deployment, the `logs` index template applies one of the following data retention policies by default: + +* In {{stack}} (self-managed and {{ecloud}}): lifecycle is managed by [Index Lifecycle Management (ILM)](../../../manage-data/lifecycle/index-lifecycle-management.md). By default, rollover occurs when the primary shard reaches 50 GB or the index age reaches 30 days. +* In {{serverless-full}}: lifecycle is managed by [Data Stream Lifecycle (DSL)](../../../manage-data/lifecycle/data-stream.md). By default, logs are retained for 30 days. + +Refer to [Logs index template reference](logs-index-template-reference.md) for instructions on how to view or edit the logs index template in {{kib}}. + +## Component templates + +The managed `logs` index template is composed of the following component templates: + +### `logs@mappings` + +Provides general mappings for logs data streams: + +* Disables automatic date detection for string fields to avoid mis-parsing. +* Defines ECS `data_stream.*` fields: + * [`data_stream.type`](ecs://reference/ecs-data_stream.md#field-data-stream-type): constant_keyword, value `logs` + * [`data_stream.dataset`](ecs://reference/ecs-data_stream.md#field-data-stream-dataset): constant_keyword, for example `nginx.access` (must be ≤ 100 characters, no `-`) + * [`data_stream.namespace`](ecs://reference/ecs-data_stream.md#field-data-stream-namespace): constant_keyword, for example `production` (must be ≤ 100 characters, no `-`) + +### `logs@settings` + +Configures default index settings for logs data streams: + + * References the managed ingest pipeline `logs@default-pipeline`, which: + * Sets `@timestamp` to the ingest time if it is missing. + * Contains a hook to the optional [`logs@custom`](#customize-preprocessing-with-logscustom) pipeline. + * Sets `ignore_malformed` to `true` globally. With this setting, documents with malformed fields can be indexed without causing ingestion failures. Refer to [ignore_malformed](elasticsearch://reference/elasticsearch/mapping-reference/ignore-malformed.md) for a list of supported fields. + * Sets `ignore_dynamic_beyond_limit` to `true`, which allows dynamically mapped fields to be added even when the total field limit is exceeded. Extra fields are ignored instead of causing ingestion to fail. Refer to [Mapping limit settings](elasticsearch://reference/elasticsearch/index-settings/mapping-limit.md) for more information. + +### `ecs@mappings` + +* Adds dynamic templates that automatically align fields with [Elastic Common Schema (ECS)](ecs://reference/index.md). + +Alignment with ECS helps ensure that dashboards, queries, and ML jobs can work consistently across different log sources. + + +## Customize preprocessing with `logs@custom` + +Each logs data stream runs through the default ingest pipeline. However, you can use the `logs@custom` component template to customize your {{es}} indices. The `logs@custom` component template is not installed by default, but you can create a component template named `logs@custom` to override and extend default mappings or settings. Refer to [Edit the `logs` index template](../logs/logs-index-template-reference.md#custom-logs-template-edit) for more information. + + + +## Using logs templates without naming conventions + +If your logs data streams do not follow the `logs-*-*` naming scheme, the managed logs index template will not apply automatically. + +You can still use the default component templates by adding them to your own index template. For example: + +```json +PUT _index_template/my-logs-template +{ + "index_patterns": ["my-logs-template"], + "composed_of": ["ecs@mappings", "logs@mappings", "logs@settings"], + "priority": 200 +} +``` + +::::{warning} +Avoid bypassing the `logs-*-*` naming scheme unless you have a specific need. Many {{kib}} features (such as Discover views and Logs ML jobs) expect the managed logs template. +:::: \ No newline at end of file diff --git a/solutions/observability/logs/logs-index-template-reference.md b/solutions/observability/logs/logs-index-template-reference.md index b302b60206..5acc0fda29 100644 --- a/solutions/observability/logs/logs-index-template-reference.md +++ b/solutions/observability/logs/logs-index-template-reference.md @@ -14,6 +14,8 @@ Index templates are used to configure the backing indices of data streams as the You can view the default `logs` index template in {{kib}}. To open **Index Management**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Select **Index Templates** and search for `logs`. Select the `logs` index templates to view relevant component templates. +Refer to [Default log component templates](../logs/logs-index-template-defaults.md) for details on the mappings, settings, and pipelines included by default. + ## Edit the `logs` index template [custom-logs-template-edit] @@ -29,7 +31,7 @@ You can use the `logs@custom` component template to customize your {{es}} indice 1. To open **Index Management**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). 2. Select **Component Templates**. 3. Click **Create component template**. -4. Name the component template logs@custom. +4. Name the component template `logs@custom`. 5. Add any custom metadata, index settings, or mappings. Changes to component templates are not applied retroactively to existing indices. For changes to take effect, create a new write index for impacted data streams by triggering a rollover. Do this using the {{es}} [Rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover). For example, to roll over the `logs-generic-default` data stream, run: @@ -43,7 +45,7 @@ POST /logs-generic-default/_rollover/ The `logs` index template uses `default_field: [*]` meaning queries without specified fields will search across all fields. You can update the `default_field` to search in the `message` field instead of all fields using the `logs@custom` component template. -If you haven’t already created the `logs@custom`component template, create it as outlined in the previous section. Then, follow these steps to update the **Index settings** of the component template: +If you haven’t already created the `logs@custom` component template, create it as outlined in the previous section. Then, follow these steps to update the **Index settings** of the component template: 1. To open **Index Management**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). 2. Select **Component Templates**. diff --git a/solutions/toc.yml b/solutions/toc.yml index b09680b4b2..fd7afad0e7 100644 --- a/solutions/toc.yml +++ b/solutions/toc.yml @@ -414,6 +414,8 @@ toc: - file: observability/logs/log-data-sources.md - file: observability/logs/add-service-name-to-logs.md - file: observability/logs/logs-index-template-reference.md + children: + - file: observability/logs/logs-index-template-defaults.md - file: observability/logs/streams/streams.md children: - file: observability/logs/streams/management/extract.md