Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions solutions/observability/logs/logs-index-template-defaults.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
---
navigation_title: Default `logs` index template
description: Learn what the default component templates and ingest pipeline hooks for logs do.
applies_to:
stack: all
serverless: all
products:
- id: observability
---

# Default `logs` index template

Elastic applies a managed index template to data streams that follow the `logs-*-*` index pattern. This index template references a set of reusable component templates and an optional `@custom` ingest pipeline that standardizes how log data streams are mapped, indexed, and preprocessed.

The managed `logs` index template has a priority of `100` and is automatically applied to any data stream following the `logs-*-*` index pattern, unless you override it with your own higher-priority index template.

Depending on your deployment, the `logs` index template applies one of the following data retention policies by default:

* In {{stack}} (self-managed and {{ecloud}}): lifecycle is managed by [Index Lifecycle Management (ILM)](../../../manage-data/lifecycle/index-lifecycle-management.md). By default, rollover occurs when the primary shard reaches 50 GB or the index age reaches 30 days.
* In {{serverless-full}}: lifecycle is managed by [Data Stream Lifecycle (DSL)](../../../manage-data/lifecycle/data-stream.md). By default, logs are retained for 30 days.

Refer to [Logs index template reference](logs-index-template-reference.md) for instructions on how to view or edit the logs index template in {{kib}}.

## Component templates

The managed `logs` index template is composed of the following component templates:

### `logs@mappings`

Provides general mappings for logs data streams:

* Disables automatic date detection for string fields to avoid mis-parsing.
* Defines ECS `data_stream.*` fields:
* `data_stream.type`: constant_keyword, value `logs`
* `data_stream.dataset`: constant_keyword, for example `nginx.access` (must be ≤ 100 characters, no `-`)
* `data_stream.namespace`: constant_keyword, for example `production` (must be ≤ 100 characters, no `-`)

### `logs@settings`

Configures default index settings for logs data streams:

* Sets `@timestamp` to ingest time if missing.
* Contains a hook to the optional [`logs@custom`](#customize-preprocessing-with-logscustom) pipeline.
* Sets `ignore_malformed` to `true` globally. With this setting, documents with malformed fields can be indexed without causing ingestion failures. Refer to [ignore_malformed](elasticsearch://reference/elasticsearch/mapping-reference/ignore-malformed.md) for a list of supported fields.

### `ecs@mappings`

* Adds dynamic templates that automatically align fields with [Elastic Common Schema (ECS)](ecs://reference/index.md).

Alignment with ECS helps ensure that dashboards, queries, and ML jobs can work consistently across different log sources.


## Customize preprocessing with `logs@custom`

Each logs data stream runs through the default ingest pipeline. However, you can use the `logs@custom` component template to customize your {{es}} indices. The `logs@custom` component template is not installed by default, but you can create a component template named `logs@custom` to override and extend default mappings or settings. Refer to [Edit the `logs` index template](../logs/logs-index-template-reference.md#custom-logs-template-edit) for more information.



## Using logs templates without naming conventions

If your logs data streams do not follow the `logs-*-*` naming scheme, the managed logs index template will not apply automatically.

You can still use the default component templates by adding them to your own index template. For example:

```json
PUT _index_template/my-logs-template
{
"index_patterns": ["my-logs-template"],
"composed_of": ["ecs@mappings", "logs@mappings", "logs@settings"],
"priority": 200
}
```

::::{warning}
Avoid bypassing the `logs-*-*` naming scheme unless you have a specific need. Many {{kib}} features (such as Discover views and Logs ML jobs) expect the managed logs template.
::::
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ Index templates are used to configure the backing indices of data streams as the

You can view the default `logs` index template in {{kib}}. To open **Index Management**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). Select **Index Templates** and search for `logs`. Select the `logs` index templates to view relevant component templates.

Refer to [Default log component templates](../logs/logs-index-template-defaults.md) for details on the mappings, settings, and pipelines included by default.


## Edit the `logs` index template [custom-logs-template-edit]

Expand All @@ -29,7 +31,7 @@ You can use the `logs@custom` component template to customize your {{es}} indice
1. To open **Index Management**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
2. Select **Component Templates**.
3. Click **Create component template**.
4. Name the component template logs@custom.
4. Name the component template `logs@custom`.
5. Add any custom metadata, index settings, or mappings.

Changes to component templates are not applied retroactively to existing indices. For changes to take effect, create a new write index for impacted data streams by triggering a rollover. Do this using the {{es}} [Rollover API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover). For example, to roll over the `logs-generic-default` data stream, run:
Expand All @@ -43,7 +45,7 @@ POST /logs-generic-default/_rollover/

The `logs` index template uses `default_field: [*]` meaning queries without specified fields will search across all fields. You can update the `default_field` to search in the `message` field instead of all fields using the `logs@custom` component template.

If you haven’t already created the `logs@custom`component template, create it as outlined in the previous section. Then, follow these steps to update the **Index settings** of the component template:
If you haven’t already created the `logs@custom` component template, create it as outlined in the previous section. Then, follow these steps to update the **Index settings** of the component template:

1. To open **Index Management**, find **Stack Management** in the main menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
2. Select **Component Templates**.
Expand Down
2 changes: 2 additions & 0 deletions solutions/toc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -417,6 +417,8 @@ toc:
- file: observability/logs/log-data-sources.md
- file: observability/logs/add-service-name-to-logs.md
- file: observability/logs/logs-index-template-reference.md
children:
- file: observability/logs/logs-index-template-defaults.md
- file: observability/logs/streams/streams.md
children:
- file: observability/logs/streams/management/extract.md
Expand Down
Loading