-
Notifications
You must be signed in to change notification settings - Fork 159
Add reference page for default log component templates #2689
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
76 changes: 76 additions & 0 deletions
76
solutions/observability/logs/logs-index-template-defaults.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,76 @@ | ||
--- | ||
navigation_title: Default log component templates | ||
description: Learn what the default component templates and ingest pipeline hooks for logs do. | ||
applies_to: | ||
stack: all | ||
serverless: all | ||
products: | ||
- id: observability | ||
--- | ||
|
||
# Default log component templates | ||
|
||
Elastic ships a set of reusable component templates and an optional `@custom` ingest pipeline that standardize how log data streams are mapped, indexed, and preprocessed. | ||
alexandra5000 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
These defaults apply automatically to any data stream named `logs-*-*`, unless you override them with your own higher-priority index template. | ||
alexandra5000 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
All of the default log components templates follow the same lifecycle: | ||
alexandra5000 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
* In {{stack}} (self-managed and {{ecloud}}): lifecycle is managed by [Index Lifecycle Management (ILM)](../../../manage-data/lifecycle/index-lifecycle-management.md). By default, rollover occurs when the primary shard reaches 50 GB or the index age reaches 30 days. | ||
* In {{serverless-full}}: lifecycle is managed by [Data Stream Lifecycle (DSL)](../../../manage-data/lifecycle/data-stream.md). By default, logs are retained for 30 days. | ||
|
||
Refer to [Logs index template reference](logs-index-template-reference.md) for instructions on how to view or edit the logs index template in {{kib}}. | ||
|
||
## Component templates | ||
|
||
The managed logs index template is composed of the following component templates: | ||
alexandra5000 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
### `logs@mappings` | ||
|
||
Provides general mappings for logs data streams: | ||
|
||
* Disables automatic date detection for string fields to avoid mis-parsing. | ||
* Defines ECS `data_stream.*` fields: | ||
* `data_stream.type`: constant_keyword, value `logs` | ||
* `data_stream.dataset`: constant_keyword, for example `nginx.access` (must be ≤ 100 characters, no `-`) | ||
* `data_stream.namespace`: constant_keyword, for example `production` (must be ≤ 100 characters, no `-`) | ||
alexandra5000 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
### `logs@settings` | ||
|
||
Configures default index settings for logs data streams: | ||
|
||
* Sets `@timestamp` from ingest time if missing. | ||
alexandra5000 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
* Contains a hook to the optional [`logs@custom`](#customize-preprocessing-with-logscustom) pipeline. | ||
alexandra5000 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
* Sets `ignore_malformed` to `true` globally. With this setting, documents with malformed fields can be indexed without causing ingestion failures. Refer to [ignore_malformed](elasticsearch://reference/elasticsearch/mapping-reference/ignore-malformed.md) for a list of supported fields. | ||
alexandra5000 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
### `ecs@mappings` | ||
|
||
* Adds dynamic templates that automatically align fields with [Elastic Common Schema (ECS)](ecs://reference/index.md). | ||
|
||
Alignment with ECS helps ensure that dashboards, queries, and ML jobs can work consistently across different log sources. | ||
|
||
|
||
## Customize preprocessing with `logs@custom` | ||
|
||
Each logs data stream runs through the default ingest pipeline. However, you can use the `logs@custom` component template to customize your {{es}} indices. The `logs@custom` component template is not installed by default, but you can create a component template named `logs@custom` to override and extend default mappings or settings. Refer to [Edit the `logs` index template](../logs/logs-index-template-reference.md#custom-logs-template-edit) for more information. | ||
|
||
|
||
|
||
## Using logs templates without naming conventions | ||
|
||
If you don’t follow the `logs-*-*` naming scheme, the managed logs index template will not apply automatically. | ||
alexandra5000 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
You can still use the defaults by composing standard component templates into your own index template. For example: | ||
alexandra5000 marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
||
```json | ||
PUT _index_template/my-logs-template | ||
{ | ||
"index_patterns": ["my-logs-template"], | ||
"composed_of": ["ecs@mappings", "logs@mappings", "logs@settings"], | ||
"priority": 200 | ||
} | ||
``` | ||
alexandra5000 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
::::{warning} | ||
Avoid bypassing the `logs-*-*` naming scheme unless you have a specific need. Many {{kib}} features (such as Discover views and Logs ML jobs) expect the managed logs template. | ||
:::: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.