Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions docs/en/ingest-management/data-streams.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ For data streams, the index template configures the stream's backing indices as
These templates are loaded when the integration is installed, and are used to configure the integration's data streams.

[discrete]
[[data-streams-index-templates-edit]]
== Edit the {es} index template

WARNING: Custom index mappings may conflict with the mappings defined by the integration
Expand Down Expand Up @@ -187,6 +188,8 @@ this pipeline can be used for custom data processing, adding fields, sanitizing

Starting in version 8.12, ingest pipelines can be configured to process events at various levels of customization.

NOTE: If you create a custom index pipeline, Elastic is not responsible for ensuring that it indexes and behaves as expected. Creating a custom pipeline involves custom processing of the incoming data, which should be done with caution and tested carefully.

`global@custom`::
Apply processing to all events
+
Expand Down Expand Up @@ -305,7 +308,6 @@ Specifically, apply the built-in `90-days-default` {ilm-init} policy so that dat

NOTE: If you cloned an index template to customize the data retention policy on an {es} version prior to 8.13, you must update the index template in the clone to use the `ecs@mappings` component template on {es} version 8.13 or later. See <<data-streams-pipeline-update-cloned-template-before-8.13,Update index template cloned before {es} 8.13>> for the step-by-step instructions.


[discrete]
[[data-streams-ilm-one]]
== Step 1: View data streams
Expand Down Expand Up @@ -370,7 +372,16 @@ Now that you've created a component template,
you need to create an index template to apply the changes to the correct data stream.
The easiest way to do this is to duplicate and modify the integration's existing index template.

WARNING: When duplicating the index template, do not change or remove any managed properties. This may result in problems when upgrading.
[WARNING]
====
Please note the following:

* When duplicating the index template, do not change or remove any managed properties. This may result in problems when upgrading. Cloning the index template of an integration package involves some risk as any changes made to the original index template when it is upgraded will not be propagated to the cloned version.

* These steps assume that you want to have a namespace specific ILM policy, which requires index template cloning. Cloning the index template of an integration package involves some risk because any changes made to the original index template as part of package upgrades are not propagated to the cloned version. See <<assets-restrictions-cloning-index-template>> for details.
+
If you want to change the ILM Policy, the number of shards, or other settings for the datastreams of one or more integrations, but **the changes do not need to be specific to a given namespace**, it's strongly recommended to use the `<package>@custom` component template so as to avoid the problems mentioned above. See the <<data-streams-ilm,ILM>> section for details.
====

. Navigate to **{stack-manage-app}** > **Index Management** > **Index Templates**.
. Find the index template you want to clone. The index template will have the `<type>` and `<dataset>` in its name,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
[[integrations-assets-best-practices]]
= Best practices for integrations assets
= Best practices for integration assets

When you use integrations with {fleet} and {agent} there are some restrictions to be aware of.

* <<assets-restrictions-standalone>>
* <<assets-restrictions-without-agent>>
* <<assets-restrictions-custom-integrations>>
* <<assets-restrictions-copying>>
* <<assets-restrictions-editing-assets>>
* <<assets-restrictions-custom-component-templates>>
* <<assets-restrictions-custom-ingest-pipeline>>
* <<assets-restrictions-cloning-index-template>>

[discrete]
[[assets-restrictions-standalone]]
Expand Down Expand Up @@ -46,3 +50,41 @@ This way, because the assets are not managed by another integration, there is le

Note, however, that creating standalone integration assets based off of {fleet} and {agent} integrations is considered a custom configuration that is not tested nor supported. Whenever possible it's recommended to use standard integrations.

[discrete]
[[assets-restrictions-editing-assets]]
== Editing assets managed by {fleet}

{fleet}-managed integration assets should not be edited. Examples of these assets include an integration index template, the `@package` component templates, and ingest pipelines that are bundled with integrations. Any changes made to these assets will be overwritten when the integration is upgraded.

[discrete]
[[assets-restrictions-custom-component-templates]]
== Creating custom component templates

While creating a `@custom` component template for a package integration is supported, it involves risks which can prevent data from being ingested correctly. This practice can lead to broken indexing, data loss, and breaking of integration package upgrades.

For example:

* If the `@package` compoment template of an integration is changed from a "normal" datastream to `TSDB` or `LogsDB`, some of the custom settings or mappings introduced may not be compatible with these indexing modes.
* If the type of an ECS field is overridden from, for example, `keyword` to `text`, aggregations based on that field may be prevented for built-in dashboards.

A similar caution against custom index mappings is noted in <<data-streams-index-templates-edit>>.

[discrete]
[[assets-restrictions-custom-ingest-pipeline]]
== Creating a custom ingest pipeline

If you create a custom index pipeline (as documented in the <<data-streams-pipeline-tutorial,Transform data with custom ingest pipelines>> tutorial), Elastic is not responsible for ensuring that it indexes and behaves as expected. Creating a custom pipeline involves custom processing of the incoming data, which should be done with caution and tested carefully.

Refer to <<data-streams-pipelines>> to learn more.

[discrete]
[[assets-restrictions-cloning-index-template]]
== Cloning the index template of an integration package

When you clone the index template of an integration package, this involves risk as any changes made to the original index template when it is upgraded will not be propagated to the cloned version. That is, the structure of the new index template is effectively frozen at the moment that it is cloned. Cloning an index template of an integration package can therefore lead to broken indexing, data loss, and breaking of integration package upgrades.

Additionally, cloning index templates to add or inject additional component templates cannot be tested by Elastic, so we cannot guarantee that the template will work in future releases.

If you want to change the ILM Policy, the number of shards, or other settings for the datastreams of one or more integrations, but the changes do not need to be specific to a given namespace, it's highly recommended to use the `<package>@custom` component template so as to avoid the problems mentioned above.

If you want to change these settings for datastreams of one or more integrations and the changes **need to be namespace specific**, then you can do so following the steps in the <<data-streams-ilm-tutorial,Customize data retention policies tutorial>>, but be aware of the restrictions mentioned above.