Enable failure store for logs by default

[flash1293]

Description

Failure store is released and the data set quality page in Kibana is providing insights for the user.

To avoid dropping data silently, the failure store should be enabled by default for logs data (everything covered by the logs template, as well as integrations).

While this might lead to increased disk usage for wrongly configured systems in which lots of malformed data is dropped by Elasticsearch, the benefits for regular system outweigh the downsides.

Expected behavior

In 9.2 (and on serverless right away), newly created data streams that rely on the logs template or are created by integrations should have the failure store enabled by default. Existing data streams don't have to be changed.

It should be possible to achieve this by adding the failure store setting to the existing logs@settings component template - @felixbarny could you confirm this?

[felixbarny]

It should be possible to achieve this by adding the failure store setting to the existing logs@settings component template - @felixbarny could you confirm this?

I'd defer to @jbaiera on how to best enable the failure store by default.

One issue with enabling the failure store via logs@settings is that this only applies to newly created data streams. From the docs:

Unlike the settings and mappings fields on an index template which are repeatedly applied to new data stream write indices on rollover, the data_stream_options section of a template is applied to a data stream only once when the data stream is first created. To configure existing data streams, use the put data stream options API.

There's also the data_streams.failure_store.enabled cluster setting that we may want to change. Not exactly sure if that enables the failure store for existing data streams though. The docs don't mention this explicitly (maybe this should be clarified). I suppose a rollover isn't here as the failure store is not configured on a backing index level but at the data stream level.

Should we also enable the failure store for other data streams by default, for example, traces-* and metrics-*?

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[flash1293]

The fact that it's just for new data streams is how it's supposed to work, so I think we are good here (cc @LucaWintergerst ). Good point about metrics and traces, I don't really see why we shouldn't enable it there as well. What do you think @LucaWintergerst ?

[dakrone]

+1 for newly created data streams by modifying only the template at this time. I think the setting is too broad of strokes and potentially changes behavior too much.

[LucaWintergerst]

@felixbarny we will do something similar for traces in 9.2

outcomes of the APM discussion, with link to Elastic internal Slack thread here:

summary of it:

- will not to enable on existing data streams in any version
- no default change in 8.x for new data streams (users can manually enable if they want to in 8.19, using the UI in Stack Management)
- make it the default in >=9.2, for NEW data streams only (reasoning: low risk, data was sent to Elastic in the first place, so we should not drop it - users can easily disable using the UI in either Stack Management or Streams in future releases)
- watermarking is respected in general, but FS doesn't have separate settings or less priority
- failure store retention will be separately controlled from the 'real' backing indices (TODO: decide on APM default retention)

[LucaWintergerst]

This is the relevant PR for APM:
#125672

[felixbarny]

Sounds good. We should also do the same for OTel data streams.