Enable failure store to APM datastreams

[rubvs]

Closes https://github.com/elastic/apm-managed-service/issues/1541

Enable failure store to APM datastreams

[jbaiera]

LGTM, though this will likely need to wait until after the feature is out from behind the feature flag to be merged.

[jbaiera]

This won't handle APM data streams that already exist when applied, is that something that needs to be considered as well for this rollout?

[rubvs]

This won't handle APM data streams that already exist when applied, is that something that needs to be considered as well for this rollout?

@1pkg and @simitt can you please give your input on this?

[simitt]

@jbaiera is there a different way to also apply it to existing data streams?

[jbaiera]

is there a different way to also apply it to existing data streams?

@simitt This is what the data_streams.failure_store.enabled cluster setting is for. It takes a list of index wildcard patterns as a value, and any data streams that match the patterns will have failure store enabled, as long as those data streams do not have failure store explicitly enabled/disabled. It's effectively selecting the data stream names that will have the feature enabled by default if not currently specified.

We'll need to use both the template and the cluster setting as part of the rollout. The template to enable the failure store on all data streams going forward, and the cluster setting to pick up any existing data streams from before the template update.

[carsonip]

does changing a setting require a plugin version bump in

elasticsearch/x-pack/plugin/apm-data/src/main/resources/resources.yaml

Line 4 in f7c1294

version: 14

?

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @rubvs, I've created a changelog YAML for you.

[dakrone]

LGTM

[simitt]

is there a different way to also apply it to existing data streams?

@simitt This is what the data_streams.failure_store.enabled cluster setting is for. It takes a list of index wildcard patterns as a value, and any data streams that match the patterns will have failure store enabled, as long as those data streams do not have failure store explicitly enabled/disabled. It's effectively selecting the data stream names that will have the feature enabled by default if not currently specified.

@jbaiera should the data_streams.failure_store.enabled cluster settings be changed as part of this PR then or is this done elsewhere?

@rubvs @1pkg please clarify with PMs (@akhileshpok @mlunadia @LucaWintergerst ) that enabling FS for all apm data streams by default is ok (in the past we have focused on MIS/serverless, but afaics this PR would enable it also for APM Server, so we need to coordinate).

[rubvs]

Left a comment in the original issue: https://github.com/elastic/apm-managed-service/issues/1541#issuecomment-3028247155

[rubvs]

Superseded by #131296