[Logsdb] Strategy to enable logsdb it on stateful & documentation

[lucabelluccini]

On serverless, logsdb is enabled by default globally.

On stateful, we need to define and document how to:

• Enable logsdb on a single integration (as long as the data stream is logs)
• Enable logsdb on all logs-* integrations

Considerations:

• Applicable only to 8.17+ as Logsdb became GA
• We need to make sure logsdb is safe to be enabled on all integrations
  • If this assumption is not correct, then we need to find a way to mark integrations as "logsdb-compatible"

Strategies

Possible strategies (manual)

Those strategies should work (tested in 8.17), but they're not yet documented

A) To enable logsdb across all logs-* data streams.

• set cluster.logsdb.enabled: true and restart the master node (?)
• on next rollover, ANY data stream matching logs-*-* will start using Logsdb (it is injected live by ES code Introduce an IndexSettingsProvider to inject logsdb index mode elasticsearch#113505)

B) To enable logsdb across all logs-* data streams which follow the Fleet Index Template structure

• create/update the logs@custom component template and add settings.index.mode to logsdb
• rollover manually or wait next rollover
  Risks:
• If we change settings.index.mode in the future, users might have the burden or take all the @custom component templates and update them

C) To enable logsdb on a specific logs-* data stream belonging to an integration.

• create/update the logs-<dataset>@custom component template and add settings.index.mode to logsdb
• rollover manually or wait next rollover
  Risks:
• If we change settings.index.mode in the future, users might have the burden or take all the @custom component templates and update them
• Users might attempt to use this on non-logs data streams...

Possible strategies (via UI)

Offer some "opt-in" button to automate the processes above via Stack Management.

Related

• Enable LogsDB by default for logs-*-* for stateful deployments elasticsearch#106489

Relevant SMEs

• @jsoriano
• @felixbarny
• @ruflin
• @martijnvg
• feel free to add/edit

[lucabelluccini]

FYI @Gunnerva as this is 90% stack, but it has impact on Fleet too.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[lucabelluccini]

FYI @daniela-elastic + @jamiehynds as this is overall all integrations

[lucabelluccini]

As LogsDB might be more and more demanded and interesting, I would recommend prioritizing this item for 9.0
FYI @nimarezainia

Obviously this cannot be purely driven by Fleet. Integrations MUST test this (we already enabled it by default on Serverless afaik), but it is likely work for integration developers to approve this.

[eedugon]

Great proposals @lucabelluccini, I think options B and C should probably be the recommendation except if we prefer logsdb to be enabled globally.

I think that's aligned with my comment and proposal on this issue.

[lucabelluccini]

I've not checked if on stateful ES 9.0, the setting cluster.logsdb.enabled: true is the default.
Also LogsDB is Enterprise Synthetic _source is enterprise feature since 8.17 so we will need to add a warning/heads up on it

It seems to depend on logsdb.prior_logs_usage at https://github.com/elastic/elasticsearch/blob/4b4c2d3901e40762d9b04f3e6321ef144f2627d8/x-pack/plugin/logsdb/src/main/java/org/elasticsearch/xpack/logsdb/LogsDBPlugin.java#L35
I think @martijnvg knows more about this (FYI @tylerperk)

[martijnvg]

Starting from 9.0.0, the cluster.logsdb.enabled cluster setting defaults to true in new clusters. If the cluster upgraded from 8.18/8.19 then it depends on whether logsdb data streams starting with logs-- pattern existed prior to upgrading. If that is the case then logsdb will be enabled by default, otherwise it will be enabled by default.

[felixbarny]

Also LogsDB is Enterprise since 8.17 so we will need to add a warning similar to the one of Synthetic _source w.r.t. "it requires the appropriate license".

LogsDB does not require an enterprise license. Only synthetic _source does. So a user with a basic license can still enable LogsDB and get storage savings. But ES will use regular _source instead of synthetic _source.

[lucabelluccini]

LogsDB does not require an enterprise license. Only synthetic _source does. So a user with a basic license can still enable LogsDB and get storage savings. But ES will use regular _source instead of synthetic _source.

Right

Starting from 9.0.0, the cluster.logsdb.enabled cluster setting defaults to true in new clusters. If the cluster upgraded from 8.18/8.19 then it depends on whether logsdb data streams starting with logs-- pattern existed prior to upgrading. If that is the case then logsdb will be enabled by default, otherwise it will be enabled by default.

I think this should e covered in the ES docs of 9.0. We will need:

• Tell that in Serverless we enable logsdb by default (this is for serverless docs)
• How do I tell if logsdb is enabled or not by default
• if you're starting from a brand new ES cluster on 9.0, you have logsdb enabled by default. If you had already logs-* data streams, logsdb is not enabled by default
• we need to tell how to opt-in on logsdb and what are the checks to be done being doing so

-> Who can track this for ES docs? Do we have something already planned for this @Gunnerva ?

For this specific integrations/Fleet, here I would:

• point to the ES docs above (once they exist) for enabling this at cluster level
• tell if one can enable logsdb on a specific integration and how (does the integration need to "claim it's compatible?"
• tell how to know if logsdb is in use on a fleet data stream
• warn to not enable logsdb per integration (in 9.1 we offer the possibility to have a @custom component template for all the data streams of the integration package, but it can break the non-logs data streams)

[marciw]

cc @kkrik-es who recently documented the defaults: v9 docs staging site

and the top of that page says "enabled by default for logs in Elastic Cloud Serverless"