[streams][lifecycle] update `inherit` behavior for classic streams

[klacabane]

Follow up from #221364 (comment)

Currently calling Reset to default action (inherit option) on an unwired stream will do nothing, it just means that streams will stop managing the lifecycle and it can be updated outside of the APIs. As a result the lastly applied lifecycle will remain in place.

This is both not that useful and unexpected, instead one would expect the initial configuration (before onboarding the stream) to be reverted

[elasticmachine]

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

[klacabane]

Options for reset to default

1. revert back to the configuration set at the time on onboarding - when we first persist the unwired stream, we save the current data stream lifecycle configuration and apply it back when the option is selected. One risk is that the configuration points to an ilm policy that may not exist anymore. While the UI shows that error, a change via the API would go through without error. An additional side effect is that the revert may not be completely true to its initial configuration: when we apply a dsl or ilm via stream we update all the backing indices so we get a consistent retention throughout the data. An unwired stream at the time of onboarding could have mixed retention throughout its backing indices that we won't be able to restore
2. revert back to the template's configuration - apply configuration set on index/component template. One risk is that we'd lose any datastream-level configuration set by users before onboarding (+ the risks about deleted ilm and mixed configuration mentioned in 1.). This could be explicitly mentioned, and the option named Reset to template configuration so there's no magic. One note on this option is that with the current APIs the logic would be 1. figure out what is the index template lifecycle configuration 2. apply the configuration on the data stream. The implication is that, if the index template changes its lifecycle configuration, the new settings won't be picked up by the data stream because we've set an override. If we need the datastream to follow the template's configuration (ie new backing indices are created with the template's configuration) we'd need a way to delete the overrides which I think is not doable at the moment (I didn't look too much into it yet, but maybe @masseyke knows a way?)
3. do not offer this option at all for unwired (current behavior) - although the stream starts in an inherit state, it is not possible to revert back to inherit once updated to dsl or ilm via streams

I'm leaning towards 2. if we can actually fall back to the live template's configuration, otherwise 3. sounds reasonable. @flash1293 @LucaWintergerst Thoughts ?

[flash1293]

I would like number 2 as well, but without using an override, that sounds confusing. There should probably be a way to remove the override and revert to what the template dictates anyway, even for non-stream use cases. But interested in what @LucaWintergerst and @masseyke think

[LucaWintergerst]

I would also go with number 2 - that seems to be the most predictable even though it has some caveats.

[flash1293]

@klacabane since you self-assigned, are you going to pick this up?

[klacabane]

@flash1293 yes, I've looked into how we could reset data stream level settings to fallback to the one configured at the template level but couldn't find a way with the current GET, PUT apis (eg index.lifecycle.name: null would still take precedence) so I think we need a DELETE or a way to remove the data stream settings.

@masseyke is there currently a way to fallback to the template settings once data stream settings have been set ?

[flash1293]

@klacabane isn't this just

PUT _data_stream/logs-generic-default/_settings
{
  "index": {
    "lifecycle": {
      "name": null
    }
  }
}

or am I missing something here?

[klacabane]

(eg index.lifecycle.name: null would still take precedence)

@flash1293 I think index.lifecycle.name: null is considered a valid option, and in that case no lifecycle would be applied:

// created with default logs lifecycle
POST logs-lifecycle-overwrite/_doc
{
    "@timestamp": "2025-06-16"
}

// new write index picks up `logs` lifecycle
POST logs-lifecycle-overwrite/_rollover

// overwrite lifecycle at data stream level
PUT _data_stream/logs-lifecycle-overwrite/_settings
{
    "index.lifecycle.name": ".alerts-ilm-policy"
}

// new write index picks up `.alerts-ilm-policy` lifecycle
POST logs-lifecycle-overwrite/_rollover

// attempt the remove overwritten setting
PUT _data_stream/logs-lifecycle-overwrite/_settings
{
    "index.lifecycle.name": null
}

// new write index is `Unmanaged`
POST logs-lifecycle-overwrite/_rollover

[flash1293]

Oh, I see, good point... Looks like we need a follow-up here, waiting for @dakrone and @masseyke

[dakrone]

I think it would be beneficial to have a way to remove an override, @masseyke what do you think?

[flash1293]

FWIW, what I tried out of instinct is

PUT _data_stream/logs-generic-default/_settings
{
  "index": {
    "lifecycle": null
  }
}

but it didn't work