Correctly handling data stream settings when component templates are used

[masseyke]

If a composable template had no template structure of its own, and only used settings and mappings pulled in from component templates, then updating settings on a data stream built from that composable template could fail. For example:

// 1. create data stream that picks up the builtin `logs` template
POST logs-foo-bar/_doc
{
    "@timestamp": "2025-06-16"
}

// 2. set overrides
PUT _data_stream/logs-foo-bar/_settings
{
  "index.lifecycle.name": "foo",
  "index.lifecycle.prefer_ilm": true
}

// 3. unset overrides
PUT _data_stream/logs-foo-bar/_settings
{
  "index.lifecycle.name": null,
  "index.lifecycle.prefer_ilm": null
}

--->

{
  "data_streams": [
    {
      "name": "logs-foo-bar",
      "applied_to_data_stream": false,
      "error": """Cannot invoke "org.elasticsearch.cluster.metadata.Template.settings()" because the return value of "org.elasticsearch.cluster.metadata.ComposableIndexTemplate.template()" is null""",
      "settings": {},
      "effective_settings": {},
      "index_settings_results": {
        "applied_to_data_stream_only": [],
        "applied_to_data_stream_and_backing_indices": []
      }
    }
  ]
}

This change fixes this bug by correctly (and safely) resolving all settings for a data stream for validation, rather than only the ones from the top-level composable template.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @masseyke, I've created a changelog YAML for you.

[lukewhiting]

LGTM 👍🏻

[nielsbauman]

I don't think we need to label this as >bug since none of these versions have been released, right? I'm mostly referring to the changelog here.

[masseyke]

I don't think we need to label this as >bug since none of these versions have been released, right? I'm mostly referring to the changelog here.

Yeah, I initially labeled it as >non-issue and then added >bug for 9.1.0... and then remembered that we haven't actually released 9.1.0 yet. Removing the label now.

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.1	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 130394

[masseyke]

💚 All backports created successfully

Status	Branch	Result
✅	9.1
✅	8.19

Questions ?

Please refer to the Backport tool documentation