Some Elasticearch settings are mutually exclusive when specified as YAML

[pebrc]

Elasticsearch Version

8.17.1

Installed Plugins

No response

Java Version

bundled

OS Version

Linux

Problem Description

cluster.max_shards_per_node: 1000
cluster.max_shards_per_node.frozen: 1000 

in flat notation expands to the following (invalid) YAML

cluster:
  max_shards_per_node: 1000
  max_shards_per_node:
    frozen: 1000 

^ max_shards_per_node cannot be both a scalar value and an object

According to the official Elasticsearch documentation both formats can be used interchangeably: https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html#_config_file_format

This becomes a problem for example for the ECK operator that normalizes user submitted configuration in order to merge ECK operator specific configuration into it, making it impossible to use both settings from elasticsearch.yml

See elastic/cloud-on-k8s#8470

Steps to Reproduce

Configure Elasticsearch with the configuration snippet from the problem description:

cluster:
  max_shards_per_node: 1000
  max_shards_per_node:
    frozen: 1000 

or via ECK

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es 
spec:
  version: 8.17.0
  nodeSets:
  - name: default
    config:
      cluster.max_shards_per_node: 2000
      cluster.max_shards_per_node.frozen: 4000

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

This appears to be related to #85789, @rjernst do you know whether this is a format we intend to support this format in the settings infrastructure?

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

in flat notation

flat notation would be using dots for all field names, so that it appears like a single map with one set of keys

making it impossible to use both settings

It should be possible by flattening the fields. What was the elasticsearch.yml produced from this example?

  - name: default
    config:
      cluster.max_shards_per_node: 2000
      cluster.max_shards_per_node.frozen: 4000

Note that I believe settings like this are newer, we have typically had a different structure such as using .default on the end of the first setting in order to ensure uniqueness. Doesn't work in this case since the setting has existed forever, but just want to point out nothing has changed in settings, we just started creating these settings that conflict with each other.

[pebrc]

It should be possible by flattening the fields. What was the elasticsearch.yml produced from this example?

  - name: default
    config:
      cluster.max_shards_per_node: 2000
      cluster.max_shards_per_node.frozen: 4000

This leads to the same problem. ECK does normalise this by de-flattening it to the nested settings hierarchy before rendering elasticsearch.yml. That's necessary to be able merge different sources of configuration together into one resulting file (we are using https://pkg.go.dev/github.com/elastic/go-ucfg behind the scenes for that, not that it matters much in this context)

Note that I believe settings like this are newer, we have typically had a different structure such as using .default on the end of the first setting in order to ensure uniqueness. Doesn't work in this case since the setting has existed forever, but just want to point out nothing has changed in settings, we just started creating these settings that conflict with each other.

I understand that. Maybe we could still deconflict these two settings by deprecating cluster.max_shards_per_node.frozen and replacing it with something unambiguous like cluster.frozen.max_shards_per_node or something along those lines (does not have to be this exact hierarchy, this is just meant as an example for a previously unused settings path)

[rjernst]

Certainly possible, but would need distrib team to decide on that. @henningandersen wdyt of the above?