Allow private settings when system provided

[felixbarny]

As discussed in #132566, we'd like to provide private index settings with an IndexSettingsProvider. This is currently not possible because it fails validation for composable index templates (template v2). That's because we first merge all settings from the index template with the ones from the IndexSettingsProviders and then check whether there are private settings.

I've added a way so pass down the settings provided by the IndexSettingsProviders to the validation so that these system-provided settings are allowed to contain private settings.

Another area that needs to be enhanced is to allow downsampling to create indices containing private settings that are copied from the source index. For that reason, I've added a settingsSystemProvided flag to CreateIndexClusterStateUpdateRequest.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[kkrik-es]

The LogsdbIndexModeSettingsProvider already injects private settings:
https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/logsdb/src/main/java/org/elasticsearch/xpack/logsdb/LogsdbIndexModeSettingsProvider.java#L143

I'm probably missing something here..

[felixbarny]

Looks like we're skipping adding the private settings during template validation via this hack:

elasticsearch/x-pack/plugin/logsdb/src/main/java/org/elasticsearch/xpack/logsdb/LogsdbIndexModeSettingsProvider.java

Lines 210 to 214 in eb005b0

	if (MetadataIndexTemplateService.VALIDATE_INDEX_NAME.equals(indexName)) {
	// This index name is used when validating component and index templates, we should skip this check in that case.
	// (See MetadataIndexTemplateService#validateIndexTemplateV2(...) method)
	return MappingHints.EMPTY;
	}

I'd rather have the template validation work on the actual settings than the provider behaving differently during validation.

[dakrone]

I looked at this and left a couple of comments (nothing of substance). I keep having a sneaking suspicion that there's a more elegant way to do this, but it might be the late-Friday-afternoon brain telling me that. I'm going to take another look at it next week.

[dakrone]

Would it be better to leave the if statements alone and instead do something like:

for (final String key : settings.filter(s -> systemProvided.hasValue(s.getKey()) == false)).keySet()) {
    …
}

to filter the system-provided settings out of the validation list at the beginning?

[felixbarny]

It would be possible but we can't filter out all settings if they're also in the systemProvided settings. We also need to check if the value is the same to avoid allowing users to override private settings that are also system provided with a different value.

At this point, I'm not sure if the filtering is more readable. Another option to make it more readable is to extract an isSystemProvided method so that the condition becomes:

else if (setting.isPrivateIndex() && isSystemProvided(key, settings, systemProvided) == false)

[dakrone]

Can you add a comment about which setting requires this flag?

[felixbarny]

None, yet. I can remove this from the PR and re-add it when introducing the index.dimensions setting in #132566.