Adding index_template_substitutions to the simulate ingest API

[masseyke]

This adds support for a new index_template_substitutions field to the body of an ingest simulate API request. These substitutions can be used to change the pipeline(s) used for ingest, or to change the mappings used for validation. It is similar to the component_template_substitutions added in #113276. Here is an example that shows both of those usages working together:

## First, add a couple of pipelines that set a field to a boolean:
PUT /_ingest/pipeline/foo-pipeline?pretty
{
  "processors": [
    {
      "set": {
        "field": "foo",
        "value": true
      }
    }
  ]
}

PUT /_ingest/pipeline/bar-pipeline?pretty
{
  "processors": [
    {
      "set": {
        "field": "bar",
        "value": true
      }
    }
  ]
}

## Now, create three component templates. One provides a mapping enforces that the only field is "foo"
## and that field is a keyword. The next is similar, but adds a `bar` field. The final one provides a setting
## that makes "foo-pipeline" the default pipeline.
## Remember that the "foo-pipeline" sets the "foo" field to a boolean, so using both of these templates
## together would cause a validation exception. These could be in the same template, but are provided
## separately just so that later we can show how multiple templates can be overridden.
PUT _component_template/mappings_template
{
  "template": {
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "foo": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT _component_template/mappings_template_with_bar
{
    "template": {
      "mappings": {
        "dynamic": "strict",
        "properties": {
          "foo": {
            "type": "keyword"
          },
          "bar": {
            "type": "boolean"
          }
        }
      }
    }
}

PUT _component_template/settings_template
{
  "template": {
    "settings": {
      "index": {
        "default_pipeline": "foo-pipeline"
      }
    }
  }
}

## Here we create an index template  pulling in both of the component templates above
PUT _index_template/template_1
{
  "index_patterns": ["foo*"],
  "composed_of": ["mappings_template", "settings_template"]
}

## We can index a document here to create the index, or not. Either way the simulate call ought to work the same
POST foo-1/_doc
{
  "foo": "FOO"
}

## This will not blow up with validation exceptions because the substitute "index_template_substitutions"
## uses `mappings_template_with_bar`, which adds the bar field.
## And the bar-pipeline is executed rather than the foo-pipeline because the substitute
## "index_template_substitutions" uses a substitute `settings_template`, so the value of "foo"
## does not get set to an invalid type.
POST _ingest/_simulate?pretty&index=foo-1
{
  "docs": [
    {
      "_id": "asdf",
      "_source": {
        "foo": "foo",
        "bar": "bar"
      }
    }
  ],
  "component_template_substitutions": {
    "settings_template": {
      "template": {
        "settings": {
          "index": {
            "default_pipeline": "bar-pipeline"
          }
        }
      }
    }
  },
  "index_template_substitutions": {
    "template_1": {
      "index_patterns": ["foo*"],
      "composed_of": ["mappings_template_with_bar", "settings_template"]
    }
  }
}

[elasticsearchmachine]

Hi @masseyke, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[masseyke]

The changes to this file are really just part of #113908 that I missed. They're not really worth their own PR (and backport), so I piggybacked them onto this one.

[parkertimmins]

Should this also include settings_template?

[parkertimmins]

Or maybe that doesn't work with the 3rd simulate ingest usage in this group, since you want to show that the pipeline is added by the substitution. But I wondered if the first two usages of simulate ingest, you wanted to check that pipeline added in the settings_template was removed due to the substituted settings_template.

[masseyke]

Yeah I had intentionally left it out to test it being pulled in by the substitute index template.

[parkertimmins]

Should this be boolean to show that it's the keyword from mappings_templates_2 that's being used?

[masseyke]

Oops yeah I think that was my intention.

[parkertimmins]

Looks good to me 👍

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.x