Adding index_template_substitutions to the simulate ingest API #114128

masseyke · 2024-10-04T12:46:57Z

This adds support for a new index_template_substitutions field to the body of an ingest simulate API request. These substitutions can be used to change the pipeline(s) used for ingest, or to change the mappings used for validation. It is similar to the component_template_substitutions added in #113276. Here is an example that shows both of those usages working together:

## First, add a couple of pipelines that set a field to a boolean:
PUT /_ingest/pipeline/foo-pipeline?pretty
{
  "processors": [
    {
      "set": {
        "field": "foo",
        "value": true
      }
    }
  ]
}

PUT /_ingest/pipeline/bar-pipeline?pretty
{
  "processors": [
    {
      "set": {
        "field": "bar",
        "value": true
      }
    }
  ]
}

## Now, create three component templates. One provides a mapping enforces that the only field is "foo"
## and that field is a keyword. The next is similar, but adds a `bar` field. The final one provides a setting
## that makes "foo-pipeline" the default pipeline.
## Remember that the "foo-pipeline" sets the "foo" field to a boolean, so using both of these templates
## together would cause a validation exception. These could be in the same template, but are provided
## separately just so that later we can show how multiple templates can be overridden.
PUT _component_template/mappings_template
{
  "template": {
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "foo": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT _component_template/mappings_template_with_bar
{
    "template": {
      "mappings": {
        "dynamic": "strict",
        "properties": {
          "foo": {
            "type": "keyword"
          },
          "bar": {
            "type": "boolean"
          }
        }
      }
    }
}

PUT _component_template/settings_template
{
  "template": {
    "settings": {
      "index": {
        "default_pipeline": "foo-pipeline"
      }
    }
  }
}

## Here we create an index template  pulling in both of the component templates above
PUT _index_template/template_1
{
  "index_patterns": ["foo*"],
  "composed_of": ["mappings_template", "settings_template"]
}

## We can index a document here to create the index, or not. Either way the simulate call ought to work the same
POST foo-1/_doc
{
  "foo": "FOO"
}

## This will not blow up with validation exceptions because the substitute "index_template_substitutions"
## uses `mappings_template_with_bar`, which adds the bar field.
## And the bar-pipeline is executed rather than the foo-pipeline because the substitute
## "index_template_substitutions" uses a substitute `settings_template`, so the value of "foo"
## does not get set to an invalid type.
POST _ingest/_simulate?pretty&index=foo-1
{
  "docs": [
    {
      "_id": "asdf",
      "_source": {
        "foo": "foo",
        "bar": "bar"
      }
    }
  ],
  "component_template_substitutions": {
    "settings_template": {
      "template": {
        "settings": {
          "index": {
            "default_pipeline": "bar-pipeline"
          }
        }
      }
    }
  },
  "index_template_substitutions": {
    "template_1": {
      "index_patterns": ["foo*"],
      "composed_of": ["mappings_template_with_bar", "settings_template"]
    }
  }
}

elasticsearchmachine · 2024-10-04T12:47:21Z

Hi @masseyke, I've created a changelog YAML for you.

elasticsearchmachine · 2024-10-07T18:05:15Z

Pinging @elastic/es-data-management (Team:Data Management)

masseyke · 2024-10-07T18:06:37Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataIndexTemplateService.java

            return Settings.EMPTY;
        }
-        return resolveSettings(template, metadata.componentTemplates(), templateSubstitutions);
+        return resolveSettings(template, metadata.componentTemplates());


The changes to this file are really just part of #113908 that I missed. They're not really worth their own PR (and backport), so I piggybacked them onto this one.

parkertimmins · 2024-10-08T20:25:11Z

...all-dependencies/src/yamlRestTest/resources/rest-api-spec/test/ingest/80_ingest_simulate.yml

+          index_patterns:
+            - foo*
+          composed_of:
+            - mappings_template


Should this also include settings_template?

Or maybe that doesn't work with the 3rd simulate ingest usage in this group, since you want to show that the pipeline is added by the substitution. But I wondered if the first two usages of simulate ingest, you wanted to check that pipeline added in the settings_template was removed due to the substituted settings_template.

Yeah I had intentionally left it out to test it being pulled in by the substitute index template.

parkertimmins · 2024-10-08T20:36:16Z

...all-dependencies/src/yamlRestTest/resources/rest-api-spec/test/ingest/80_ingest_simulate.yml

+              dynamic: strict
+              properties:
+                foo:
+                  type: keyword


Should this be boolean to show that it's the keyword from mappings_templates_2 that's being used?

Oops yeah I think that was my intention.

parkertimmins

Looks good to me 👍

elasticsearchmachine · 2024-10-08T23:16:59Z

💚 Backport successful

Status	Branch	Result
✅	8.x

…ic#114128) This adds support for a new `index_template_substitutions` field to the body of an ingest simulate API request. These substitutions can be used to change the pipeline(s) used for ingest, or to change the mappings used for validation. It is similar to the `component_template_substitutions` added in elastic#113276. Here is an example that shows both of those usages working together: ``` ## First, add a couple of pipelines that set a field to a boolean: PUT /_ingest/pipeline/foo-pipeline?pretty { "processors": [ { "set": { "field": "foo", "value": true } } ] } PUT /_ingest/pipeline/bar-pipeline?pretty { "processors": [ { "set": { "field": "bar", "value": true } } ] } ## Now, create three component templates. One provides a mapping enforces that the only field is "foo" ## and that field is a keyword. The next is similar, but adds a `bar` field. The final one provides a setting ## that makes "foo-pipeline" the default pipeline. ## Remember that the "foo-pipeline" sets the "foo" field to a boolean, so using both of these templates ## together would cause a validation exception. These could be in the same template, but are provided ## separately just so that later we can show how multiple templates can be overridden. PUT _component_template/mappings_template { "template": { "mappings": { "dynamic": "strict", "properties": { "foo": { "type": "keyword" } } } } } PUT _component_template/mappings_template_with_bar { "template": { "mappings": { "dynamic": "strict", "properties": { "foo": { "type": "keyword" }, "bar": { "type": "boolean" } } } } } PUT _component_template/settings_template { "template": { "settings": { "index": { "default_pipeline": "foo-pipeline" } } } } ## Here we create an index template pulling in both of the component templates above PUT _index_template/template_1 { "index_patterns": ["foo*"], "composed_of": ["mappings_template", "settings_template"] } ## We can index a document here to create the index, or not. Either way the simulate call ought to work the same POST foo-1/_doc { "foo": "FOO" } ## This will not blow up with validation exceptions because the substitute "index_template_substitutions" ## uses `mappings_template_with_bar`, which adds the bar field. ## And the bar-pipeline is executed rather than the foo-pipeline because the substitute ## "index_template_substitutions" uses a substitute `settings_template`, so the value of "foo" ## does not get set to an invalid type. POST _ingest/_simulate?pretty&index=foo-1 { "docs": [ { "_id": "asdf", "_source": { "foo": "foo", "bar": "bar" } } ], "component_template_substitutions": { "settings_template": { "template": { "settings": { "index": { "default_pipeline": "bar-pipeline" } } } } }, "index_template_substitutions": { "template_1": { "index_patterns": ["foo*"], "composed_of": ["mappings_template_with_bar", "settings_template"] } } } ```

…) (#114374) This adds support for a new `index_template_substitutions` field to the body of an ingest simulate API request. These substitutions can be used to change the pipeline(s) used for ingest, or to change the mappings used for validation. It is similar to the `component_template_substitutions` added in #113276. Here is an example that shows both of those usages working together: ``` ## First, add a couple of pipelines that set a field to a boolean: PUT /_ingest/pipeline/foo-pipeline?pretty { "processors": [ { "set": { "field": "foo", "value": true } } ] } PUT /_ingest/pipeline/bar-pipeline?pretty { "processors": [ { "set": { "field": "bar", "value": true } } ] } ## Now, create three component templates. One provides a mapping enforces that the only field is "foo" ## and that field is a keyword. The next is similar, but adds a `bar` field. The final one provides a setting ## that makes "foo-pipeline" the default pipeline. ## Remember that the "foo-pipeline" sets the "foo" field to a boolean, so using both of these templates ## together would cause a validation exception. These could be in the same template, but are provided ## separately just so that later we can show how multiple templates can be overridden. PUT _component_template/mappings_template { "template": { "mappings": { "dynamic": "strict", "properties": { "foo": { "type": "keyword" } } } } } PUT _component_template/mappings_template_with_bar { "template": { "mappings": { "dynamic": "strict", "properties": { "foo": { "type": "keyword" }, "bar": { "type": "boolean" } } } } } PUT _component_template/settings_template { "template": { "settings": { "index": { "default_pipeline": "foo-pipeline" } } } } ## Here we create an index template pulling in both of the component templates above PUT _index_template/template_1 { "index_patterns": ["foo*"], "composed_of": ["mappings_template", "settings_template"] } ## We can index a document here to create the index, or not. Either way the simulate call ought to work the same POST foo-1/_doc { "foo": "FOO" } ## This will not blow up with validation exceptions because the substitute "index_template_substitutions" ## uses `mappings_template_with_bar`, which adds the bar field. ## And the bar-pipeline is executed rather than the foo-pipeline because the substitute ## "index_template_substitutions" uses a substitute `settings_template`, so the value of "foo" ## does not get set to an invalid type. POST _ingest/_simulate?pretty&index=foo-1 { "docs": [ { "_id": "asdf", "_source": { "foo": "foo", "bar": "bar" } } ], "component_template_substitutions": { "settings_template": { "template": { "settings": { "index": { "default_pipeline": "bar-pipeline" } } } } }, "index_template_substitutions": { "template_1": { "index_patterns": ["foo*"], "composed_of": ["mappings_template_with_bar", "settings_template"] } } } ```

…ic#114128) This adds support for a new `index_template_substitutions` field to the body of an ingest simulate API request. These substitutions can be used to change the pipeline(s) used for ingest, or to change the mappings used for validation. It is similar to the `component_template_substitutions` added in elastic#113276. Here is an example that shows both of those usages working together: ``` ## First, add a couple of pipelines that set a field to a boolean: PUT /_ingest/pipeline/foo-pipeline?pretty { "processors": [ { "set": { "field": "foo", "value": true } } ] } PUT /_ingest/pipeline/bar-pipeline?pretty { "processors": [ { "set": { "field": "bar", "value": true } } ] } ## Now, create three component templates. One provides a mapping enforces that the only field is "foo" ## and that field is a keyword. The next is similar, but adds a `bar` field. The final one provides a setting ## that makes "foo-pipeline" the default pipeline. ## Remember that the "foo-pipeline" sets the "foo" field to a boolean, so using both of these templates ## together would cause a validation exception. These could be in the same template, but are provided ## separately just so that later we can show how multiple templates can be overridden. PUT _component_template/mappings_template { "template": { "mappings": { "dynamic": "strict", "properties": { "foo": { "type": "keyword" } } } } } PUT _component_template/mappings_template_with_bar { "template": { "mappings": { "dynamic": "strict", "properties": { "foo": { "type": "keyword" }, "bar": { "type": "boolean" } } } } } PUT _component_template/settings_template { "template": { "settings": { "index": { "default_pipeline": "foo-pipeline" } } } } ## Here we create an index template pulling in both of the component templates above PUT _index_template/template_1 { "index_patterns": ["foo*"], "composed_of": ["mappings_template", "settings_template"] } ## We can index a document here to create the index, or not. Either way the simulate call ought to work the same POST foo-1/_doc { "foo": "FOO" } ## This will not blow up with validation exceptions because the substitute "index_template_substitutions" ## uses `mappings_template_with_bar`, which adds the bar field. ## And the bar-pipeline is executed rather than the foo-pipeline because the substitute ## "index_template_substitutions" uses a substitute `settings_template`, so the value of "foo" ## does not get set to an invalid type. POST _ingest/_simulate?pretty&index=foo-1 { "docs": [ { "_id": "asdf", "_source": { "foo": "foo", "bar": "bar" } } ], "component_template_substitutions": { "settings_template": { "template": { "settings": { "index": { "default_pipeline": "bar-pipeline" } } } } }, "index_template_substitutions": { "template_1": { "index_patterns": ["foo*"], "composed_of": ["mappings_template_with_bar", "settings_template"] } } } ```

Adding index_template_substitutions to the simulate ingest API

33efab8

masseyke added >enhancement :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP v8.16.0 v9.0.0 labels Oct 4, 2024

Update docs/changelog/114128.yaml

87b70a9

masseyke added 6 commits October 4, 2024 08:17

removing incorrect supported query params

7f3d998

minor improvement to TransportSimulateBulkActionIT

96c702c

Adding API documentation

640af1d

improving yaml rest tests

7542c35

merging main

b6a4408

fixing tests

b3b2801

masseyke marked this pull request as ready for review October 7, 2024 18:04

elasticsearchmachine added the Team:Data Management Meta label for data/management team label Oct 7, 2024

masseyke commented Oct 7, 2024

View reviewed changes

Merge branch 'main' into simulate-ingest-index-template-substitutions

4750ae5

parkertimmins reviewed Oct 8, 2024

View reviewed changes

parkertimmins approved these changes Oct 8, 2024

View reviewed changes

masseyke added 3 commits October 8, 2024 16:44

improving a yaml rest test

32cb224

merging main

f7a68a5

merging main

b658fc7

masseyke added auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) auto-backport Automatically create backport pull requests when merged labels Oct 8, 2024

elasticsearchmachine merged commit fb482f8 into elastic:main Oct 8, 2024
16 checks passed

masseyke deleted the simulate-ingest-index-template-substitutions branch October 8, 2024 23:16

masseyke mentioned this pull request Oct 8, 2024

[8.x] Adding index_template_substitutions to the simulate ingest API (#114128) #114374

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Adding index_template_substitutions to the simulate ingest API #114128

Adding index_template_substitutions to the simulate ingest API #114128

Uh oh!

masseyke commented Oct 4, 2024 •

edited

Loading

Uh oh!

elasticsearchmachine commented Oct 4, 2024

Uh oh!

elasticsearchmachine commented Oct 7, 2024

Uh oh!

masseyke Oct 7, 2024

Uh oh!

parkertimmins Oct 8, 2024

Uh oh!

parkertimmins Oct 8, 2024 •

edited

Loading

Uh oh!

masseyke Oct 8, 2024

Uh oh!

parkertimmins Oct 8, 2024

Uh oh!

masseyke Oct 8, 2024

Uh oh!

parkertimmins left a comment

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Adding index_template_substitutions to the simulate ingest API #114128

Adding index_template_substitutions to the simulate ingest API #114128

Uh oh!

Conversation

masseyke commented Oct 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 4, 2024

Uh oh!

elasticsearchmachine commented Oct 7, 2024

Uh oh!

masseyke Oct 7, 2024

Choose a reason for hiding this comment

Uh oh!

parkertimmins Oct 8, 2024

Choose a reason for hiding this comment

Uh oh!

parkertimmins Oct 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

masseyke Oct 8, 2024

Choose a reason for hiding this comment

Uh oh!

parkertimmins Oct 8, 2024

Choose a reason for hiding this comment

Uh oh!

masseyke Oct 8, 2024

Choose a reason for hiding this comment

Uh oh!

parkertimmins left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 8, 2024

💚 Backport successful

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

masseyke commented Oct 4, 2024 •

edited

Loading

parkertimmins Oct 8, 2024 •

edited

Loading