Skip to content

Conversation

masseyke
Copy link
Member

@masseyke masseyke commented Sep 11, 2024

This adds the ability to pass substitute component template definitions to the simulate ingest API. These definitions can be used to change the pipeline(s) used for ingest, or to change the mappings used for validation. Here is an example that shows both of those usages working together:

## First, add a couple of pipelines that set a field to a boolean:
PUT /_ingest/pipeline/foo-pipeline?pretty
{
  "processors": [
    {
      "set": {
        "field": "foo",
        "value": true
      }
    }
  ]
}

PUT /_ingest/pipeline/bar-pipeline?pretty
{
  "processors": [
    {
      "set": {
        "field": "bar",
        "value": true
      }
    }
  ]
}

## Now, create two component templates. One provides a mapping enforces that the only field is "foo"
## and that field is a keyword. The other provides a setting that makes "foo-pipeline" the default pipeline.
## Remember that the "foo-pipeline" sets the "foo" field to a boolean, so using both of these templates
## together would cause a validation exception. These could be in the same template, but are provided
## separately just so that later we can show how multiple templates can be overridden.
PUT _component_template/mappings_template
{
  "template": {
    "mappings": {
      "dynamic": "strict",
      "properties": {
        "foo": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT _component_template/settings_template
{
  "template": {
    "settings": {
      "index": {
        "default_pipeline": "foo-pipeline"
      }
    }
  }
}

## Here we create an index template  pulling in both of the component templates above
PUT _index_template/template_1
{
  "index_patterns": ["foo*"],
  "composed_of": ["mappings_template", "settings_template"]
}

## We can index a document here to create the index, or not. Either way the simulate call ought to work the same
POST foo-1/_doc
{
  "foo": "FOO"
}

## This will not blow up with validation exceptions because the substitute "mappings_template" sets
## dynamic to true so that the introduction of the "bar" field doesn't cause problems.
## And the bar-pipeline is executed rather than the foo-pipeline because of the substitute 
## "settings_template", so the value of "foo" does not get set to an invalid type.
POST _ingest/_simulate?pretty&index=foo-1
{
  "docs": [
    {
      "_id": "asdf",
      "_source": {
        "foo": "foo",
        "bar": "bar"
      }
    }
  ],
  "template_substitutions": {
    "mappings_template": {
      "mappings": {
        "dynamic": "strict",
        "properties": {
          "foo": {
            "type": "keyword"
          },
          "bar": {
            "type": "boolean"
          }
        }
      }
    },
    "settings_template": {
      "settings": {
        "index": {
          "default_pipeline": "bar-pipeline"
        }
      }
    }
  }
}

@masseyke masseyke added >enhancement :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP v8.16.0 v9.0.0 labels Sep 11, 2024
@elasticsearchmachine
Copy link
Collaborator

Hi @masseyke, I've created a changelog YAML for you.

@masseyke masseyke changed the title Simulate ingest API template_substitutions prototype Adding component template substitutions to the simulate ingest API Sep 13, 2024
@masseyke
Copy link
Member Author

Closing in favor of #113276

@masseyke masseyke closed this Sep 20, 2024
@masseyke masseyke deleted the component-template-substitutions-for-simulate branch September 20, 2024 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement v8.16.0 v9.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants