FR: Allow Append processor to ignore empty values

[ulab]

Description

I am appending two values to a field using the Append processor.

Doing so can result in one of them being empty. For example:

related.ip: [(empty), 192.168.1.1]

It would be nice if there was a switch to "ignore empty values".

To avoid that issue, I currently have to split the single Append processor into two and use a condition in each to check if the value is null.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[joegallo]

edit: On reading your description closer, I see that you already realize what I explain this comment:

To avoid that issue, I currently have to split the single Append processor into two and use a condition in each to check if the value is null.

I've preserved the comment regardless in case others find the demonstration useful.

---

It's a bit janky, but you can accomplish this using an if attached to the associated append processor, as long as you also use a separate append processor for each field to be appended, like this:

PUT _ingest/pipeline/append-with-if-demo
{
  "description": "_description",
  "processors": [
    {
      "append": {
        "field": "target",
        "value": "{{foo}}",
        "if": "ctx.foo != null"
      }
    },
    {
      "append": {
        "field": "target",
        "value": "{{bar}}",
        "if": "ctx.bar != null"
      }
    }
  ]
}

POST /_ingest/pipeline/append-with-if-demo/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "foo1",
        "bar": "bar1"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "otherstuff": "quux"
      }
    }
  ]
}

See this example from the elastic/integrations for a more elaborate demonstration of this pattern in more 'real world' code.

My observation that this is currently possible by other means aside, I think this would be a good feature to add. Thank you for the suggestion, @ulab!

[limotova]

Hi, I decided to take a shot at implementing this feature #105718