[Dashboards as code] Define schemas for API filter interface

[nickpeihl]

Part of #237535

Summary

Introduce validation schemas for an API Filter interface in the @kbn/es-query-server package, enabling "* as Code" API consumers to work with filters using a human-readable, type-safe format instead of the complex and weakly typed Filter interface that is currently stored.

The current Filter interface, while extensible, is unsuitable for "* as Code" API consumers:

• Contains redundant metadata (meta.key, meta.field, meta.params) that duplicates information already in the query
• Leaks internal Kibana state $state
• No clear validation boundaries so it is very easy to create malformed filters
• Type safety is weak due to Record<string, any> in query structures

Advantages

Our proposed API Filter schema offers more advantages for API consumers:

• Self-documented as operators are semantic: is, is_not, is_one_of, between, exists
• No Elasticsearch DSL knowledge required, but DSL filters are still supported
• Discriminated unions for operators (existence checks don't have value, range filters require RangeValue)
• Inferred types, Open API specifications, and validation using @kbn/config-schema package
• Impossible to create malformed filters
• Backwards compatible with lossless conversion to/from legacy filters
• Backwards compatibility has been round-trip tested against hundreds of real-world filters in saved objects in Beats and Integrations repositories
• No breaking changes to existing dashboards

Reviewer notes

This PR only introduces new schemas and does not make any behavioral changes to code. Subsequent PRs will be created:

1. To introduce to/from conversion functions between API filters and legacy filters
2. To integrate API filters with Unified Search in Kibana applications such as Dashboard, Lens, Maps.

Technical details

Schema structure

introduces discriminated union schemas ensuring exactly one of:

• condition: Simple filter (phrase, range, existence)
• group and nested groups: Logical AND/OR combinations
• dsl: Raw Elasticsearch DSL

See Examples below

Validation benefits

• Compile-time: TypeScript types are inferred from schemas rather than separate concrete types
• Runtime: @kbn/config-schema validates input at API boundaries
• Documentation: Schema descriptions auto-generate OpenAPI specs

Examples of the new API filter schemas compared to the legacy filter

API Filter

Legacy format

{
  "$state": {
    "store": "appState"
  },
  "meta": {
    "alias": null,
    "disabled": false,
    "negate": false,
    "key": "host.keyword",
    "field": "host.keyword",
    "type": "phrase",
    "params": {
      "query": "www.elastic.co"
    },
    "index": "90943e30-9a47-11e8-b64d-95841ca0b247"
  },
  "query": {
    "match_phrase": {
      "host.keyword": "www.elastic.co"
    }
  }
}

API format

{
  "pinned": false,
  "disabled": false,
  "dataViewId": "90943e30-9a47-11e8-b64d-95841ca0b247",
  "negate": false,
  "condition": {
    "field": "host.keyword",
    "operator": "is",
    "value": "www.elastic.co"
  }
}

Grouped filter

Legacy format

[
  {
    "meta": {
      "type": "combined",
      "relation": "AND",
      "params": [
        {
          "query": {
            "match_phrase": {
              "host.keyword": "www.elastic.co"
            }
          },
          "meta": {
            "negate": false,
            "key": "host.keyword",
            "field": "host.keyword",
            "type": "phrase",
            "params": {
              "query": "www.elastic.co"
            },
            "index": "90943e30-9a47-11e8-b64d-95841ca0b247",
            "disabled": false
          }
        },
        {
          "meta": {
            "negate": true,
            "index": "90943e30-9a47-11e8-b64d-95841ca0b247",
            "key": "machine.os.keyword",
            "field": "machine.os.keyword",
            "params": [
              "ios",
              "osx"
            ],
            "value": [
              "ios",
              "osx"
            ],
            "type": "phrases",
            "disabled": false
          },
          "query": {
            "bool": {
              "minimum_should_match": 1,
              "should": [
                {
                  "match_phrase": {
                    "machine.os.keyword": "ios"
                  }
                },
                {
                  "match_phrase": {
                    "machine.os.keyword": "osx"
                  }
                }
              ]
            }
          }
        }
      ],
      "disabled": false,
      "negate": false,
      "alias": null,
      "index": "90943e30-9a47-11e8-b64d-95841ca0b247"
    },
    "query": {},
    "$state": {
      "store": "appState"
    }
  }
]

API format

[
  {
    "pinned": false,
    "disabled": false,
    "dataViewId": "90943e30-9a47-11e8-b64d-95841ca0b247",
    "negate": false,
    "group": {
      "type": "AND",
      "conditions": [
        {
          "field": "host.keyword",
          "operator": "is",
          "value": "www.elastic.co"
        },
        {
          "type": "OR",
          "conditions": [
            {
              "field": "machine.os.keyword",
              "operator": "is_not",
              "value": "ios"
            },
            {
              "field": "machine.os.keyword",
              "operator": "is_not",
              "value": "osx"
            }
          ]
        }
      ]
    }
  }
]

Nested group filter

Legacy format

[
  {
    "$state": {
      "store": "appState"
    },
    "meta": {
      "alias": "My nested filter",
      "disabled": false,
      "negate": false,
      "type": "combined",
      "relation": "AND",
      "params": [
        {
          "query": {
            "match_phrase": {
              "host.keyword": "www.elastic.co"
            }
          },
          "meta": {
            "index": "90943e30-9a47-11e8-b64d-95841ca0b247",
            "negate": false,
            "key": "host.keyword",
            "field": "host.keyword",
            "type": "phrase",
            "params": {
              "query": "www.elastic.co"
            }
          }
        },
        {
          "meta": {
            "index": "90943e30-9a47-11e8-b64d-95841ca0b247",
            "negate": false,
            "type": "combined",
            "relation": "OR",
            "params": [
              {
                "query": {
                  "match_phrase": {
                    "index.keyword": "kibana_sample_data_logs"
                  }
                },
                "meta": {
                  "index": "90943e30-9a47-11e8-b64d-95841ca0b247",
                  "negate": false,
                  "key": "index.keyword",
                  "field": "index.keyword",
                  "type": "phrase",
                  "params": {
                    "query": "kibana_sample_data_logs"
                  }
                }
              },
              {
                "query": {
                  "bool": {
                    "should": [
                      {
                        "match_phrase": {
                          "extension.keyword": "css"
                        }
                      },
                      {
                        "match_phrase": {
                          "extension.keyword": "gz"
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                },
                "meta": {
                  "index": "90943e30-9a47-11e8-b64d-95841ca0b247",
                  "negate": false,
                  "key": "extension.keyword",
                  "type": "phrases",
                  "params": [
                    "css",
                    "gz"
                  ]
                }
              }
            ]
          }
        }
      ],
      "index": "90943e30-9a47-11e8-b64d-95841ca0b247"
    }
  }
]

API format

[
  {
    "pinned": false,
    "disabled": false,
    "dataViewId": "90943e30-9a47-11e8-b64d-95841ca0b247",
    "negate": false,
    "label": "My nested filter",
    "group": {
      "type": "AND",
      "conditions": [
        {
          "field": "host.keyword",
          "operator": "is",
          "value": "www.elastic.co"
        },
        {
          "type": "OR",
          "conditions": [
            {
              "field": "index.keyword",
              "operator": "is",
              "value": "kibana_sample_data_logs"
            },
            {
              "type": "OR",
              "conditions": [
                {
                  "field": "extension.keyword",
                  "operator": "is",
                  "value": "css"
                },
                {
                  "field": "extension.keyword",
                  "operator": "is",
                  "value": "gz"
                }
              ]
            }
          ]
        }
      ]
    }
  }
]

DSL filter

Legacy format

[
  {
    "meta": {
      "type": "custom",
      "disabled": false,
      "negate": false,
      "alias": null,
      "key": "query",
      "index": "90943e30-9a47-11e8-b64d-95841ca0b247"
    },
    "query": {
      "match": {
        "agent": {
          "query": "Mozilla",
          "fuzziness": "AUTO"
        }
      }
    },
    "$state": {
      "store": "appState"
    }
  }
]

API format

[
  {
    "pinned": false,
    "disabled": false,
    "dataViewId": "90943e30-9a47-11e8-b64d-95841ca0b247",
    "negate": false,
    "dsl": {
      "query": {
        "match": {
          "agent": {
            "query": "Mozilla",
            "fuzziness": "AUTO"
          }
        }
      }
    }
  }
]

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[elasticmachine]

Pinging @elastic/kibana-visualizations (Team:Visualizations)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: fb0f138

Failed CI Steps

• Check Types

History