ES|QL: automatic prefiltering for semantic_text match queries

[ioanatia]

tracked in #123043

When using match with semantic_text fields that are backed by dense_vector embeddings, we should automatically apply pre-filters similarly to what we do for knn #131004.

The fact that we don't apply pre-filters right now constitutes a bug.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[ioanatia]

There's multiple things to consider here.

Let's take the following example, where our semantic_text field is backed by dense embeddings:

FROM test METADATA _score
WHERE match(semantic_title, "Harry Potter") and match(title, "Harry Potter")
SORT _score DESC

In this case we want match(title, "Harry Potter") to be a prefilter for the knn query that match(semantic_title, "Harry Potter") expands to.

In ES|QL this expression will be pushed down to lucene, meaning we replace these functions call to query builders that will be encapsulated in a BooleanQueryBuilder.

This is great news if we want to have a solution for auto prefiltering that happens during query rewrite - we can reuse the same approach for DSL and ES|QL.

However in ES|QL there are many, many things that we don't push down to lucene.
Take for example:

FROM test METADATA _score
WHERE match(semantic_title, "Harry Potter") and length(title) > 10
SORT _score DESC

length(title) > 10 needs to be a prefilter for match(semantic_title, "Harry Potter"), but it's not pushed down.
the way we approached this in ES|QL for the knn function, is that in this case knn translates to exact vector search using similarity functions.

So it might be that we need to:

1. First have a solution that applies when all filters are pushed down, where we can do the prefiltering during query rewriting.
2. Handle the case where the prefilters cannot be pushed down - there are some ways we can do this, but we need to discuss in more detail.