ESQL: mv_expand default limit fix

[astefan]

Adds Analyzer level default limit for certain queries using mv_expand.
Fixes #101266

Fixes other small issues with different types of queries using mv_expand.
Fixes #102084
Fixes #102061

[elasticsearchmachine]

Pinging @elastic/es-ql (Team:QL)

[elasticsearchmachine]

Hi @astefan, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL)

[luigidellaquila]

I'm afraid this doesn't work: adding a limit there changes the query semantics by filtering out candidate records too early.
In this specific case, the | sort emp_no will be overridden by | sort first_name, so it can be completely removed and the query will become

from test
| mv_expand first_name
| sort first_name
| limit 15

In general, the problem is not that we cannot push LIMIT past MV_EXPAND; the following transformation is absolutelly valid:
| mv_expand x | limit 10 -> | limit 10 | mv_expand x | limit 10
and also the default limit is valid as long as it's higher than the original limit:
| mv_expand x | limit 10 -> | limit 10000 | mv_expand x | limit 10

The real problem here is that we cannot completely remove the limit 10 after mv_expand, so we end up trying to push it down it again in an infinite loop.
My biggest concern is that even if we find a way to avoid it now, we'll eventually add new rules to the planner and we'll hardly take this infinite loop into account, so it will eventually happen again.

[astefan]

I'm afraid this doesn't work: adding a limit there changes the query semantics by filtering out candidate records too early.

Agree with you, but as the planner works now - where a TopN is needed - a limit is required at that specific place in the query. As I mentioned in the code comments for now I decided to use the implicit limit.

In this specific case, the | sort emp_no will be overridden by | sort first_name

What do you mean "overriden"? The sorts do not change. If you meant "the sort first_name anyway changes the order of the final documents, so it makes sort emp_no obsolete" then this is not true. sort emp_no followed by mv_expand and then sort first_name has to happen in this specific order for things to make sense.

cannot push LIMIT past MV_EXPAN

I think this is a misunderstanding on your part. The code in the optimizer doesn't push limit, but it makes a copy of limit and pushes the copy. The limit after mv_expand doesn't disappear.

[luigidellaquila]

What do you mean "overriden"?

I mean that the following operations do not depend on the input order (none of the existing commands do, apart from LIMIT), so the fact that you sort records by emp_no doesn't make any difference: data will be re-sorted by first_name afterwards anyway

The code in the optimizer doesn't push limit, but it makes a copy of limit and pushes the copy

conceptually there is no big difference, you are adding a new limit before mv_expand (that can be considered a pushdown) and you are also preserving the original limit.

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[costin]

I believe this is superseded by #103582 - if so, let's close this one.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)