ESQL: Remove redundant TopN

[kanoshiou]

Redundant TopN operations were not being removed from the execution plan, leading to unnecessary performance overhead.

We can safely consider a child TopN plan redundant if it shares the same sort order as its parent. There's no need to compare their limit values, since PushDownAndCombineLimits already pushes down the lower limit value, making it safe to remove the child TopN.

Also, the PruneRedundantOrderBy rule was only executed before the ReplaceLimitAndSortAsTopN, so it could only detect redundant OrderBy operations but not redundant TopN operations that were created later in the optimization pipeline.

Closes #131233

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[smalyshev]

@elasticsearchmachine test this please

[kanoshiou]

Hi, @smalyshev. It looks like some CI tests are failing, possibly due to the branch being out-of-date with main. Could you try updating the branch before re-running the tests?

[alex-spies]

Thanks @kanoshiou and apologies that it took us so long to get back to you! Your contributions are very much appreciated!

Unfortunately, I think the new version of PruneRedundantOrderBy is affecting too many queries, including ones where removing an upstream TopN is incorrect. I give examples below.

I think we should limit the scope by adding a new rule that only looks for subsequent TopNs, specifically, and place that rule into the cleanup batch of the LogicalPlanOptimizer. That's more obviously correct. Cases when there are query plan nodes between two TopNs and the upstream one should be pruned are trickier to handle and likely require to mark query plans in some way when they are compatible with this optimization.

[alex-spies]

Caution: equals will currently still return true even if we order by 2 attributes that have the same name+type, but not the same NameId - i.e. attributes that mean something else.

This is a different bug that needs fixing separately, but I wanted to point out that currently it will prevent this from rightfully triggering in a couple situations.

[kanoshiou]

Oh, I’m currently encountering this bug while resolving another issue. This is sad 😭

Is this bug touching too much code, so it’s hard to fix?

[alex-spies]

The new version of the rule is incorrect, I'm afraid.

We continue going down the plan as long as the children are SortAgnostic. However, being SortAgnostic doesn't automatically imply that the plans behave the same when we feed them an unlimited number of rows.

For instance, INLINE STATS is SortAgnostic (resp. InlineJoin is). But removing a TopN from before an INLINE STATS changes the meaning of the query.

Also, queries can become much more expensive: For instance, MvExpand is SortAgnostic, but SORT a | LIMIT 10 | MV_EXPAND b | SORT a | LIMIT 10 is generally cheaper to run than just MV_EXPAND b | SORT a | LIMIT 3.

[kanoshiou]

Thank you for your review @alex-spies! I've made several changes based on your feedback. However, I'm still finding it challenging to handle cases where there are query plan nodes between two TopN. Any guidance on how to approach these scenarios would be greatly appreciated.

From the examples you provided in your comment, removing the second TopN in such cases could introduce performance issues (except for nodes like DROP and KEEP, perhaps?) or even alter query semantics. In your experience, under what conditions can we safely remove the redundant TopN without compromising correctness or efficiency?

[kanoshiou]

Hi @alex-spies, I’ve updated the branch. The optimization now removes redundant child TopN when specific plans appear between this TopN and its parent TopN.