ESQL: Push filters on right hand-side of joins #132635
Conversation
This adds the logic to push filters down the right hand-side of joins (excepting `InlineJoin`). This only happens for those filters that can be pushed down all the way to Lucene.
|
Hi @bpintea, I've created a changelog YAML for you. |
| return false; | ||
| } | ||
| // TODO: the flag isn't relevant for the pushdown decision? | ||
| LucenePushdownPredicates pushdownPredicates = LucenePushdownPredicates.from(ctx.searchStats(), new EsqlFlags(true)); |
There was a problem hiding this comment.
This is problematic - thanks @alex-spies for raising the obvious point - since the ctx doesn't contain the stats for the lookup index. Getting the lookup index stats in there on the data node seems doable, since this is done using the SearchService on the local node based on a ShardId; these are sent from coordinator in the DataNodeRequest. And on the coordinator, they are resolved based on the index name. Currently, lookup indices are filtered out from resolving this index name to shard ids mapping (PlannerUtils#forEachRelation).
However, even if this filter is removed, the lookup index shard ids need to be handled differently when selecting a data node to send a compute request to, since only the nodes with FROM indices need to be considered (and not those with a LOOKUP JOIN index).
This adds the logic to push filters down the right hand-side of joins (excepting
InlineJoin). This is done for those filters that can be pushed down all the way to Lucene only.Closes #130024