Skip to content

Conversation

@julian-elastic
Copy link
Contributor

@julian-elastic julian-elastic commented Sep 18, 2025

Implement two performance improvements for Lookup Join on Expression

  1. Do not send duplicate columns more than once to the lookup node. Example column left_id will be just sent once instead of 3 times, even though it is used 3 times in the expression
  2. Use the termsQueryList that we build for field based join for Expression Based Joins too, if it uses equals operator. This allows us to take advantage of the optimizations made for field based join and apply them for expression based join whenever possible. We end up doing less work per row, so we expect this to be faster.

Covered by existing UTs lookup-join-expression.LookupJoinOnSameLeftFieldTwice and multiple UTs on equals.

Benchmarks show only moderate improvement 50sec to 43sec (14% improvement) in esql_lookup_join_expression_eq_100k_keys_where_few_matching_like_limit_10000. See #135036 (comment)

@elasticsearchmachine elasticsearchmachine added needs:triage Requires assignment of a team area label v9.2.0 labels Sep 18, 2025
@julian-elastic julian-elastic self-assigned this Sep 18, 2025
@julian-elastic julian-elastic added >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL and removed needs:triage Requires assignment of a team area label labels Sep 18, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@elasticsearchmachine
Copy link
Collaborator

Hi @julian-elastic, I've created a changelog YAML for you.

@julian-elastic
Copy link
Contributor Author

Buildkite benchmark this with esql-joins please

1 similar comment
@julian-elastic
Copy link
Contributor Author

Buildkite benchmark this with esql-joins please

@julian-elastic
Copy link
Contributor Author

Buildkite benchmark this with esql-joins please

@julian-elastic
Copy link
Contributor Author

Buildkite benchmark this with esql-joins please

@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 25, 2025

💚 Build Succeeded

This build ran two esql-joins benchmarks to evaluate performance impact of this PR.

History

cc @julian-elastic

@julian-elastic julian-elastic merged commit 89bb134 into elastic:main Sep 26, 2025
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.2.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants