[ES|QL] Non-Correlated Subquery in FROM command #135744

fang-xing-esql · 2025-09-30T22:22:54Z

This PR enables support for non-correlated subqueries within the FROM command. Related to https://github.com/elastic/esql-planning/issues/89

A non-correlated subquery in this context is one that is fully self-contained and does not reference attributes from the outer query. Enabling support for these subqueries in the FROM command provides an additional way to define a data source, beyond directly specifying index patterns in an ES|QL query.

Example

FROM index1, (FROM index2
              | WHERE a > 10
              | EVAL b = a * 2
              | STATS cnt = COUNT(*) BY c
              | SORT cnt desc
              | LIMIT 10)
, index3, (FROM index4 
           | stats count(*))
| WHERE d > 10
| STATS max = max(*) BY e
| SORT max desc

This feature is built on top of Fork. Subqueries are processed in a manner similar to how Fork operates today, with modifications made to the following components to support this functionality:

Grammar: FROM_MODE is updated to support subquery syntax.
Parser: LogicalPlanBuilder creates a UnionAll logical plan on top of multiple data sources. Each data source can be either index patterns or subqueries. UnionAll extends Fork, but unlike Fork, each UnionAll leg may fetch data from different indices—this is one of the key differences between UnionAll and Fork.
PreAnalyzer: Extracts index patterns from subqueries and issues fieldcaps calls to build an IndexResolution for each subquery.
Analyzer: Resolves indices referenced by subqueries and handles union-typed fields referenced within them. Since subquery index patterns and main query index patterns are accessed separately behind each UnionAll leg, InvalidMappedField are not created across them. If conversion functions are required for common fields between the main index and subquery indices, those conversion functions must be pushed down into each UnionAll leg.
LogicalPlanOptimizer: Pushes down eligible filters/predicates from the main query into subqueries. This is another key distinction between UnionAll and Fork, as predicate pushdown applies only to UnionAll, while Fork remains unchanged.

Restrictions and follow ups to be addressed in the next PRs:

CCS: Grammar and parser don't block remote clusters, however subqueries with CCS is not fully tested yet. This is tracked in [ES|QL] Support remote cluster in subqueries in from command #136035.
Nested subqueries: Most of the nested subqueries are not allowed at the moment, verifier of LogicalPlanOptimizer will error out, if the subquery has commands besides FROM command. This is tracked in [ES|QL] Allow nested non-correlated subqueries in from command #136034.
Improve FieldNameUtils.resolveFieldNames to identify subquery field names for field caps call, instead of using all fields *.
Documentation related to subqueries.

elasticsearchmachine · 2025-09-30T22:23:48Z

Hi @fang-xing-esql, I've created a changelog YAML for you.

elasticsearchmachine · 2025-09-30T22:28:08Z

Hi @fang-xing-esql, I've created a changelog YAML for you.

Copilot

Pull Request Overview

This PR introduces support for non-correlated subqueries within the FROM command in ES|QL, allowing queries to reference multiple data sources including both index patterns and subqueries. The implementation enables subqueries to be processed similarly to Fork operations, with key distinctions in index resolution and predicate pushdown capabilities.

Adds grammar and parser support for subquery syntax in FROM commands
Implements UnionAll logical plan to handle mixed index patterns and subqueries
Enables predicate pushdown optimization specifically for UnionAll operations

Reviewed Changes

Copilot reviewed 36 out of 39 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
EsqlBaseParser.g4	Updates grammar to support subquery syntax in FROM_MODE
LogicalPlanBuilder.java	Creates UnionAll plans and handles subquery/index pattern combinations
UnionAll.java	New logical plan extending Fork with union-typed field support
Subquery.java	New logical plan node representing subquery placeholders
Analyzer.java	Resolves subquery indices and handles union-typed fields
PushDownAndCombineFilters.java	Adds predicate pushdown optimization for UnionAll
EsqlSession.java	Implements subquery index resolution during pre-analysis
Various test files	Adds comprehensive test coverage for subquery functionality

Comments suppressed due to low confidence (1)

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/SubqueryTests.java:1

There's a typo in "nested fork/subquery is not supported, it passes Analyzer" - should be "nested fork/subquery is not supported; it passes Analyzer" (semicolon instead of comma for better grammar).

/*

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-02T13:46:39Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java

+            }
+            return parent;
+        } else { // We should not reach here as the grammar does not allow it
+            throw new ParsingException("FROM is required in a subquery");


The error message "FROM is required in a subquery" is misleading since the grammar already enforces this requirement. Consider a more descriptive message like "Invalid subquery structure" or remove the comment and exception if this code path is truly unreachable.

Suggested change

throw new ParsingException("FROM is required in a subquery");

throw new ParsingException("Invalid subquery structure");

Copilot · 2025-10-02T13:46:40Z

...ain/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownAndCombineFilters.java

+            LogicalPlan newChild = switch (child) {
+                case Project project -> maybePushDownFilterPastProjectForUnionAllChild(pushable, project);
+                case Limit limit -> maybePushDownFilterPastLimitForUnionAllChild(pushable, limit);
+                default -> null; // TODO add a general push down for unexpected pattern


The TODO comment indicates incomplete functionality. Consider implementing the general push down logic or at least provide a more specific plan for when this will be addressed, as returning null could lead to silent failures in optimization.

Suggested change

default -> null; // TODO add a general push down for unexpected pattern

default -> {

// Fallback: unknown child type, do not push down filter for this child.

// Consider implementing general push down logic here in the future.

yield child;

}

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

fang-xing-esql · 2025-10-02T13:53:01Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java

        boolean supportsAggregateMetricDouble,
-        boolean supportsDenseVector
+        boolean supportsDenseVector,
+        Set<IndexPattern> subqueryIndices


Merging this subqueryIndices into the mainIndices is another option, it will require changes to EsqlCCSUtils.initCrossClusterState and EsqlCCSUtils.createIndexExpressionFromAvailableClusters, as they associate the ExecutionInfo with only one index pattern today.

fang-xing-esql · 2025-10-02T13:55:28Z

...ulti-clusters/src/javaRestTest/java/org/elasticsearch/xpack/esql/ccq/MultiClusterSpecIT.java

                hasCapabilities(adminClient(), List.of(ENABLE_FORK_FOR_REMOTE_INDICES.capabilityName()))
            );
        }
+        // Subqueries in FROM are not fully tested in CCS yet


When there is subquery exists in the query convertToRemoteIndices doesn't generate a correct remote index pattern yet, the query becomes invalid. Subqueries are not fully tested in CCS yet, working on it as a follow up.

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/SubqueryTests.java

elasticsearchmachine · 2025-10-02T15:16:52Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-10-02T15:16:52Z

Pinging @elastic/kibana-esql (ES|QL-ui)

x-pack/plugin/esql/qa/testFixtures/src/main/resources/subquery.csv-spec

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

julian-elastic · 2025-10-02T21:28:17Z

...ain/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownAndCombineFilters.java

+        // then the real child, if there is unknown pattern, keep the filter and UnionAll plan unchanged
+        List<LogicalPlan> newChildren = new ArrayList<>();
+        boolean changed = false;
+        for (LogicalPlan child : unionAll.children()) {


Why do you need to special handle based on Child type? Just put a filter on top and we already have rules for handling Filter pushdown?

The main reason that the children types are checked here is that I'd like to push the predicate closer to an EsRelation, so that the predicate has more chance to be pushed down to lucene. In this PushDownAndCombineFilters rule here, if the child is a limit, filters are not pushed further. However, AddImplicitForkLimit adds a limit to each fork/unionall child, and this limit might prevent us from pushing down the predicate to lucene.

The patterns checked here are what I have seen so far that's added by fork, sometimes the other logical planner rules may eliminate a project, or swap project and limit.

astefan

I went through the PR in part and would like to provide some input. I still have to go over the tests and still to understand some parts of the Analyzer.

Thank you for providing the detailed description of the PR. It helps a lot with the review.

FROM sample_data, (FROM employees metadata _id | sort _id) metadata _index | SORT emp_no desc | KEEP _index, emp_no, languages, _id

results in

Found 1 problem\nline 1:50: Unbounded SORT not supported yet [sort _id] please add a LIMIT

This seems to imply that the "default" limit that we usually add to queries is not added to subqueries.
IF this is an acceptable and agreed upon limitation, I think it would help to have it documented in the PR/docs.

FROM (FROM *) metadata _id, _index | SORT emp_no desc | KEEP _index, emp_no, languages, _id

results in

Cannot use field [emp_no] due to ambiguities being mapped as [2] incompatible types: [integer] in [employees], [long] in [employees_incompatible]",

but
FROM *, (FROM *) metadata _id, _index | SORT emp_no desc | KEEP _index, emp_no, languages, _id
doesn't complain. Is the first error valid?

Even FROM * metadata _id, _index | SORT emp_no desc | KEEP _index, emp_no, languages, _id complains.

Apologies if this is already covered, but I wanted to mention this not to forget about it. Since this is also about field_caps calls, using a filter in the request should be something we test for this functionality. As a regular user I would expect that filter to also apply to subqueries, and I think it does.

"query":"FROM *, (FROM * metadata _index) metadata _id, _index | SORT emp_no desc | KEEP _index,  emp_no, _id | stats count=count(*) by _index",
    "filter": {
        "bool": {
            "filter": [
                {
                    "exists": {
                        "field": "emp_no"
                    }
                }
            ]
        }
    }

I am wondering if this behavior is the expected one, because I couldn't tell tbh:

FROM employees, (FROM employees | eval x = emp_no::long), (FROM employees | eval x = emp_no::string) metadata _index | keep x, emp_no, _index
results in column "x" having all values as "null" while if I run
from employees | fork (eval x = emp_no::string) (eval x = emp_no::long) | keep x, emp_no
I get an error message

"Column [x] has conflicting data types in FORK branches: [LONG] and [KEYWORD]"

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/UnionAll.java

…and limit pushdown for subqueries

fang-xing-esql · 2025-10-07T03:08:10Z

Thank you for reviewing @astefan! I replied below.

FROM sample_data, (FROM employees metadata _id | sort _id) metadata _index | SORT emp_no desc | KEEP _index, emp_no, languages, _id

results in
Found 1 problem\nline 1:50: Unbounded SORT not supported yet [sort _id] please add a LIMIT
This seems to imply that the "default" limit that we usually add to queries is not added to subqueries. IF this is an acceptable and agreed upon limitation, I think it would help to have it documented in the PR/docs.

We should be able to do better here! Thanks for pointing this out, I realized that the limit from the main query can be pushed deeper down into the subqueries, allowing the limit + orderby to be transformed to topn. The changes previously made to PushDownAndCombineFilters is moved into a new rule - PushDownFilterAndLimitIntoUnionAll, specifically for UnionAll, and it is dedicated to pushing down filter and limit further down into UnionAll children. Hopefully it looks more clear and leave less footprints to existing rules.

FROM (FROM *) metadata _id, _index | SORT emp_no desc | KEEP _index, emp_no, languages, _id

results in
Cannot use field [emp_no] due to ambiguities being mapped as [2] incompatible types: [integer] in [employees], [long] in [employees_incompatible]",
but FROM *, (FROM *) metadata _id, _index | SORT emp_no desc | KEEP _index, emp_no, languages, _id doesn't complain. Is the first error valid?

Even FROM * metadata _id, _index | SORT emp_no desc | KEEP _index, emp_no, languages, _id complains.

FROM (FROM *) and FROM *, (FROM *) have slightly different semantics, the first query returns all documents, and the second query returns twice as many documents as the first query. This is kind of related to the 4th item below - how the multi typed fields/references across different UnionAll children are resolved. Ideally, I think it will be nice if we are able to make multi-typed fields/references work out of the box for subqueries - cast them to a common type when possible, or leave them to null if they only appear in the final(UnioAll) results and are not referenced in any other commands. That's the main reason that ResolveUnionTypesInUnionAll is added in Analyzer. It worths some discussions and clarifications for sure I think, to define the expected behavior.

Apologies if this is already covered, but I wanted to mention this not to forget about it. Since this is also about field_caps calls, using a filter in the request should be something we test for this functionality. As a regular user I would expect that filter to also apply to subqueries, and I think it does.
"query":"FROM *, (FROM * metadata _index) metadata _id, _index | SORT emp_no desc | KEEP _index,  emp_no, _id | stats count=count(*) by _index",
    "filter": {
        "bool": {
            "filter": [
                {
                    "exists": {
                        "field": "emp_no"
                    }
                }
            ]
        }
    }

That's a good point. I'll double check filters in the request, and add some tests round it, thanks for reminding me. Added a test here.

I am wondering if this behavior is the expected one, because I couldn't tell tbh:

FROM employees, (FROM employees | eval x = emp_no::long), (FROM employees | eval x = emp_no::string) metadata _index | keep x, emp_no, _index results in column "x" having all values as "null" while if I run from employees | fork (eval x = emp_no::string) (eval x = emp_no::long) | keep x, emp_no I get an error message
"Column [x] has conflicting data types in FORK branches: [LONG] and [KEYWORD]"

idegtiarenko · 2025-10-07T07:14:23Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java

+        * to the subquery indices set, if Analyzer doesn't find the subquery' indexResolution,
+        * it falls back to the main query's indexResolution
+         */
+        if (isLookup || isMainIndexPattern) {


Does it mean that lookups are not supported in in sub-queries for now?

We should support lookup joins inside subqueries. The lookup indices are loaded into PreAnalyzer.lookupIndices, and it contains the lookup indices from main query and subqueries.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java

idegtiarenko · 2025-10-07T07:36:09Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

+            mainExecutionInfo.skipOnFailurePredicate(),
+            mainExecutionInfo.includeCCSMetadata()
+        );
+        EsqlCCSUtils.initCrossClusterState(indicesExpressionGrouper, verifier.licenseState(), subqueryIndexPattern, subqueryExecutionInfo);


Is it correct that EsqlExecutionInfo needs to be copied because how it initializes the state for CC?
It has to change for CPS (we add remotes based on the field caps responce opposed to pre-initializing based on indicesExpressionGrouper). Possibly that is going to allow to use the same execution info here as well as report metadata from remotes used in views.

My understanding is that this makes an implicit assumption: what is in the main query will follow the main index pattern.
I'm thinking about LOOKUP JOIN in particular.
A query like

FROM idx,(from remote1:idx2) | LOOKUP JOIN lujo

will execute the join always on the coordinator cluster (ie. the JOIN won't be pushed to the subquery), and the absence of lujo index on remote1 won't result in a validation exception.
I'm not sure this is correct, but I don't think it introduces significant problems.

A more problematic query could be:

FROM remote1:idx1,remote2:idx2,(from remote3:idx3) | LOOKUP JOIN lujo

In this case, where will the JOIN be executed? Will the coordinator need to have lujo?

I know CCS is still an open point, but it would be good to have some design decisions on this.

Is it correct that EsqlExecutionInfo needs to be copied because how it initializes the state for CC? It has to change for CPS (we add remotes based on the field caps responce opposed to pre-initializing based on indicesExpressionGrouper). Possibly that is going to allow to use the same execution info here as well as report metadata from remotes used in views.

Yes, this is correct, the main reason of copying EsqlExecutionInfo is to avoid reusing it when it is already initialized with the main index pattern, if it is reused after it is initialized with the main index pattern, it errors out. If we can make EsqlExecutionInfo reusable, copying is not needed.

I'd like a bit more clarity on the execution model for remote subqueries (is there a design doc I could read?) - if we have a remote subquery, does it work like INLINE STATS, i.e. execute remote, bring results back to coordinator, treat them as constant table from now on - or is it continuing to execute the query remotely once exiting the subquery?

idegtiarenko · 2025-10-07T07:43:43Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

+        );
+        EsqlCCSUtils.initCrossClusterState(indicesExpressionGrouper, verifier.licenseState(), subqueryIndexPattern, subqueryExecutionInfo);
+
+        return EsqlCCSUtils.createIndexExpressionFromAvailableClusters(subqueryExecutionInfo);


The way we resolve expressions is going to change soon: we are going to resolve IndexPattern directly.
Lets coordinate how to integrate that change.

If we have multiple FROM patterns, it will probably make sense to create a common field-caps call? But having separate ones may be ok for starters too.

We should be careful with subqueryExecutionInfo though - for example, if some clusters are marked as skipped there, should they also be skipped in the main query? I am concerned we could get inconsistent results where one of the subqueries skips one set of clusters, another skips other clusters, and main query would skip third set of clusters, and it would be impossible to understand what happened in the result (and also report what happened, since we only have one _clusters in the output). Would be nice to document what should happen in such case.

luigidellaquila

Thanks @fang-xing-esql

I had a first quick look and I left a couple of comments.

I also tested some queries and I got some unexpected results. see below:

count seems to be inconsistent as soon as we add conditions to the subquery

from employees,(from employees) | stats count(*)

returns 100

while

from employees,(from employees | where emp_no > 0) | stats count(*)

returns 200.

sometimes the engine complains about nested subqueries being unsupported

This works just fine

from  (from idx,( from idx | sort foo) )

This doesn't work:

from  idx,(from idx,( from idx | sort foo) )

{
    "error": {
        "root_cause": [
            {
                "type": "verification_exception",
                "reason": "Found 1 problem\nline 1:12: Nested subqueries are not supported"
            }

unsupported types...?

Using CSV dataset

from employees,(from * | where true ) | stats count(*)

{
    "error": {
        "root_cause": [
            {
                "type": "verification_exception",
                "reason": "Found 3 problems\nline 1:1: EVAL does not support type [counter_long] as the return data type of expression [from *]\nline 1:1: EVAL does not support type [counter_long] as the return data type of expression [from *]\nline 1:1: EVAL does not support type [counter_double] as the return data type of expression [from *]"
            }

x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4

luigidellaquila · 2025-10-07T12:14:35Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

+            mainExecutionInfo.skipOnFailurePredicate(),
+            mainExecutionInfo.includeCCSMetadata()
+        );
+        EsqlCCSUtils.initCrossClusterState(indicesExpressionGrouper, verifier.licenseState(), subqueryIndexPattern, subqueryExecutionInfo);


My understanding is that this makes an implicit assumption: what is in the main query will follow the main index pattern.
I'm thinking about LOOKUP JOIN in particular.
A query like

FROM idx,(from remote1:idx2) | LOOKUP JOIN lujo

will execute the join always on the coordinator cluster (ie. the JOIN won't be pushed to the subquery), and the absence of lujo index on remote1 won't result in a validation exception.
I'm not sure this is correct, but I don't think it introduces significant problems.

A more problematic query could be:

FROM remote1:idx1,remote2:idx2,(from remote3:idx3) | LOOKUP JOIN lujo

In this case, where will the JOIN be executed? Will the coordinator need to have lujo?

I know CCS is still an open point, but it would be good to have some design decisions on this.

fang-xing-esql · 2025-10-07T16:50:53Z

Thanks for reviewing @luigidellaquila ! I replied below.

count seems to be inconsistent as soon as we add conditions to the subquery
from employees,(from employees) | stats count(*)
returns 100

while
from employees,(from employees | where emp_no > 0) | stats count(*)
returns 200.

This makes me re-think about whether merging subquery index pattern into the main index pattern in parser is a good choice as an optimization, it seems like we have a good reason not to do it. @astefan mentioned a similar query pattern. I'll modify parser and not do this subquery merging. Changed here.

Currently, when there are duplicate index patterns in the main FROM command, no duplicate rows are returned (as shown in Q1 below). However, in the case of subqueries (Q3), the semantics are slightly different — and since this is a new behavior, we have some flexibility in defining what makes the most sense. In this context, a UnionAll is built on top of the main index pattern and the subqueries, so all rows from both the main index pattern and the subqueries are returned.

Q1
+ curl -u elastic:password -X POST 'localhost:9200/_query?format=txt&pretty' -H 'Content-Type: application/json' '-d
{
  "query": "from test1, test1"
}
'
         millis         |                                            nanos                                            |        num        
------------------------+---------------------------------------------------------------------------------------------+-------------------
2023-10-23T13:55:01.543Z|2023-10-23T13:55:01.543123456Z                                                               |1698069301543123456
2023-10-23T13:55:01.543Z|2023-10-23T12:55:01.543123456Z                                                               |1698069301543123456
1999-10-23T12:15:03.360Z|[2023-01-23T13:55:01.543123456Z, 2023-02-23T13:33:34.937193Z, 2023-03-23T12:15:03.360103847Z]|0                  

Q2
+ curl -u elastic:password -X POST 'localhost:9200/_query?format=txt&pretty' -H 'Content-Type: application/json' '-d
{
  "query": "from (from test1)"
}
'
         millis         |                                            nanos                                            |        num        
------------------------+---------------------------------------------------------------------------------------------+-------------------
2023-10-23T13:55:01.543Z|2023-10-23T13:55:01.543123456Z                                                               |1698069301543123456
2023-10-23T13:55:01.543Z|2023-10-23T12:55:01.543123456Z                                                               |1698069301543123456
1999-10-23T12:15:03.360Z|[2023-01-23T13:55:01.543123456Z, 2023-02-23T13:33:34.937193Z, 2023-03-23T12:15:03.360103847Z]|0                  

Q3
+ curl -u elastic:password -X POST 'localhost:9200/_query?format=txt&pretty' -H 'Content-Type: application/json' '-d
{
  "query": "from test1, (from test1)"
}
'
         millis         |                                            nanos                                            |        num        
------------------------+---------------------------------------------------------------------------------------------+-------------------
2023-10-23T13:55:01.543Z|2023-10-23T13:55:01.543123456Z                                                               |1698069301543123456
2023-10-23T13:55:01.543Z|2023-10-23T12:55:01.543123456Z                                                               |1698069301543123456
1999-10-23T12:15:03.360Z|[2023-01-23T13:55:01.543123456Z, 2023-02-23T13:33:34.937193Z, 2023-03-23T12:15:03.360103847Z]|0                  
2023-10-23T13:55:01.543Z|2023-10-23T13:55:01.543123456Z                                                               |1698069301543123456
2023-10-23T13:55:01.543Z|2023-10-23T12:55:01.543123456Z                                                               |1698069301543123456
1999-10-23T12:15:03.360Z|[2023-01-23T13:55:01.543123456Z, 2023-02-23T13:33:34.937193Z, 2023-03-23T12:15:03.360103847Z]|0

sometimes the engine complains about nested subqueries being unsupported

This works just fine

from  (from idx,( from idx | sort foo) )

This doesn't work:

from  idx,(from idx,( from idx | sort foo) )

{
   "error": {
       "root_cause": [
           {
               "type": "verification_exception",
               "reason": "Found 1 problem\nline 1:12: Nested subqueries are not supported"
           }

The first query with subqueries can be flattened at parsing time, as the main FROM command doesn't reference any index pattern directly, and there is only one subquery as the immediate child of the main FROM, so that it becomes non-nested subquery. However the second query is slightly different - with an index pattern and a subquery as the immediate children of the main FROM command, and it cannot be flattened to non-nested subquery for now. We have a plan to allow nested subqueries as the next step.

unsupported types...?

Using CSV dataset

from employees,(from * | where true ) | stats count(*)

{
   "error": {
       "root_cause": [
           {
               "type": "verification_exception",
               "reason": "Found 3 problems\nline 1:1: EVAL does not support type [counter_long] as the return data type of expression [from *]\nline 1:1: EVAL does not support type [counter_long] as the return data type of expression [from *]\nline 1:1: EVAL does not support type [counter_double] as the return data type of expression [from *]"
           }

Thank you for catching this! It is related to how the time-series data types are supported out of the TS context. These counter types are not marked as representable, and if they appear in the EVAL command, we will see this error.This commit, cast the counter types to their corresponding representable numeric types, and also added some queries on time-series indices under the FROM context. The TS context is excluded from this PR, however we do need to deal with time-series data types in a nice way in subqueries.

…est filter

astefan

Unrelated strictly to this PR and a brain dump, take it more like fyi and food for thought. CC @quackaplop.

A "meta" aspect as the more I look at code the more I start wondering about the high level differences between fork and union:
(from the PR description)

Pushes down eligible filters/predicates from the main query into subqueries. This is another key distinction between UnionAll and Fork, as predicate pushdown applies only to UnionAll, while Fork remains unchanged.

Why is that? I mean why wouldn't filters be pushed down for fork as well like for union?
Imo, pushing down filters and properly resolving union types (fork does some work around properly merging attributes having the same name) could have been done as a separate PR. And the current PR could have offered functionality on par with fork.

I keep coming back to fork because I feel like there could be many similarities between the two and, yet, union has a lot more than fork (in terms of functionality) in this PR.

As an user, I see as a main difference between fork and union the fact that union can go to different index patterns, while fork does everything on one index pattern. But apart from that, in my head the two should be identical in behavior. This raises some expectations with users, who could move from fork (because of its limited index pattern usage) to union and if, in this usage change, the UX has differences, then there will be questions/frustrations and limitations.

Another big difference I see is the presence of _fork: I kept adding metadata _index to my tests to learn where some data is coming from... I don't know if other users will feel the same urge as me, but I kind of liked the _fork column that fork is adding.

Also, what incentive would an user have to move from union to fork? Performance?

x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4

astefan · 2025-10-08T07:26:07Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

+            // build a map of UnionAll output to a list of LogicalPlan that reference this output
+            Map<Attribute, List<LogicalPlan>> outputToPlans = outputToPlans(unionAll, plan);
+
+            List<List<Attribute>> outputs = unionAll.children().stream().map(LogicalPlan::output).toList();


Wouldn't Fork.outputUnion() do the same thing as this line?

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

fang-xing-esql · 2025-10-09T03:01:30Z

Also, what incentive would an user have to move from union to fork? Performance?

Yeah, the main motivation of predicate pushdown into UnionAll is for performance, so that we could have the chance to fetch less documents from each UnionAll branch or even from Lucene. Fork has something to do with scores, predicate pushdown might affect the scores, that is one reason that I can think of not doing it for Fork.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

astefan

I am done with the review.

LGTM.

I think the amount of work and analysis that went into this PR is amazing. It took me a lot of time to go through almost everything. Any aspects that I am mentioning below or I mentioned previously can be addressed in follow up issues/PRs.

    FROM books,(FROM books | SORT score DESC, author | LIMIT 5 | KEEP author, score),(FROM books | STATS total = COUNT(*)) METADATA _score 
            | WHERE author:"Faulkner"
            | EVAL score = round(_score, 2)
            | FORK (SORT score DESC, author | LIMIT 5 | KEEP author, score)
                (STATS total = COUNT(*))
            | SORT _fork, score DESC, author

AssertionError: Nested FORKs are not yet supported
        at org.elasticsearch.xpack.esql.session.FieldNameUtils.lambda$resolveFieldNames$8(FieldNameUtils.java:118)
        at org.elasticsearch.xpack.esql.core.tree.Node.forEachDownMayReturnEarly(Node.java:90)
        at org.elasticsearch.xpack.esql.core.tree.Node.forEachDownMayReturnEarly(Node.java:98)
        at org.elasticsearch.xpack.esql.core.tree.Node.forEachDownMayReturnEarly(Node.java:84)
        at org.elasticsearch.xpack.esql.session.FieldNameUtils.resolveFieldNames(FieldNameUtils.java:222)
        at org.elasticsearch.xpack.esql.session.EsqlSession.analyzedPlan(EsqlSession.java:441)
        at org.elasticsearch.xpack.esql.session.EsqlSession.execute(EsqlSession.java:192)

FROM books,(FROM books METADATA _score | SORT score DESC, author | LIMIT 5 | KEEP author, score),(FROM books METADATA _score | STATS total = COUNT(*)) METADATA _score 
            | WHERE author:"Faulkner"
            | EVAL score = round(_score, 2)
            | SORT _fork, score DESC, author

Gives this error only: line 2:51: Unknown column [score], did you mean [_score]? even though there are other issues with the query.
Fixing this error by using sort _score (instead of sort score) leads me to another error: line 5:20: Unknown column [_fork], did you mean [_score]?

ES|QL does report verification errors all in one message.

FROM books,(FROM books METADATA _score | SORT _score DESC, author | LIMIT 5 | KEEP author, _score),(FROM books METADATA _score | STATS total = COUNT(*)) METADATA _score 
            | WHERE author:"Faulkner"
            | EVAL score = round(_score, 2)
            | SORT score DESC, author

Found 2 problems\nline 3:15: [:] operator cannot be used after FROM\nline 3:21: [:] operator cannot operate on [author], which is not a field from an index mapping

The second error ^ is incorrect: author is in fact a field from an index mapping (from all three (sub)queries).

    FROM employees, (from employees | where gender == "F" | keep emp_no, gender), (from employees | where languages > 3 | keep emp_no, languages)
    | where emp_no > 10050
    | keep emp_no, gender, languages

This query asks for all fields from field_caps (*), even though the query looks "simple" :-)). Wondering if we could do better here.
I discovered this during the logical plan optimization step where we create something like this:

\_Eval[[null[LONG] AS avg_worked_seconds#1636, null[DATETIME] AS birth_date#1637, null[KEYWORD] AS first_name#1638, null[DOUBLE] AS height#1639, null[DOUBLE] AS height.float#1640, null[DOUBLE] AS height.half_float#1641, null[DOUBLE] AS height.scaled_float#1642, null[DATETIME] AS hire_date#1643, null[BOOLEAN] AS is_rehired#1644, null[KEYWORD] AS job_positions#1645, null[INTEGER] AS languages#1646, null[INTEGER] AS languages.byte#1647, null[LONG] AS languages.long#1648, null[INTEGER] AS languages.short#1649, null[KEYWORD] AS last_name#1650, null[INTEGER] AS salary#1651, null[DOUBLE] AS salary_change#1652, null[INTEGER] AS salary_change.int#1653, null[KEYWORD] AS salary_change.keyword#1654, null[LONG] AS salary_change.long#1655, null[BOOLEAN] AS still_hired#1656]] = ull[DOUBLE] AS height#1639, null[DOUBLE] AS height.float#1640, null[DOUBLE] AS height.half_float#1641, null[DOUBLE] AS height.scaled_float#1642, null[DATETIME] AS hire_date#1643, null[BOOLEAN] AS is_rehired#1644, null[KEYWORD] AS job_positions#1645, null[INTEGER] AS languages#1646, null[INTEGER] AS languages.byte#1647, null[LONG] AS languages.long#1648, null[INTEGER] AS languages.short#1649, null[KEYWORD] AS last_name#1650, null[INTEGER] AS salary#1651, null[DOUBLE] AS salary_change#1652, null[INTEGER] AS salary_change.int#1653, null[KEYWORD] AS salary_change.keyword#1654, null[LONG] AS salary_change.long#1655, null[BOOLEAN] AS still_hired#1656]]

astefan · 2025-10-10T11:09:34Z

...lasticsearch/xpack/esql/optimizer/rules/logical/PushDownFilterAndLimitIntoUnionAllTests.java

+     *   |     \_Limit[1000[INTEGER],false]
+     *   |       \_Filter[languages{f}#19 &gt; 0[INTEGER] AND emp_no{f}#16 &gt; 10000[INTEGER]]
+     *   |         \_EsRelation[test1][_meta_field{f}#22, emp_no{f}#16, first_name{f}#17, ..]
+     *   \_LocalRelation[[_meta_field{r}#33, emp_no{r}#34, first_name{r}#35, gender{r}#36, hire_date{r}#37, job{r}#38, job.raw{r}#39, l


This LocalRelation here is a placeholder for the branch from languages...., because the filter act on the field that belong to the other branches and basically eliminates completely languages?

It is a bit confusing that this LocalRelation doesn't only hold the fields that belong to languages index only.

luigidellaquila

Thanks @fang-xing-esql, that looks pretty good.
I left another round of comments, most of them are very minor changes, but one needs to be fixed before moving forward

luigidellaquila · 2025-10-13T09:22:11Z

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

    }
+
+    public void testNestedSubqueries() {
+        VerificationException e = expectThrows(VerificationException.class, () -> planSubquery("""


This needs a capability check (otherwise it will fail in non-snapshot)

luigidellaquila · 2025-10-13T09:22:16Z

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

+    }
+
+    public void testForkInSubquery() {
+        VerificationException e = expectThrows(VerificationException.class, () -> planSubquery("""


luigidellaquila · 2025-10-13T09:30:39Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

            }

-            return changed ? new Fork(fork.source(), newSubPlans, newOutput) : fork;
+            return fork instanceof UnionAll unionAll


nit: to avoid the instanceof, this could be a new polymorphic method withSubplansAndOutput(newSubPlans, newOutput)

luigidellaquila · 2025-10-13T09:44:30Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

+     */
+    private static class ResolveUnionTypesInUnionAll extends Rule<LogicalPlan, LogicalPlan> {
+        // The mapping between explicit conversion functions and the corresponding attributes in the UnionAll output
+        private Map<AbstractConvertFunction, Attribute> convertFunctionsToAttributes;


This needs some refactoring:
the rules are treated as singletons (see private static final List<Batch<LogicalPlan>> RULES, line 202), so they have to be stateless

luigidellaquila · 2025-10-13T09:49:30Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java

-                indexPattern.set(p.indexPattern());
+                if (mainAndSubqueryIndices.isEmpty()) { // the index pattern from main query is always the first to be seen
+                    mainAndSubqueryIndices.add(p.indexPattern());
+                } else if (EsqlCapabilities.Cap.SUBQUERY_IN_FROM_COMMAND.isEnabled()) { // collect subquery index patterns


nit: Do you need a capability check here? Isn't it already guarded by the grammar?

luigidellaquila · 2025-10-13T09:54:45Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/telemetry/FeatureMetric.java

    COMPLETION(Completion.class::isInstance),
-    SAMPLE(Sample.class::isInstance);
+    SAMPLE(Sample.class::isInstance),
+    SUBQUERY(Subquery.class::isInstance);


👍
This is not properly a command, but ++ to tracking it

non-correlated subquery in from command

193d8f7

fang-xing-esql added >enhancement :Analytics/ES|QL AKA ESQL ES|QL-ui Impacts ES|QL UI labels Sep 30, 2025

elasticsearchmachine added the v9.2.0 label Sep 30, 2025

Merge branch 'main' into subquery-in-from

0c5b79d

fang-xing-esql force-pushed the subquery-in-from branch from 8a72832 to 0c5b79d Compare September 30, 2025 22:26

Update docs/changelog/135744.yaml

440e501

fang-xing-esql added the test-release Trigger CI checks against release build label Sep 30, 2025

fang-xing-esql and others added 5 commits September 30, 2025 19:30

Merge branch 'main' into subquery-in-from

61c7266

cleanup

20e543d

update tests

f7d695d

subquery serialization

fb221c9

fix predicate push down

a38f6b9

elasticsearchmachine added v9.3.0 and removed v9.2.0 labels Oct 2, 2025

fang-xing-esql requested a review from Copilot October 2, 2025 13:45

Copilot AI reviewed Oct 2, 2025

View reviewed changes

fang-xing-esql commented Oct 2, 2025

View reviewed changes

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/SubqueryTests.java Show resolved Hide resolved

fang-xing-esql marked this pull request as ready for review October 2, 2025 15:16

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Oct 2, 2025

julian-elastic reviewed Oct 2, 2025

View reviewed changes

x-pack/plugin/esql/qa/testFixtures/src/main/resources/subquery.csv-spec Show resolved Hide resolved

julian-elastic reviewed Oct 2, 2025

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java Show resolved Hide resolved

julian-elastic reviewed Oct 2, 2025

View reviewed changes

fang-xing-esql and others added 3 commits October 3, 2025 15:40

Merge branch 'main' into subquery-in-from

99fc31d

[CI] Auto commit changes from spotless

c489ab1

Merge branch 'main' into subquery-in-from

71d15e1

astefan reviewed Oct 6, 2025

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java Outdated Show resolved Hide resolved

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/UnionAll.java Outdated Show resolved Hide resolved

This was referenced Oct 6, 2025

[ES|QL] Allow nested non-correlated subqueries in from command #136034

Open

[ES|QL] Support remote cluster in subqueries in from command #136035

Open

create a new rule PushDownFilterAndLimitIntoUnionAll to do predicate …

7cc94a5

…and limit pushdown for subqueries

fang-xing-esql and others added 2 commits October 7, 2025 00:23

update according to review comments

30c2295

Merge branch 'main' into subquery-in-from

b31154f

idegtiarenko reviewed Oct 7, 2025

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java Outdated Show resolved Hide resolved

idegtiarenko reviewed Oct 7, 2025

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java Outdated Show resolved Hide resolved

idegtiarenko reviewed Oct 7, 2025

View reviewed changes

luigidellaquila reviewed Oct 7, 2025

View reviewed changes

remove subquery merging in parser, add tests for subqueries with requ…

f06b7a2

…est filter

astefan reviewed Oct 8, 2025

View reviewed changes

idegtiarenko reviewed Oct 8, 2025

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java Outdated Show resolved Hide resolved

fang-xing-esql added 2 commits October 8, 2025 22:17

cast counter types to the corresponding normal numeric types

b170df7

Merge branch 'main' into subquery-in-from

71171e1

idegtiarenko reviewed Oct 9, 2025

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java Outdated Show resolved Hide resolved

fang-xing-esql and others added 3 commits October 9, 2025 08:25

stablize tests

b501bc8

Merge branch 'main' into subquery-in-from

9c333d3

code refactor and remove copy EsqlExecutionInfo

1853615

astefan approved these changes Oct 10, 2025

View reviewed changes

handle subquery with fork and full text function better

be82a74

luigidellaquila requested changes Oct 13, 2025

View reviewed changes

	throw new ParsingException("FROM is required in a subquery");
	throw new ParsingException("Invalid subquery structure");

-                default -> null; // TODO add a general push down for unexpected pattern
+                default -> {
+                    // Fallback: unknown child type, do not push down filter for this child.
+                    // Consider implementing general push down logic here in the future.
+                    yield child;
+                }

[ES|QL] Non-Correlated Subquery in FROM command #135744

Are you sure you want to change the base?

[ES|QL] Non-Correlated Subquery in FROM command #135744

Uh oh!

Conversation

fang-xing-esql commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Sep 30, 2025

Uh oh!

elasticsearchmachine commented Sep 30, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 2, 2025

Uh oh!

elasticsearchmachine commented Oct 2, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

astefan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fang-xing-esql commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

idegtiarenko Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luigidellaquila Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

luigidellaquila left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

luigidellaquila Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

fang-xing-esql commented Sep 30, 2025 •

edited

Loading

fang-xing-esql Oct 3, 2025 •

edited

Loading

fang-xing-esql commented Oct 7, 2025 •

edited

Loading

idegtiarenko Oct 7, 2025 •

edited

Loading

luigidellaquila Oct 7, 2025 •

edited

Loading

luigidellaquila Oct 7, 2025 •

edited

Loading

fang-xing-esql commented Oct 7, 2025 •

edited

Loading

astefan Oct 10, 2025 •

edited

Loading