[ES|QL] Non-Correlated Subquery in FROM command #135744

fang-xing-esql · 2025-09-30T22:22:54Z

This PR enables support for non-correlated subqueries within the FROM command. Related to https://github.com/elastic/esql-planning/issues/89

A non-correlated subquery in this context is one that is fully self-contained and does not reference attributes from the outer query. Enabling support for these subqueries in the FROM command provides an additional way to define a data source, beyond directly specifying index patterns in an ES|QL query.

Example

FROM index1, (FROM index2
              | WHERE a > 10
              | EVAL b = a * 2
              | STATS cnt = COUNT(*) BY c
              | SORT cnt desc
              | LIMIT 10)
, index3, (FROM index4 
           | stats count(*))
| WHERE d > 10
| STATS max = max(*) BY e
| SORT max desc

This feature is built on top of Fork. Subqueries are processed in a manner similar to how Fork operates today, with modifications made to the following components to support this functionality:

Grammar: FROM_MODE is updated to support subquery syntax.
Parser: LogicalPlanBuilder creates a UnionAll logical plan on top of multiple data sources. Each data source can be either index patterns or subqueries. UnionAll extends Fork, but unlike Fork, each UnionAll leg may fetch data from different indices—this is one of the key differences between UnionAll and Fork.
PreAnalyzer: Extracts index patterns from subqueries and issues fieldcaps calls to build an IndexResolution for each subquery.
Analyzer: Resolves indices referenced by subqueries and handles union-typed fields referenced within them. Since subquery index patterns and main query index patterns are accessed separately behind each UnionAll leg, InvalidMappedField are not created across them. If conversion functions are required for common fields between the main index and subquery indices, those conversion functions must be pushed down into each UnionAll leg.
LogicalPlanOptimizer: Pushes down eligible filters/predicates from the main query into subqueries. This is another key distinction between UnionAll and Fork, as predicate pushdown applies only to UnionAll, while Fork remains unchanged.

Restrictions and follow ups to be addressed in the next PRs:

CCS: Grammar and parser don't block remote clusters, however subqueries with CCS is not fully tested yet. This is tracked in [ES|QL] Support remote cluster in subqueries in from command #136035.
Nested subqueries: Most of the nested subqueries are not allowed at the moment, verifier of LogicalPlanOptimizer will error out, if the subquery has commands besides FROM command. This is tracked in [ES|QL] Allow nested non-correlated subqueries in from command #136034.
Improve FieldNameUtils.resolveFieldNames to identify subquery field names for field caps call, instead of using all fields *. [ES|QL] Improve FieldNameUtils.resolveFieldNames to identify subquery field names for field caps call, instead of using all fields * #137283
Improve the error messages when a full text function cannot be applied to a subquery.
When a predicate from the main query eliminates a subquery, the LocalRelation created by PruneFilters includes all of the output of the subquery, and the output is the superset of the outputs from each subquery, which looks confusing, the outputs that are not directly from the current subquery can be excluded from the LocalRelation.
Push down Filters/Predicates on mixed data type UnionAll output with explicit casting. [ES|QL] Push down Filters/Predicates on mixed data type UnionAll output with explicit casting. #137284
Implicit casting for mixed data type UnionAll output
Documentation related to subqueries.
Add new performance tests for subqueries. [E|QL] Add performance tests to subqueries #137285
Remove implicit LIMIT appended to each subquery(UnionAll branch). [ES|QL] Remove the implicit LIMIT command added to each subquery (UnionAll branch) #138106

elasticsearchmachine · 2025-09-30T22:23:48Z

Hi @fang-xing-esql, I've created a changelog YAML for you.

elasticsearchmachine · 2025-09-30T22:28:08Z

Hi @fang-xing-esql, I've created a changelog YAML for you.

Copilot

Pull Request Overview

This PR introduces support for non-correlated subqueries within the FROM command in ES|QL, allowing queries to reference multiple data sources including both index patterns and subqueries. The implementation enables subqueries to be processed similarly to Fork operations, with key distinctions in index resolution and predicate pushdown capabilities.

Adds grammar and parser support for subquery syntax in FROM commands
Implements UnionAll logical plan to handle mixed index patterns and subqueries
Enables predicate pushdown optimization specifically for UnionAll operations

Reviewed Changes

Copilot reviewed 36 out of 39 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
EsqlBaseParser.g4	Updates grammar to support subquery syntax in FROM_MODE
LogicalPlanBuilder.java	Creates UnionAll plans and handles subquery/index pattern combinations
UnionAll.java	New logical plan extending Fork with union-typed field support
Subquery.java	New logical plan node representing subquery placeholders
Analyzer.java	Resolves subquery indices and handles union-typed fields
PushDownAndCombineFilters.java	Adds predicate pushdown optimization for UnionAll
EsqlSession.java	Implements subquery index resolution during pre-analysis
Various test files	Adds comprehensive test coverage for subquery functionality

Comments suppressed due to low confidence (1)

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/SubqueryTests.java:1

There's a typo in "nested fork/subquery is not supported, it passes Analyzer" - should be "nested fork/subquery is not supported; it passes Analyzer" (semicolon instead of comma for better grammar).

/*

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-02T13:46:39Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java

+            }
+            return parent;
+        } else { // We should not reach here as the grammar does not allow it
+            throw new ParsingException("FROM is required in a subquery");


The error message "FROM is required in a subquery" is misleading since the grammar already enforces this requirement. Consider a more descriptive message like "Invalid subquery structure" or remove the comment and exception if this code path is truly unreachable.

Suggested change

throw new ParsingException("FROM is required in a subquery");

throw new ParsingException("Invalid subquery structure");

Copilot · 2025-10-02T13:46:40Z

...ain/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownAndCombineFilters.java

+            LogicalPlan newChild = switch (child) {
+                case Project project -> maybePushDownFilterPastProjectForUnionAllChild(pushable, project);
+                case Limit limit -> maybePushDownFilterPastLimitForUnionAllChild(pushable, limit);
+                default -> null; // TODO add a general push down for unexpected pattern


The TODO comment indicates incomplete functionality. Consider implementing the general push down logic or at least provide a more specific plan for when this will be addressed, as returning null could lead to silent failures in optimization.

Suggested change

default -> null; // TODO add a general push down for unexpected pattern

default -> {

// Fallback: unknown child type, do not push down filter for this child.

// Consider implementing general push down logic here in the future.

yield child;

}

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

fang-xing-esql · 2025-10-02T13:53:01Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java

        boolean supportsAggregateMetricDouble,
-        boolean supportsDenseVector
+        boolean supportsDenseVector,
+        Set<IndexPattern> subqueryIndices


Merging this subqueryIndices into the mainIndices is another option, it will require changes to EsqlCCSUtils.initCrossClusterState and EsqlCCSUtils.createIndexExpressionFromAvailableClusters, as they associate the ExecutionInfo with only one index pattern today.

fang-xing-esql · 2025-10-02T13:55:28Z

...ulti-clusters/src/javaRestTest/java/org/elasticsearch/xpack/esql/ccq/MultiClusterSpecIT.java

                hasCapabilities(adminClient(), List.of(ENABLE_FORK_FOR_REMOTE_INDICES.capabilityName()))
            );
        }
+        // Subqueries in FROM are not fully tested in CCS yet


When there is subquery exists in the query convertToRemoteIndices doesn't generate a correct remote index pattern yet, the query becomes invalid. Subqueries are not fully tested in CCS yet, working on it as a follow up.

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/SubqueryTests.java

elasticsearchmachine · 2025-10-02T15:16:52Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-10-02T15:16:52Z

Pinging @elastic/kibana-esql (ES|QL-ui)

x-pack/plugin/esql/qa/testFixtures/src/main/resources/subquery.csv-spec

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

...ain/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownAndCombineFilters.java

luigidellaquila

Thanks @fang-xing-esql, that looks pretty good.
I left another round of comments, most of them are very minor changes, but one needs to be fixed before moving forward

...gin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LogicalPlanOptimizerTests.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/PreAnalyzer.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/telemetry/FeatureMetric.java

fang-xing-esql · 2025-10-14T00:10:25Z

Thank you so much for reviewing this PR in such depth, I really appreciate the time and thought you put into it, @astefan !

I'll create a follow up issue for improving FieldNameUtils.resolveFieldNames, we can do a better job here to recognize field names referenced in subqueries, instead of sending * to the field caps call., thank you for reminding me about it!

I made a couples of changes and added additional tests related to subqueries combined with fork and full-text functions, both in the main query and within subqueries. Initially, I planned to address these in a separate PR since this one is already quite large, but I think these scenarios are important enough to include here.

First, FieldNameUtils.resolveFieldNames currently asserts when encountering nested forks. For cases involving a subquery and a fork, this check has been deferred to LogicalVerifier, which now throws a VerificationException instead of an assertion error. as allowing nested subqueries and subquery + fork is planned as the next step.

Second, the validation of commands before a full-text function, as well as the verification of the field referenced by that function, is now deferred to LogicalVerifier when a UnionAll plan exists in its child plans. Since the output of a UnionAll plan is always a ReferenceAttribute, and certain full-text functions apply only to fields, deferring this validation allows the PushDownFilterAndLimitIntoUnionAll rule to push the full-text functions down into individual subqueries. This enables those validations to occur within the subquery context, increasing the chances that full-text functions can be properly evaluated.

fang-xing-esql · 2025-10-14T00:20:35Z

Thanks @fang-xing-esql, that looks pretty good. I left another round of comments, most of them are very minor changes, but one needs to be fixed before moving forward

Thank you so much for your thoughtful reviews @luigidellaquila! They really help this PR to become a better PR. I think I addressed all of them, please just let me know if there is anything that I missed.

luigidellaquila

Thanks @fang-xing-esql
There are still a couple of things I'd like to check in detail (the pushdown logic in particular), but for what I could see it looks good, so I'm approving to unblock the merge.

luigidellaquila · 2025-10-14T08:58:47Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

+                return DATE_NANOS;
+            }
+
+            if (t1.isCounter()) {


nit: I'd expect this logic to be in EsqlDataTypeConverter.commonType() (that you are using below btw).
The only logical difference is noCounter().

Maybe commonTypes() could just do

for (List<Attribute> out : outputs) { type = EsqlDataTypeConverter.commonType(type, out.get(i).dataType()).noCounter(); }

bpintea

Started my journey on this PR. Minor/optional notes only.

x-pack/plugin/esql/src/main/antlr/EsqlBaseParser.g4

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/parser/LogicalPlanBuilder.java

...org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownFilterAndLimitIntoUnionAll.java

...lasticsearch/xpack/esql/optimizer/rules/logical/PushDownFilterAndLimitIntoUnionAllTests.java

bpintea · 2025-10-15T16:52:17Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/Subquery.java

It'd be great to have some javadoc for this class.
(I would seem it's mostly useful in PushDownFilterAndLimitIntoUnionAll, where we first push "below" UnionAll and then once there, below Subquery?
Seems like the LogicalPlanBuilder#visitRelation always places the index patern first, followed by the subqueries.
I'm wondering thus if this marker node is really needed. Wouldn't these push downs be doable on each branch by existing rules, if Subquery wasn't there?
But still to look into it and didn't grasp it all yet. Javadoc would be great in any case, though. :) )

There are a couple of reasons that Subquery node exists in the plan.

First, originally I added this node because in the future we may want to support qualifiers with subqueries(like FROM idx1, (FROM idx2, idx3) as idx23), and this is a place that we can store the name(qualifier) of the subqueries, that can be referenced by the parent query.

Second, when I started working on pushing down filters into subqueries, I realize PushDownAndCombineFilters does not push down filters below limit intentionally, refer to here, however UnionAll/Fork adds a implicit limit for each branch, so this rule doesn't help the predicate pushdown for subqueries. The Subquery node is also used as a pattern for predicate pushdown. I'll add more comments in the code.

bpintea

One issue I see that we could align is the treatment of the "union"/conflicting types: in case these emerge out of an index pattern, we return the type as unsupported, along with the original_types and a suggested_cast. User is in the know, all good.

With UnionAll, we return it of type keyword, even if no index in the union has the conflicting field of that type and null values, but with no other indication whatsoever that there's a conflict. The user won't know of the conflict, which I think is problematic, as this could occur frequently.

Don't know if we have a decision here or track this somewhere?

(Some more comments to follow, sorry :) )

bpintea · 2025-10-16T12:07:12Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

+            ThreadPool.Names.SEARCH_COORDINATION,
+            ThreadPool.Names.SYSTEM_READ
+        );
+        if (subqueryIndexPattern != null) {


Can this now ever happen? Shouldn't it be guaranteed by the grammar that it's never null?

The piece of code that processes subquery index patterns in EsqlSession will be in a separate PR from @craigtaverner, and it will be shared by subqueries and views, stay tuned :).

bpintea · 2025-10-16T12:07:33Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/session/EsqlSession.java

+            );
+
+        } else {
+            // occurs when dealing with local relations (row a = 1)


This isn't currently supported by the grammar, no?

bpintea · 2025-10-16T12:34:25Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

+                    && indexResolution.matches(indexPattern) == false
+                    && context.subqueryResolution().isEmpty() == false) {
+                    // index pattern does not match main index
+                    indexResolution = context.subqueryResolution().getOrDefault(indexPattern, indexResolution);


Suggested change

indexResolution = context.subqueryResolution().getOrDefault(indexPattern, indexResolution);

indexResolution = context.subqueryResolution().get(indexPattern);

if (indexResolution == null) {...

Wouldn't it be a bug if the simple get() would return a null? And wouldn't we introduce a new one, if an "unkown" pattern resolves then to the "main" index?

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

...rc/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/FullTextFunction.java

bpintea

I think there are some fixes needed on conflicting/union types from different branches. I've left a few notes, out of which one is, I think, a bug.

Potentially related to that: when running the query in that comment - with index1 only having "x": 1 and index2 having "x": "2" (i.e. both text and keyword fields) - I get a CCE:

[2025-10-17T12:49:31,742][ERROR][o.e.r.ChunkedRestResponseBodyPart] [runTask-0] failure encoding chunk java.lang.ClassCastException: class org.elasticsearch.compute.data.IntVectorBlock cannot be cast to class org.elasti
csearch.compute.data.LongBlock (org.elasticsearch.compute.data.IntVectorBlock and org.elasticsearch.compute.data.LongBlock are in unnamed module of loader java.net.FactoryURLClassLoader @11d2dd2d)
        at org.elasticsearch.xpack.esql.action.PositionToXContent$1.valueToXContent(PositionToXContent.java:72)
        at org.elasticsearch.xpack.esql.action.PositionToXContent.positionToXContent(PositionToXContent.java:53)
        at org.elasticsearch.xpack.esql.action.ResponseXContentUtils.lambda$rowValues$7(ResponseXContentUtils.java:114)
        at [email protected]/org.elasticsearch.rest.ChunkedRestResponseBodyPart$1.encodeChunk(ChunkedRestResponseBodyPart.java:161)
        at [email protected]/org.elasticsearch.rest.RestController$EncodedLengthTrackingChunkedRestResponseBodyPart.encodeChunk(RestController.java:1012)
        at [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.writeChunk(Netty4HttpPipeliningHandler.java:436)
        at [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.doWriteChunkedResponse(Netty4HttpPipeliningHandler.java:263)
        at [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.doWrite(Netty4HttpPipeliningHandler.java:231)
        at [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.write(Netty4HttpPipeliningHandler.java:184)
        at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:891)
        at [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:956)
        at [email protected]/io.netty.channel.AbstractChannelHandlerContext$WriteTask.run(AbstractChannelHandlerContext.java:1263)
        at [email protected]/io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
        at [email protected]/io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
        at [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at [email protected]/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
        at [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
        at [email protected]/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:1575)

Also, something like:

FROM index1, (FROM index2) | EVAL l = x::LONG | EVAL i = x::INTEGER | KEEP x, l, i

fails with a verification exception:

"org.elasticsearch.xpack.esql.VerificationException: Found 1 problem\nline 130:47: Output has changed from [[x{r}#204, l{r}#191, i{r}#194]] to [[x{r}#204, l{r}#191, i{r}#194]].
	at org.elasticsearch.xpack.esql.optimizer.LogicalPlanOptimizer.optimize(LogicalPlanOptimizer.java:121)
	at org.elasticsearch.xpack.esql.session.EsqlSession.optimizedPlan(EsqlSession.java:947)
	at org.elasticsearch.xpack.esql.session.EsqlSession$1.lambda$onResponse$1(EsqlSession.java:228)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:406)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:326)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.addListener(SubscribableListener.java:222)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.lambda$andThen$1(SubscribableListener.java:534)
	at [email protected]/org.elasticsearch.action.ActionListener.run(ActionListener.java:465)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.newForked(SubscribableListener.java:138)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.andThen(SubscribableListener.java:534)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.andThen(SubscribableListener.java:489)
	at org.elasticsearch.xpack.esql.session.EsqlSession$1.onResponse(EsqlSession.java:228)
	at org.elasticsearch.xpack.esql.session.EsqlSession$1.onResponse(EsqlSession.java:219)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:406)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:326)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:355)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:262)
	at org.elasticsearch.xpack.esql.session.EsqlSession.analyzeWithRetry(EsqlSession.java:885)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$resolveIndices$14(EsqlSession.java:538)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:406)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:326)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:355)
	at [email protected]/org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:262)
	at org.elasticsearch.xpack.esql.session.EsqlSession.preAnalyzeSubqueryIndices(EsqlSession.java:554)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$preAnalyzeSubqueryIndices$15(EsqlSession.java:551)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at org.elasticsearch.xpack.esql.session.EsqlSession.lambda$preAnalyzeSubqueryIndex$16(EsqlSession.java:589)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:233)
	at org.elasticsearch.xpack.esql.session.IndexResolver.lambda$resolveAsMergedMapping$0(IndexResolver.java:100)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$ResponseWrappingActionListener.onResponse(ActionListenerImplementations.java:261)
	at [email protected]/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)
	at [email protected]/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:228)
	at [email protected]/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:222)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:350)
	at [email protected]/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)
	at [email protected]/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at [email protected]/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:111)
	at org.elasticsearch.xpack.esql.action.EsqlResolveFieldsAction.finishHim(EsqlResolveFieldsAction.java:338)
	at org.elasticsearch.xpack.esql.action.EsqlResolveFieldsAction.lambda$doExecuteForked$7(EsqlResolveFieldsAction.java:232)
	at [email protected]/org.elasticsearch.core.AbstractRefCounted$1.closeInternal(AbstractRefCounted.java:125)
	at [email protected]/org.elasticsearch.core.AbstractRefCounted.decRef(AbstractRefCounted.java:77)
	at [email protected]/org.elasticsearch.action.support.RefCountingRunnable.close(RefCountingRunnable.java:113)
	at [email protected]/org.elasticsearch.core.Releasables$4.close(Releasables.java:178)
	at [email protected]/org.elasticsearch.common.util.concurrent.RunOnce.run(RunOnce.java:41)
	at [email protected]/org.elasticsearch.action.fieldcaps.RequestDispatcher.innerExecute(RequestDispatcher.java:177)
	at [email protected]/org.elasticsearch.action.fieldcaps.RequestDispatcher$1.doRun(RequestDispatcher.java:146)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at [email protected]/org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction$1.onResponse(TransportFieldCapabilitiesAction.java:328)
	at [email protected]/org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction$1.onResponse(TransportFieldCapabilitiesAction.java:324)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractThrottledTaskRunner$1.doRun(AbstractThrottledTaskRunner.java:136)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at [email protected]/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:35)
	at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1067)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1575)\n"

bpintea · 2025-10-17T11:04:27Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

+            Map<String, AbstractConvertFunction> convertFunctions = new HashMap<>();
+            plan.forEachDown(p -> p.forEachExpression(AbstractConvertFunction.class, f -> {
+                if (f.field() instanceof Attribute attr && unionAll.output().contains(attr)) {
+                    convertFunctions.putIfAbsent(attr.name(), f);


I think this is problematic: we convert an emerging union output attribute to the first[*] type showing up in a conversion function, irrespective of this conversion function applying to the union output attribute or not.
That is: if x is a field in two indices (of different types), part of the output of something like:

FROM index1, (FROM index2) | EVAL some_field = x::LONG | EVAL some_other_field = x::INTEGER

x's type will be either LONG or INTEGER, depending on the order of those EVALs. Which I think is wrong.

[*] first in order of walking the tree.

I guess we probably want to restrict the map to only those attributes that apply a conversion function to another one with the same name? In which case, I guess we'll want to simply push down the conversion function to the UnionAll branches -- so not just copy it, but push it away downwards: right now the conversions stay, even if done on the same name attribute.

It is a good idea, I'll see how to deal with it. I'd like to keep the explicit casting push down in this PR. And I'm going to remove the implicit casting among subqueries and main index patterns from this PR, and do it as a follow up, as suggested by @alex-spies in the design review, if you haven't review the implicit casting part, you can skip it for now.

Thank you so much for catching this! I think we can handle multiple conversion functions on the same UnionAll output better. The collectConvertFunctions method has been updated to collect all explicit conversion functions from the main query that reference the outputs of UnionAll. These explicit conversion functions are now pushed down into each UnionAll branch and returned as new output attributes. And the subsequent commands after UnionAll can reference the outputs without confusion.

bpintea · 2025-10-17T11:55:52Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

+
+        private LogicalPlan maybePushDownConvertFunctionsToChild(LogicalPlan child, List<Alias> aliases, List<Attribute> output) {
+            // Fork/UnionAll adds an EsqlProject on top of each child plan during resolveFork, check this pattern before pushing down
+            if (aliases.isEmpty() == false && child instanceof EsqlProject esqlProject) {


Why do we need to check for the EsqlProject presence / pattern?

Up to here, the branches of UnionAll follow a common pattern like below:

UnionAll/Fork Project Eval (optional) Subquery (main index patterns do not have this node) ...

In both ResolveUnionTypesInUnionAll and PushDownFilterAndLimitIntoUnionAll, the pattern checks are intentionally strict — these rules only apply their transformations when a branch of UnionAll exactly matches the expected patterns. The main goal is to have a tight/better control over their behavior and ensure they perform only the intended transformations. If the structure of subqueries changes in the future and no longer fits the expected pattern, these rules will simply stop applying, allowing us to detect such changes early in the process.

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

bpintea · 2025-10-17T12:17:46Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

+                DataType targetType = convertFunctions.containsKey(oldAttr.name())
+                    ? convertFunctions.get(oldAttr.name()).dataType()
+                    : oldAttr.dataType();
+                if (oldAttr.dataType() != targetType) {


I think this condition can only be true if convertFunctions.containsKey(oldAttr.name()). Maybe we can simplify a bit the code here for better legibility.

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/tree/EsqlNodeSubclassTests.java

…across subqueries

…on only on coordinator node

non-correlated subquery in from command

193d8f7

fang-xing-esql added >enhancement :Analytics/ES|QL AKA ESQL ES|QL-ui Impacts ES|QL UI labels Sep 30, 2025

elasticsearchmachine added the v9.2.0 label Sep 30, 2025

Merge branch 'main' into subquery-in-from

0c5b79d

fang-xing-esql force-pushed the subquery-in-from branch from 8a72832 to 0c5b79d Compare September 30, 2025 22:26

Update docs/changelog/135744.yaml

440e501

fang-xing-esql added the test-release Trigger CI checks against release build label Sep 30, 2025

fang-xing-esql and others added 5 commits September 30, 2025 19:30

Merge branch 'main' into subquery-in-from

61c7266

cleanup

20e543d

update tests

f7d695d

subquery serialization

fb221c9

fix predicate push down

a38f6b9

elasticsearchmachine added v9.3.0 and removed v9.2.0 labels Oct 2, 2025

fang-xing-esql requested a review from Copilot October 2, 2025 13:45

Copilot AI reviewed Oct 2, 2025

View reviewed changes

fang-xing-esql commented Oct 2, 2025

View reviewed changes

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/SubqueryTests.java Show resolved Hide resolved

fang-xing-esql marked this pull request as ready for review October 2, 2025 15:16

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Oct 2, 2025

julian-elastic reviewed Oct 2, 2025

View reviewed changes

x-pack/plugin/esql/qa/testFixtures/src/main/resources/subquery.csv-spec Show resolved Hide resolved

julian-elastic reviewed Oct 2, 2025

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java Outdated Show resolved Hide resolved

julian-elastic reviewed Oct 2, 2025

View reviewed changes

...ain/java/org/elasticsearch/xpack/esql/optimizer/rules/logical/PushDownAndCombineFilters.java Outdated Show resolved Hide resolved

handle subquery with fork and full text function better

be82a74

luigidellaquila requested changes Oct 13, 2025

View reviewed changes

fang-xing-esql added 2 commits October 13, 2025 16:33

add capabilities and refactor according to review comments

d672fb8

Merge branch 'main' into subquery-in-from

e776729

luigidellaquila approved these changes Oct 14, 2025

View reviewed changes

bpintea reviewed Oct 15, 2025

View reviewed changes

bpintea reviewed Oct 16, 2025

View reviewed changes

carlosdelest reviewed Oct 17, 2025

View reviewed changes

...rc/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/FullTextFunction.java Outdated Show resolved Hide resolved

bpintea reviewed Oct 17, 2025

View reviewed changes

craigtaverner reviewed Oct 19, 2025

View reviewed changes

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/tree/EsqlNodeSubclassTests.java Outdated Show resolved Hide resolved

craigtaverner reviewed Oct 19, 2025

View reviewed changes

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/tree/EsqlNodeSubclassTests.java Outdated Show resolved Hide resolved

fang-xing-esql and others added 10 commits October 20, 2025 00:02

refactor and fix multiple conversion functions on mixed typed fields …

d69f2a4

…across subqueries

Merge branch 'main' into subquery-in-from

5fe91d8

more comments and refactor

2770735

Merge branch 'main' into subquery-in-from

b7b913d

inline stats after subquery is not supported, verify full text functi…

aea1ca8

…on only on coordinator node

Merge branch 'main' into subquery-in-from

c722dd9

update according to the recent change in full text functions

8de1d6e

Merge branch 'main' into subquery-in-from

35cb80a

add more test for subquery with full text functions

f533cb6

Merge branch 'main' into subquery-in-from

9c451d7

fang-xing-esql merged commit 20a8513 into elastic:main Oct 27, 2025
35 checks passed

fang-xing-esql mentioned this pull request Nov 14, 2025

[ES|QL] Remove the implicit LIMIT command added to each subquery (UnionAll branch) #138106

Open

	throw new ParsingException("FROM is required in a subquery");
	throw new ParsingException("Invalid subquery structure");

-                default -> null; // TODO add a general push down for unexpected pattern
+                default -> {
+                    // Fallback: unknown child type, do not push down filter for this child.
+                    // Consider implementing general push down logic here in the future.
+                    yield child;
+                }

	indexResolution = context.subqueryResolution().getOrDefault(indexPattern, indexResolution);
	indexResolution = context.subqueryResolution().get(indexPattern);
	if (indexResolution == null) {...

[ES|QL] Non-Correlated Subquery in FROM command #135744

[ES|QL] Non-Correlated Subquery in FROM command #135744

Uh oh!

Conversation

fang-xing-esql commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Sep 30, 2025

Uh oh!

elasticsearchmachine commented Sep 30, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 2, 2025

Uh oh!

elasticsearchmachine commented Oct 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

luigidellaquila left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fang-xing-esql commented Oct 14, 2025

Uh oh!

fang-xing-esql commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luigidellaquila left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bpintea left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bpintea left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql commented Sep 30, 2025 •

edited

Loading

fang-xing-esql commented Oct 14, 2025 •

edited

Loading

fang-xing-esql Oct 17, 2025 •

edited

Loading

bpintea left a comment •

edited

Loading

fang-xing-esql Oct 17, 2025 •

edited

Loading

fang-xing-esql Oct 20, 2025 •

edited

Loading