Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
9f24a18
Initial
svilen-mihaylov-elastic Jul 22, 2025
594d76a
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 22, 2025
69fff05
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 23, 2025
b64ec91
Update docs/changelog/131723.yaml
svilen-mihaylov-elastic Jul 24, 2025
666382c
Update docs/changelog/131723.yaml
svilen-mihaylov-elastic Jul 24, 2025
15480b6
Handle fork references correctly
svilen-mihaylov-elastic Jul 24, 2025
96951c7
Merge branch 'svilen/127208' of https://github.com/svilen-mihaylov-el…
svilen-mihaylov-elastic Jul 24, 2025
8a98b45
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 24, 2025
bb7082d
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 24, 2025
a20fb5b
Remove _fork
svilen-mihaylov-elastic Jul 24, 2025
b159bd7
[CI] Auto commit changes from spotless
Jul 24, 2025
3b7f546
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 25, 2025
f9f26d4
Fix merge
svilen-mihaylov-elastic Jul 25, 2025
917b273
Merge branch 'svilen/127208' of https://github.com/svilen-mihaylov-el…
svilen-mihaylov-elastic Jul 25, 2025
cb412e6
Return all fields
svilen-mihaylov-elastic Jul 25, 2025
c5564f0
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 28, 2025
7d67986
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 29, 2025
2479bfa
Fix
svilen-mihaylov-elastic Jul 29, 2025
3ebe919
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 30, 2025
482fabe
tweak
svilen-mihaylov-elastic Jul 30, 2025
300f169
Merge branch 'svilen/127208' of https://github.com/svilen-mihaylov-el…
svilen-mihaylov-elastic Jul 30, 2025
6378f76
Add tests
svilen-mihaylov-elastic Jul 30, 2025
f337b02
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 30, 2025
82f189d
Add test
svilen-mihaylov-elastic Jul 30, 2025
199392f
Add more tests
svilen-mihaylov-elastic Jul 31, 2025
b7a7854
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 31, 2025
5dd7462
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Jul 31, 2025
98aa887
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Aug 1, 2025
7bc4a87
Address feedback
svilen-mihaylov-elastic Aug 4, 2025
b017321
not
svilen-mihaylov-elastic Aug 4, 2025
d14bf24
Update x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/…
svilen-mihaylov-elastic Aug 6, 2025
2931d83
Update x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/…
svilen-mihaylov-elastic Aug 6, 2025
82672be
Update x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/…
svilen-mihaylov-elastic Aug 6, 2025
94da34b
Update
svilen-mihaylov-elastic Aug 6, 2025
9606c04
[CI] Auto commit changes from spotless
Aug 6, 2025
3f55419
Separate implementation
svilen-mihaylov-elastic Aug 7, 2025
00a885a
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Aug 8, 2025
4313222
Merge branch 'main' into svilen/127208
svilen-mihaylov-elastic Aug 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -811,11 +811,6 @@ static PreAnalysisResult fieldNames(LogicalPlan parsed, Set<String> enrichPolicy
return result.withFieldNames(IndexResolver.ALL_FIELDS);
}

// TODO: Improve field resolution for FORK - right now we request all fields
if (parsed.anyMatch(p -> p instanceof Fork)) {
return result.withFieldNames(IndexResolver.ALL_FIELDS);
}

Holder<Boolean> projectAll = new Holder<>(false);
parsed.forEachExpressionDown(UnresolvedStar.class, us -> {// explicit "*" fields selection
if (projectAll.get()) {
Expand All @@ -828,6 +823,32 @@ static PreAnalysisResult fieldNames(LogicalPlan parsed, Set<String> enrichPolicy
return result.withFieldNames(IndexResolver.ALL_FIELDS);
}

Holder<Boolean> projectAfterFork = new Holder<>(false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these look like the changes from #128193 - these are not going to work.

please check what happens later in the fieldNames method. For example here where we remove field references - this has the potential to work incorrectly for FORK:

if (canRemoveAliases[0]) {
// remove any already discovered UnresolvedAttributes that are in fact aliases defined later down in the tree
// for example "from test | eval x = salary | stats max = max(x) by gender"
// remove the UnresolvedAttribute "x", since that is an Alias defined in "eval"
// also remove other down-the-tree references to the extracted fields from "grok" and "dissect"
AttributeSet planRefs = p.references();
Set<String> fieldNames = planRefs.names();
p.forEachExpressionDown(NamedExpression.class, ne -> {
if ((ne instanceof Alias || ne instanceof ReferenceAttribute) == false) {
return;
}
// do not remove the UnresolvedAttribute that has the same name as its alias, ie "rename id AS id"
// or the UnresolvedAttributes that are used in Functions that have aliases "STATS id = MAX(id)"
if (fieldNames.contains(ne.name())) {
return;
}
referencesBuilder.removeIf(
attr -> matchByName(attr, ne.name(), keepRefs.contains(attr) || dropWildcardRefs.contains(attr))
);
});
}

As an example, I tested the following queries locally:

FROM employees
| FORK
       ( STATS x = VALUES(last_name))
       ( EVAL last_name = first_name  | STATS y = VALUES(last_name))
       
FROM employees
| FORK
       ( EVAL last_name = first_name  | STATS y = VALUES(last_name))
       ( STATS x = VALUES(last_name))     

They are pretty much the same - just the order of the FORK branches is different.
The first one breaks with:

line 4:26: Unknown column [last_name], did you mean [first_name]?

the second one works.

the first one breaks because most likely we remove the reference to last_name as part of the logic from the snippet I shared earlier.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sharing those. For the future, please record such failing cases in the original github issue. This will be helpful for folks working on it to understand the context better.

Holder<Boolean> hasFork = new Holder<>(false);

parsed.forEachDown(plan -> {
if (projectAll.get()) {
return;
}

if (hasFork.get() == false && shouldCollectReferencedFields(plan, inlinestatsAggs)) {
projectAfterFork.set(true);
}

if (plan instanceof Fork fork && projectAfterFork.get() == false) {
hasFork.set(true);
fork.children().forEach(child -> {
if (child.anyMatch(p -> shouldCollectReferencedFields(p, inlinestatsAggs)) == false) {
projectAll.set(true);
}
});
}
});

if (projectAll.get()) {
return result.withFieldNames(IndexResolver.ALL_FIELDS);
}

var referencesBuilder = AttributeSet.builder();
// "keep" and "drop" attributes are special whenever a wildcard is used in their name, as the wildcard can cover some
// attributes ("lookup join" generated columns among others); steps like removal of Aliases should ignore fields matching the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2080,7 +2080,7 @@ public void testForkFieldsWithKeepAfterFork() {
(WHERE d > 1000 AND e == "aaa" | EVAL c = a + 200)
| WHERE x > y
| KEEP a, b, c, d, x
""", ALL_FIELDS);
""", Set.of("a", "x", "y", "d", "e", "e.*", "d.*", "y.*", "x.*", "a.*"));
}

public void testForkFieldsWithKeepBeforeFork() {
Expand All @@ -2092,7 +2092,7 @@ public void testForkFieldsWithKeepBeforeFork() {
| FORK (WHERE c > 1 AND a < 10000 | EVAL d = a + 500)
(WHERE d > 1000 AND e == "aaa" | EVAL c = a + 200)
| WHERE x > y
""", ALL_FIELDS);
""", Set.of("x", "y", "a", "d", "e", "b", "c", "e.*", "d.*", "y.*", "x.*", "a.*", "c.*", "b.*"));
}

public void testForkFieldsWithNoProjection() {
Expand All @@ -2118,17 +2118,41 @@ public void testForkFieldsWithStatsInOneBranch() {
}

public void testForkFieldsWithEnrichAndLookupJoins() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add some more tests?

it would be great if you can test this case and see that the following 2 queries have the same references:


FROM test
| FORK
       ( STATS x = VALUES(last_name))
       ( EVAL last_name = first_name  | STATS y = VALUES(last_name))
       
FROM test
| FORK
       ( EVAL last_name = first_name  | STATS y = VALUES(last_name))
       ( STATS x = VALUES(last_name))     

Then I would also add some more tests where the FORK branches themselves use KEEP and DROP (with wildcard field patterns too). When I added these tests the FORK branches did not support commands like KEEP and DROP, but now they do.
If you can add other tests, like FORK branches that use ENRICH for example, that would be great - thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying! Again, for future reference, the original issue is a great place to record those particulars.

assertFieldNames("""
FROM test
| KEEP a, b, abc, def, z, xyz
| ENRICH enrich_policy ON abc
| EVAL b = a + 100
| LOOKUP JOIN my_lookup_index ON def
| FORK (WHERE c > 1 AND a < 10000 | EVAL d = a + 500)
(STATS x = count(*), y=min(z))
| LOOKUP JOIN my_lookup_index ON xyz
| WHERE x > y OR _fork == "fork1"
""", ALL_FIELDS);
assertFieldNames(
"""
FROM test
| KEEP a, b, abc, def, z, xyz
| ENRICH enrich_policy ON abc
| EVAL b = a + 100
| LOOKUP JOIN my_lookup_index ON def
| FORK (WHERE c > 1 AND a < 10000 | EVAL d = a + 500)
(STATS x = count(*), y=min(z))
| LOOKUP JOIN my_lookup_index ON xyz
| WHERE x > y OR _fork == "fork1"
""",
Set.of(
"x",
"y",
"_fork",
"a",
"c",
"abc",
"b",
"def",
"z",
"xyz",
"def.*",
"_fork.*",
"y.*",
"x.*",
"xyz.*",
"z.*",
"abc.*",
"a.*",
"c.*",
"b.*"
)
);
}

public void testForkWithStatsInAllBranches() {
Expand All @@ -2140,7 +2164,7 @@ public void testForkWithStatsInAllBranches() {
(EVAL z = a * b | STATS m = max(z))
(STATS x = count(*), y=min(z))
| WHERE x > y
""", ALL_FIELDS);
""", Set.of("c", "a", "z", "z.*", "a.*", "c.*"));
}

public void testForkWithStatsAndWhere() {
Expand Down