ES|QL: Disallow brackets in unquoted index patterns #130427

luigidellaquila · 2025-07-02T07:44:17Z

To avoid collisions with subquery syntax, we disallow the usage of ( and ) in unquoted index patterns and enrich policy names.

Fixes: #130378

elasticsearchmachine · 2025-07-03T10:32:39Z

Hi @luigidellaquila, I've created a changelog YAML for you.

…indexes' into esql/no_parentheses_in_indexes

elasticsearchmachine · 2025-07-04T09:30:45Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

idegtiarenko · 2025-07-04T11:51:58Z

I believe we keep enrich policy definition in cluster state. Should we prohibit creation of new ones with brackets? What should we do with ones previously created with brackets in names? Please let me know if this concerns are separate from this change

luigidellaquila · 2025-07-07T08:05:54Z

Should we prohibit creation of new ones with brackets?

Enrich policies allow brackets, so I don't think we can just prohibit it.
With this change, we can allow brackets also in policy names, as long as they are quoted.

Yaml tests apparently have problems creating policies with brackets, but I'll add a Java integration test for it.

luigidellaquila · 2025-07-07T14:01:01Z

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/61_enrich_bracket.yml

@@ -0,0 +1,102 @@
+---


This has to be a separate file and with a single test, just because the policy delete doesn't seem to work with brackets

- do: enrich.delete_policy: name: <name with brackets here>

Apparently the bracket collides with some regex evaluation that happens during the process.
I couldn't find a way to work around this problem, but it seems unrelated to ES|QL, so I guess we can handle it separately.

No way, I had to skip the test.
If I don't delete the enrich policy, it will make other tests fail.

luigidellaquila · 2025-07-07T14:02:21Z

@idegtiarenko I added a test that uses an enrich policy with brackets (with quotes), and it works as expected.

can't delete enrich policy

astefan

Looks good, but you need to be paranoid with grammar changes and add more tests. I've put in some examples which should be extended to other commands that use index names patterns.

astefan · 2025-07-08T07:34:13Z

docs/changelog/130427.yaml

@@ -0,0 +1,5 @@
+pr: 130427
+summary: Disallow brackets in unquoted index pattersn


astefan · 2025-07-08T08:22:34Z

x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/parser/StatementParserTests.java

+        UnresolvedRelation right = as(lookup.right(), UnresolvedRelation.class);
+        assertThat(right.indexPattern().indexPattern(), is("foo)"));
+    }
+


Please, add more unit tests here. The above ones are the bare minimum imo, but it's always good to be paranoid with grammar changes as the devil is in the edge cases :-)).

from te()st
from \"te()st\"
from concat(foo, bar)
from ((((()))))
from (((abc)))
from *()*
from *test()*
from *:test()
from *:()
from *:test)
from remote1:test(),remote2:test

and use these also for other places where index names are used (enrich, lookup join, fork etc).

astefan

LGTM. Thanks

astefan · 2025-07-09T15:29:07Z

No 9.1 version backport? Just curious

luigidellaquila · 2025-07-09T15:32:22Z

No 9.1 version backport?

We absolutely need to backport it to 9.1 as well, thanks for pointing it out.
Fixed!

luigidellaquila · 2025-07-09T15:35:35Z

Actually, I think we want it only in 9.2, 9.1, and 8.19.
It's a bug fix, but it's breaking, so I'm not sure we want to port it further back.
I'm removing 9.0 label for now, in case I'll do further backports manually

elasticsearchmachine · 2025-07-09T15:38:10Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.1	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 130427

…lastic#130938)

…130938) (#130984)

luigidellaquila · 2025-07-10T13:28:00Z

Backports here:

[9.1] ES|QL: Disallow brackets in unquoted index patterns (#130427) #130938
[8.19] ES|QL: Disallow brackets in unquoted index patterns (#130427) (#130938) #130984

ioanatia · 2025-07-10T18:20:24Z

x-pack/plugin/src/yamlRestTest/resources/rest-api-spec/test/esql/60_enrich.yml

+
+
+---
+"Enrich in fork":


this test is causing bwc test failures since FORK is not available in 8.19

luigidellaquila added 2 commits July 1, 2025 13:39

No parentheses in unquoted index patterns

16458de

ES|QL: Disallow brackets in unquoted index patterns

1b179e5

elasticsearchmachine added the v9.2.0 label Jul 2, 2025

luigidellaquila added 3 commits July 2, 2025 10:07

Merge branch 'main' into esql/no_parentheses_in_indexes

dac8f66

Merge branch 'main' into esql/no_parentheses_in_indexes

db51487

Merge branch 'main' into esql/no_parentheses_in_indexes

32450cd

luigidellaquila added >bug :Analytics/ES|QL AKA ESQL v9.0.0 auto-backport Automatically create backport pull requests when merged labels Jul 3, 2025

luigidellaquila added 3 commits July 3, 2025 12:32

Update docs/changelog/130427.yaml

09dfc8a

Fix bwc

f31c857

Merge remote-tracking branch 'luigidellaquila/esql/no_parentheses_in_…

3d83060

…indexes' into esql/no_parentheses_in_indexes

luigidellaquila marked this pull request as ready for review July 4, 2025 09:30

Merge branch 'main' into esql/no_parentheses_in_indexes

60c87d5

luigidellaquila added the v8.19.0 label Jul 4, 2025

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jul 4, 2025

luigidellaquila added 2 commits July 7, 2025 12:34

Merge branch 'main' into esql/no_parentheses_in_indexes

96ce20d

Add test for ENRICH

5039cc9

luigidellaquila commented Jul 7, 2025

View reviewed changes

Disable test

88fc89a

can't delete enrich policy

astefan reviewed Jul 8, 2025

View reviewed changes

luigidellaquila changed the title ~~ES|QL: Disallow brackets in unquoted index pattersn~~ ES|QL: Disallow brackets in unquoted index patterns Jul 8, 2025

luigidellaquila added 2 commits July 8, 2025 17:16

Merge branch 'main' into esql/no_parentheses_in_indexes

0807cf1

Merge branch 'main' into esql/no_parentheses_in_indexes

8783684

luigidellaquila added 2 commits July 9, 2025 15:53

More tests

72b833a

spotless

7d6ae9e

astefan approved these changes Jul 9, 2025

View reviewed changes

luigidellaquila added the v9.1.0 label Jul 9, 2025

luigidellaquila added v9.0.4 and removed v9.0.0 v9.0.4 labels Jul 9, 2025

luigidellaquila merged commit 71434e6 into elastic:main Jul 9, 2025
34 checks passed

elasticsearchmachine added the backport pending label Jul 9, 2025

luigidellaquila added a commit to luigidellaquila/elasticsearch that referenced this pull request Jul 9, 2025

ES|QL: Disallow brackets in unquoted index patterns (elastic#130427)

c3004db

luigidellaquila mentioned this pull request Jul 9, 2025

[9.1] ES|QL: Disallow brackets in unquoted index patterns (#130427) #130938

Merged

elasticsearchmachine pushed a commit that referenced this pull request Jul 9, 2025

ES|QL: Disallow brackets in unquoted index patterns (#130427) (#130938)

024228d

luigidellaquila added a commit to luigidellaquila/elasticsearch that referenced this pull request Jul 10, 2025

ES|QL: Disallow brackets in unquoted index patterns (elastic#130427) (e…

4770fa6

…lastic#130938)

This was referenced Jul 10, 2025

[8.19] ES|QL: Disallow brackets in unquoted index patterns (#130427) (#130938) #130984

Merged

ES|QL: Add breaking change note for brackets in index names #131012

Merged

luigidellaquila added a commit that referenced this pull request Jul 10, 2025

[8.19] ES|QL: Disallow brackets in unquoted index patterns (#130427) (#…

ce51a84

…130938) (#130984)

ioanatia reviewed Jul 10, 2025

View reviewed changes

mridula-s109 pushed a commit to mridula-s109/elasticsearch that referenced this pull request Jul 17, 2025

ES|QL: Disallow brackets in unquoted index patterns (elastic#130427)

7c9adc6

mridula-s109 pushed a commit to mridula-s109/elasticsearch that referenced this pull request Jul 17, 2025

ES|QL: Disallow brackets in unquoted index patterns (elastic#130427)

8be168b

		@@ -0,0 +1,5 @@
		pr: 130427
		summary: Disallow brackets in unquoted index pattersn



		---
		"Enrich in fork":

ES|QL: Disallow brackets in unquoted index patterns #130427

ES|QL: Disallow brackets in unquoted index patterns #130427

Uh oh!

Conversation

luigidellaquila commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 3, 2025

Uh oh!

elasticsearchmachine commented Jul 4, 2025

Uh oh!

idegtiarenko commented Jul 4, 2025

Uh oh!

luigidellaquila commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

luigidellaquila Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

luigidellaquila Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

luigidellaquila Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

luigidellaquila commented Jul 7, 2025

Uh oh!

astefan left a comment

Choose a reason for hiding this comment

Uh oh!

astefan Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

astefan Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

astefan left a comment

Choose a reason for hiding this comment

Uh oh!

astefan commented Jul 9, 2025

Uh oh!

luigidellaquila commented Jul 9, 2025

Uh oh!

luigidellaquila commented Jul 9, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 9, 2025

💔 Backport failed

Uh oh!

luigidellaquila commented Jul 10, 2025

Uh oh!

ioanatia Jul 10, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

luigidellaquila commented Jul 2, 2025 •

edited

Loading

luigidellaquila commented Jul 7, 2025 •

edited

Loading