ES|QL - Allow multivalued query parameters #134317

carlosdelest · 2025-09-08T18:10:21Z

Adds support for multivalues as named and unnamed ES\QL query parameters.

The following is now possible:

POST _query
{
  "query": "from test | eval x = ?n1, y = ?n2, z = ?n3, t = ?n4, u = ?n5 | keep x, y, z, t, u | limit ?l"
  "params": [{"n1" : ["1", "2"]}, {"n2" : [2.0, 3.0]}, {"n3" : [true, false]}, {"n4" : [123, 456]}, {"n5": [1674835275193, -1674835275193]}, {"l": 3}]
}

Parameters cannot contain a null value. Multivalues are not expected to contain them as they are not kept at the storage layer.

This will allow for dense_vector values to be provided to dense_vector related functions.

elasticsearchmachine · 2025-09-08T18:10:47Z

Hi @carlosdelest, I've created a changelog YAML for you.

elasticsearchmachine · 2025-09-10T09:16:14Z

Hi @carlosdelest, I've updated the changelog YAML for you.

carlosdelest · 2025-09-10T09:24:53Z

x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DataType.java

+            case ZonedDateTime zonedDateTime -> {
+                return DATETIME;
+            }
+            case List<?> list -> {


Add data type resolution for lists

carlosdelest · 2025-09-10T09:26:16Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/RequestXContent.java

                    }
                } else {
-                    paramValue = null;
-                    if (token == XContentParser.Token.VALUE_STRING) {


This code moved to parseSingleParamValue - it can be confusing, switching to split view can be a better experience for reviewing this class

…i-valued-params # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java

carlosdelest · 2025-09-10T10:30:06Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

+        /**
+         * Multivalued query parameters
+         */
+        QUERY_PARAMS_MULTI_VALUES(Build.current().isSnapshot());


I'm thinking on not making this available just for snapshots, but would like to hear concerns about it

I think the best practice is to remove snapshot in a followup PR right?

We might want to give a heads up to kibana, and add test-release label if this is behind snapshot.

I think the best practice is to remove snapshot in a followup PR right?

We tend to do that when there's incomplete work or we have to make a decision about it being useful or not as it is. I think this doesn't need to be hidden behind snapshot, as it's useful and even if we want to improve it in the future we can do out of snapshot.

Happy to hear other thoughts!

I've removed the snapshot capability in b9a03ff - this change will go on all builds

elasticsearchmachine · 2025-09-10T12:53:01Z

Hi @carlosdelest, I've updated the changelog YAML for you.

elasticsearchmachine · 2025-09-10T12:53:02Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-09-10T12:53:02Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

kderusso

Changes LGTM

kderusso · 2025-09-10T18:20:22Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

+        /**
+         * Multivalued query parameters
+         */
+        QUERY_PARAMS_MULTI_VALUES(Build.current().isSnapshot());


I think the best practice is to remove snapshot in a followup PR right?

fang-xing-esql

Thank you for picking this up @carlosdelest. LGTM, I added some comments around the error message.

fang-xing-esql · 2025-09-11T02:47:15Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/RequestXContent.java

+                            ParamValueAndType valueAndDataType = parseSingleParamValue(p, errors);
+                            DataType currentType = valueAndDataType.type;
+                            if (currentType == DataType.NULL) {
+                                errors.add(new XContentParseException(loc, "Unnamed parameter contains a null in a multivalued value"));


It would be better if error messages were consistent between named and unnamed parameters.

Does this sound good?

[paramValues.toString()] contains a null entry. Null values are not allowed in multivalued params.

If we want a complete list of paramValues in the error message, a flag can be set if there is null value in the list, and this error can be added after this while loop.

That sounds good. I did that on a846f9a

fang-xing-esql · 2025-09-11T02:50:12Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/RequestXContent.java

+                                errors.add(
+                                    new XContentParseException(
+                                        loc,
+                                        "Unnamed parameter has values from different types, found " + arrayType + " and " + currentType


Same as above, it would be better if error messages were consistent between named and unnamed parameters.

How about

[paramValues.toString()] contains mixed data types. Mixed data types are not allowed in multivalued params.

Do we want to allow mixed numeric values like [1, 2.0]? If it is not relevant to the current context, it can be added later.

Changed on a846f9a .

Do we want to allow mixed numeric values like [1, 2.0]? If it is not relevant to the current context, it can be added later.

I'd say we can do that later if needed.

fang-xing-esql · 2025-09-11T02:52:50Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/RequestXContent.java

+                    errors.add(
+                        new XContentParseException(
+                            loc,
+                            "Parameter [" + entry.getKey() + "] has values from different types, found " + arrayType + " and " + currentType


A consistent error message as the unnamed param might be better.

[valueList.toString()] contains mixed data types. Mixed data types are not allowed in multivalued params., Perhaps refactor this error message to a method that can be access by both named and unnamed params.

Let me know whether a846f9a fixes this.

fang-xing-esql · 2025-09-11T02:53:49Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/RequestXContent.java

+                    errors.add(
+                        new XContentParseException(
+                            loc,
+                            "Parameter [" + entry.getKey() + "] contains a null value. Null values are not allowed for multivalues"


[valueList.toString()] contains a null entry. Null values are not allowed in multivalued params.

fang-xing-esql · 2025-09-11T03:00:17Z

...gin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/plugin/KnnFunctionIT.java

+        Arrays.fill(queryVector, 0);
+        EsqlQueryRequest queryRequest = new EsqlQueryRequest();
+        QueryParams queryParams = new QueryParams(
+            List.of(new QueryParam("queryVector", Arrays.asList(queryVector), DataType.INTEGER, ParserUtils.ParamClassification.PATTERN))


Is queryVector a constant value? Why this param set as a PATTERN?

My bad - fixed in f1909da

fang-xing-esql · 2025-09-11T03:03:09Z

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

+        /**
+         * Multivalued query parameters
+         */
+        QUERY_PARAMS_MULTI_VALUES(Build.current().isSnapshot());


We might want to give a heads up to kibana, and add test-release label if this is behind snapshot.

…ued-params' into enhancement/esql-multi-valued-params # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

…i-valued-params # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerTests.java

elasticsearchmachine · 2025-09-11T11:17:27Z

Pinging @elastic/kibana-esql (ES|QL-ui)

First version

e534c8b

carlosdelest added >enhancement :Analytics/ES|QL AKA ESQL v9.2.0 :Search Relevance/ES|QL Search functionality in ES|QL labels Sep 8, 2025

Update docs/changelog/134317.yaml

a30d9ae

elasticsearchmachine and others added 9 commits September 8, 2025 18:17

[CI] Auto commit changes from spotless

3112666

Fix changelog

f4f0678

Fix param validation

cc4d17e

Add REST tests and fix parsing

a72a293

Spotless

1dd6669

Add capabilities guard

2c95a66

Handle null multivalues

ceccf94

Don't allow null in multivalues

ab1146d

Fix tests

b13d467

carlosdelest force-pushed the enhancement/esql-multi-valued-params branch from 179d9b2 to b13d467 Compare September 10, 2025 09:05

carlosdelest mentioned this pull request Sep 4, 2025

ES|QL - dense_vector approximate nearest neighbour search support (GA) #126710

Open

8 tasks

carlosdelest and others added 3 commits September 10, 2025 11:16

Update docs/changelog/134317.yaml

21c222a

Update changelog

5ddf1bf

Undo EvalMapper change

7722600

carlosdelest commented Sep 10, 2025

View reviewed changes

[CI] Auto commit changes from spotless

8aa4a9a

carlosdelest marked this pull request as ready for review September 10, 2025 12:52

Update docs/changelog/134317.yaml

6d7e94e

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Sep 10, 2025

elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Sep 10, 2025

carlosdelest requested review from fang-xing-esql, ioanatia, kderusso and svilen-mihaylov-elastic September 10, 2025 12:53

kderusso approved these changes Sep 10, 2025

View reviewed changes

fang-xing-esql reviewed Sep 11, 2025

View reviewed changes

carlosdelest added 6 commits September 11, 2025 08:02

Fix QueryParam classification in test

f1909da

Improve error messages

a846f9a

Remove snapshot from capability

b9a03ff

Merge remote-tracking branch 'carlosdelest/enhancement/esql-multi-val…

6bbc621

…ued-params' into enhancement/esql-multi-valued-params # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

Merge tests

205f920

carlosdelest added the ES|QL-ui Impacts ES|QL UI label Sep 11, 2025

fang-xing-esql approved these changes Sep 11, 2025

View reviewed changes

Merge branch 'main' into enhancement/esql-multi-valued-params

e199aef

carlosdelest enabled auto-merge (squash) September 11, 2025 14:01

Merge branch 'main' into enhancement/esql-multi-valued-params

1117aff

carlosdelest merged commit f6547d6 into elastic:main Sep 11, 2025
34 checks passed

pquentin mentioned this pull request Sep 29, 2025

[BREAKING] Support multivalued ES|QL parameters elastic/elasticsearch-specification#5362

Merged

ES|QL - Allow multivalued query parameters #134317

ES|QL - Allow multivalued query parameters #134317

Uh oh!

Conversation

carlosdelest commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Sep 8, 2025

Uh oh!

elasticsearchmachine commented Sep 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Sep 10, 2025

Uh oh!

elasticsearchmachine commented Sep 10, 2025

Uh oh!

elasticsearchmachine commented Sep 10, 2025

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Sep 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

carlosdelest commented Sep 8, 2025 •

edited

Loading

carlosdelest Sep 11, 2025 •

edited

Loading

carlosdelest Sep 11, 2025 •

edited

Loading

carlosdelest Sep 11, 2025 •

edited

Loading