ES|QL - Full text functions accept null as field parameter #137430

carlosdelest · 2025-10-31T10:03:08Z

Full text functions that take a field as a parameter should allow null as a field parameter. This is necessary as the field may not be present on an index mapping, and the local optimizer replaces it by null.

We allow null as a field in FullTextFunctions, meaning that they are nullable and will be replaced by NULL.

This PR also refactors full text functions that depend on a single field (MATCH, MATCH_PHRASE, KNN) to use a common superclass (SingleFieldFullTextFunction) that contains the common field logic.

… and add the FoldNull rule to LocalLogicalPlanOptimizer

…down to an index that doesn't contain the field

elasticsearchmachine · 2025-10-31T10:03:34Z

Hi @carlosdelest, I've created a changelog YAML for you.

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

…-functions-accept-null # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizerTests.java

carlosdelest · 2025-10-31T10:04:53Z

...ugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/Match.java

@@ -300,39 +286,7 @@ public final void writeTo(StreamOutput out) throws IOException {

    @Override
    protected TypeResolution resolveParams() {
-        return resolveField().and(resolveQuery())


Refactoring - all of these are moved up to SingleFieldFullTextFunction

carlosdelest · 2025-10-31T10:05:57Z

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

+    }
+
+    @Override
+    public boolean foldable() {


This is what changed on this PR besides all the refactoring. When the FTF field is null, the FTF is foldable to NULL, and nullable.

carlosdelest · 2025-10-31T11:32:32Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java

                new InferNonNullAggConstraint(),
-                new ReplaceDateTruncBucketWithRoundTo()
+                new ReplaceDateTruncBucketWithRoundTo(),
+                new FoldNull()


@alex-spies I added foldNull to local logical optimization, so full text functions can be folded to null when null has been replaced. Does it look OK to you?

I think this should be beneficial, yes. Not sure why we didn't have it earlier.
Do you think you could please add one/some LocalLogicalPlanOptimizerTests with maybe some "easy" other function(s) (string or math?) to check that they locally now get folded too. This should add a tiny bit of performance to other functions too.

the FoldNull rule is in the localOperators batch, that is applied after the localRewrites batch. Is it too late to wait for it to be executed in localOperators? I wonder why we need a duplicate of it here?

Oh, I see - I didn't understand that this rule was already used as part of localOperators(). Removing it, thanks @fang-xing-esql !

carlosdelest · 2025-10-31T14:34:59Z

...lasticsearch/xpack/esql/optimizer/rules/logical/PushDownFilterAndLimitIntoUnionAllTests.java

+            """);
+
+        // Limit[1000[INTEGER],false,false]
+        Limit limit = as(plan, Limit.class);


@fang-xing-esql , now FTFs can be pushed down - when a field does not exist in the index, it will become NULL instead.

Does this make sense from the union all perspective?

elasticsearchmachine · 2025-10-31T14:35:33Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-10-31T14:35:34Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2025-10-31T14:35:34Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

kderusso

Changes look good to me, but will defer to someone with more ES|QL experience to 👍 😁

kderusso · 2025-10-31T17:54:52Z

...lasticsearch/xpack/esql/optimizer/rules/logical/PushDownFilterAndLimitIntoUnionAllTests.java

    public void testFullTextFunctionCannotBePushedDownPastUnionAll() {
        assumeTrue("Requires subquery in FROM command support", EsqlCapabilities.Cap.SUBQUERY_IN_FROM_COMMAND.isEnabled());
-        VerificationException e = expectThrows(VerificationException.class, () -> planSubquery("""
+        var plan = planSubquery("""


Do we need to specify a new capability for this test?

We don't, as the MATCH function is already released (so it's not conditionally added for release / snapshots) and this test is not executed for BwC purposes.

kderusso · 2025-10-31T18:09:59Z

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

+ * - Field verification
+ * - Serialization patterns
+ */
+public abstract class SingleFieldFullTextFunction extends FullTextFunction


Nice refactoring!

carlosdelest · 2025-11-04T11:36:52Z

@elasticsearchmachine run elasticsearch-ci/bwc-snapshots-part3

bpintea

Left a few comments, but after those it should look good.
I think some simple CSV-spec tests would be good to have too, just to check that they work with literals, centrally too. (MATCH(null, ..), ..| EVAL x = null | WHERE MATCH(x, ...)).

Would be great to wait for @fang-xing-esql check too.

bpintea · 2025-11-04T11:14:29Z

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

+    static String expectedTypesAsString(Set<DataType> FieldDataTypes) {
+        return String.join(", ", FieldDataTypes.stream().map(dt -> dt.name().toLowerCase(Locale.ROOT)).toList());


Suggested change

static String expectedTypesAsString(Set<DataType> FieldDataTypes) {

return String.join(", ", FieldDataTypes.stream().map(dt -> dt.name().toLowerCase(Locale.ROOT)).toList());

static String expectedTypesAsString(Set<DataType> dataTypes) {

return String.join(", ", dataTypes.stream().map(DataType::typeName).toList());

bpintea · 2025-11-04T11:15:47Z

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

+     * Returns a human-readable string listing the expected field types.
+     * Used in error messages.
+     */
+    protected String getExpectedFieldTypesString() {


Suggested change

protected String getExpectedFieldTypesString() {

protected String expectedFieldTypesString() {

for consistency (with the static method).

bpintea · 2025-11-04T11:16:04Z

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

+     * Returns a human-readable string listing the expected query types.
+     * Used in error messages.
+     */
+    protected String getExpectedQueryTypesString() {


Suggested change

protected String getExpectedQueryTypesString() {

protected String expectedQueryTypesString() {

bpintea · 2025-11-04T11:26:18Z

...rc/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/FullTextFunction.java

+    protected FieldAttribute fieldAsFieldAttribute(Expression field) {
        Expression fieldExpression = field;
        // Field may be converted to other data type (field_name :: data_type), so we need to check the original field
        if (fieldExpression instanceof AbstractConvertFunction convertFunction) {


A somewhat lateral note: we could have an unpacking loop here, for the (more likely automated query generation) case of nested conversions, like: field::INTEGER::LONG.

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/MultiMatch.java

bpintea · 2025-11-04T11:46:00Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java

+    @Override
+    protected List<Batch<LogicalPlan>> batches() {
+        return RULES;
+    }
+


This is just a move, right? Was it intentional?

No, reverting

bpintea · 2025-11-04T14:26:04Z

...lasticsearch/xpack/esql/expression/function/fulltext/AbstractMatchFullTextFunctionTests.java

        AbstractMatchFullTextFunctionTests.addQueryAsStringTestCases(suppliers);
        AbstractMatchFullTextFunctionTests.addStringTestCases(suppliers);
-        return suppliers;
+        return addNullFieldTestCases(suppliers);


I think these aren't added to all functions, like MatchPhraseTests (possibly others, Knn?), right?
Raising this since MatchPhrase.FIELD_DATA_TYPES doesn't include NULL.

Good call! Done in 845cd35

bpintea · 2025-11-04T14:30:42Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java

                new InferNonNullAggConstraint(),
-                new ReplaceDateTruncBucketWithRoundTo()
+                new ReplaceDateTruncBucketWithRoundTo(),
+                new FoldNull()


I think this should be beneficial, yes. Not sure why we didn't have it earlier.
Do you think you could please add one/some LocalLogicalPlanOptimizerTests with maybe some "easy" other function(s) (string or math?) to check that they locally now get folded too. This should add a tiny bit of performance to other functions too.

Co-authored-by: Bogdan Pintea <[email protected]>

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

elasticsearchmachine added 10 commits October 29, 2025 18:43

Remove null check for fields in FTFs

1364419

First version - allow nulls as params, implement nullable() in Match,…

bd00de1

… and add the FoldNull rule to LocalLogicalPlanOptimizer

Introduce base class for Match and MatchPhrase

20b202d

Make Knn a subclass of SingleFieldFullTextFunction, refactor

c7bda0f

Refactor

7f5f5c6

Fix PushDownFilterAndLimitIntoUnionAllTests, now match can be pushed …

8fa9556

…down to an index that doesn't contain the field

Spotless

bd6e99d

Refactor

8ce4820

Fix error tests

1d98c74

Add check for single unique signature

ec81181

carlosdelest added >bug :Analytics/ES|QL AKA ESQL :Search Relevance/ES|QL Search functionality in ES|QL v9.3.0 labels Oct 31, 2025

Update docs/changelog/137430.yaml

336b33c

carlosdelest added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :SearchOrg/Relevance Label for the Search (solution/org) Relevance team Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch labels Oct 31, 2025

elasticsearchmachine added 5 commits October 31, 2025 11:51

Fix tests and add null cases

959cac7

Merge remote-tracking branch 'carlosdelest/bugfix/esql-full-text-func…

21daf3d

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

Add test cases

b737b03

Fix tests

f68b569

carlosdelest commented Oct 31, 2025

View reviewed changes

carlosdelest marked this pull request as ready for review October 31, 2025 14:35

elasticsearchmachine added the Team:Search - Relevance The Search organization Search Relevance team label Oct 31, 2025

carlosdelest requested review from alex-spies, fang-xing-esql and ioanatia October 31, 2025 14:41

kderusso reviewed Oct 31, 2025

View reviewed changes

julian-elastic mentioned this pull request Nov 3, 2025

ES|QL - Full text functions should work with LOOKUP JOIN #129153

Open

bpintea approved these changes Nov 4, 2025

View reviewed changes

elasticsearchmachine and others added 7 commits November 4, 2025 16:58

Remove FoldNull as it is already applied in localOperators()

2436a22

Apply suggestion from @bpintea

012378b

Co-authored-by: Bogdan Pintea <[email protected]>

Apply suggestion from @bpintea

30a8319

Co-authored-by: Bogdan Pintea <[email protected]>

Apply suggestion from @bpintea

569b084

Co-authored-by: Bogdan Pintea <[email protected]>

Merge remote-tracking branch 'carlosdelest/bugfix/esql-full-text-func…

04b7dd9

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

Feedback from @bpintea

ad811e5

Add null test cases for Knn and MatchPhrase

845cd35

		static String expectedTypesAsString(Set<DataType> FieldDataTypes) {
		return String.join(", ", FieldDataTypes.stream().map(dt -> dt.name().toLowerCase(Locale.ROOT)).toList());

	protected String getExpectedFieldTypesString() {
	protected String expectedFieldTypesString() {

	protected String getExpectedQueryTypesString() {
	protected String expectedQueryTypesString() {

ES|QL - Full text functions accept null as field parameter #137430

Are you sure you want to change the base?

ES|QL - Full text functions accept null as field parameter #137430

Conversation

carlosdelest commented Oct 31, 2025

Uh oh!

elasticsearchmachine commented Oct 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bpintea Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 31, 2025

Uh oh!

elasticsearchmachine commented Oct 31, 2025

Uh oh!

elasticsearchmachine commented Oct 31, 2025

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest commented Nov 4, 2025

Uh oh!

bpintea left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bpintea Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bpintea Nov 4, 2025 •

edited

Loading

bpintea left a comment •

edited

Loading

bpintea Nov 4, 2025 •

edited

Loading