ES|QL: Add PRESENT ES|QL function #133986

leontyevdv · 2025-09-02T13:47:26Z

Add a new ES|QL function that checks for the presence of a field in the output result. Presence means that the input expression yields any non-null value.

Part of #131069

Add a new ES|QL function that checks for the presence of a field in the output result. Presence means that the input expression yields any non-null value. Part of elastic#131069

Add unit tests and documentation for the PRESENT function. Part of elastic#131069

…nto feature/esql-present-function

github-actions · 2025-09-02T15:22:46Z

🔍 Preview links for changed docs

github-actions · 2025-09-02T15:22:47Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

Clean-up of the PRESENT function. Part of elastic#131069

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

- Change intermediate state for using boolean - Add unit tests for PresentAggregatorFunctionTests and PresentGroupingAggregatorFunctionTests Part of elastic#131069

- Add VerifierTests Part of elastic#131069

- Add union_types csv tests Part of elastic#131069

elasticsearchmachine · 2025-09-04T09:37:51Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-09-04T09:37:51Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

- Fix unit tests Part of elastic#131069

- Comment out TestLogging on CsvTests - Add missing DataTypes to the function Part of elastic#131069

- Improve documentation Part of elastic#131069

dnhatn

I've left some comments around the seen block and the state. The PR looks great. Thanks @leontyevdv

Note that we can rewrite PRESENT as COUNT, but I'm okay with implementing the aggregator for Present as in this PR. The Present aggregator should be lighter than COUNT, but COUNT can be pushed down to Lucene. Present can be pushed down too, though I don't think it's used enough to consider it.

dnhatn · 2025-09-05T00:04:22Z

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

+
+    private static final List<IntermediateStateDesc> INTERMEDIATE_STATE_DESC = List.of(
+        new IntermediateStateDesc("present", ElementType.BOOLEAN),
+        new IntermediateStateDesc("seen", ElementType.BOOLEAN)


I think we don't need seen which tracks groups without values.

I removed it, thanks!

dnhatn · 2025-09-05T00:06:14Z

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

+        new IntermediateStateDesc("seen", ElementType.BOOLEAN)
+    );
+
+    private final BooleanArrayState state;


If we remove seen, we can use BitArray to track whether a group has a value.

Done! BitArray is now being used in this class.

dnhatn · 2025-09-05T00:06:40Z

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

+    public AddInput prepareProcessRawInputPage(SeenGroupIds seenGroupIds, Page page) {
+        Block valuesBlock = page.getBlock(blockIndex());
+
+        if (valuesBlock.mayHaveNulls()) {


Remove this tracking once we remove seen

dnhatn · 2025-09-05T00:08:10Z

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

+                continue;
+            }
+            int groupId = groups.getInt(groupPosition);
+            state.set(groupId, state.getOrDefault(groupId) || values.getValueCount(position) > 0);


We don't to check getValueCount after checking isNull (line 88).

Thanks, I cleaned this up in all places within the class.

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

dnhatn · 2025-09-05T00:12:21Z

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

+            int groupEnd = groupStart + groups.getValueCount(groupPosition);
+            for (int g = groupStart; g < groupEnd; g++) {
+                int groupId = groups.getInt(g);
+                state.set(groupId, state.getOrDefault(groupId) || present.getBoolean(groupPosition + positionOffset));


Can we avoid reading the value from state?

if(present.getBoolean(groupPosition + positionOffset)){ state.set(groupId, true); }

Good point, thank you! Done!

dnhatn · 2025-09-05T00:24:44Z

...l/compute/src/main/java/org/elasticsearch/compute/aggregation/PresentAggregatorFunction.java

+        return INTERMEDIATE_STATE_DESC;
+    }
+
+    private final BooleanState state;


Can we remove the seen block and use a boolean to track present instead?

This is done. I use AtomicBoolean now to preserve the field count and have an ability to update it from the methods.

Can we avoid using AtomicBoolean since this will only be accessed by a single thread?

Got it! Removed. Thank you!

- Optimize AggregatorFunctions Part of elastic#131069

- Fix Rest Tests Part of elastic#131069

Optimize AggregatorFunction Part of elastic#131069

dnhatn

I've left some nits, but this looks great. Thanks Dima!

dnhatn · 2025-09-05T15:42:53Z

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

+        assert page.getBlockCount() >= blockIndex() + intermediateStateDesc().size();
+        BooleanVector present = page.<BooleanBlock>getBlock(channels.get(0)).asVector();
+        for (int groupPosition = 0; groupPosition < groups.getPositionCount(); groupPosition++) {
+            if (groups.isNull(groupPosition)) {


nit: can we read the value from the values block once each position?

if (groups.isNull(groupPosition) || present.getBoolean(groupPosition + positionOffset) == false) { continue; }

Improved! Thank you!

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

dnhatn · 2025-09-05T15:44:16Z

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

+        try (var valuesBuilder = driverContext.blockFactory().newBooleanBlockBuilder(selected.getPositionCount())) {
+            for (int i = 0; i < selected.getPositionCount(); i++) {
+                int group = selected.getInt(i);
+                if (group < state.size()) {


nit: We don't need to check the size; if it's out of range, BitArray will return false.

dnhatn · 2025-09-05T15:44:55Z

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java

+
+    @Override
+    public void evaluateIntermediate(Block[] blocks, int offset, IntVector selected) {
+        try (var valuesBuilder = driverContext.blockFactory().newBooleanBlockBuilder(selected.getPositionCount())) {


nit: can we use newBooleanVectorFixedBuilder instead?

Optimize PresentGroupingAggregatorFunction Part of elastic#131069

Add PresentErrorTests Part of elastic#131069

Add docs Part of elastic#131069

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/AggregateWritables.java

Add docs Part of elastic#131069

* ES|QL: Add PRESENT ES|QL function Add a new ES|QL function that checks for the presence of a field in the output result. Presence means that the input expression yields any non-null value. Part of elastic#131069 * [CI] Auto commit changes from spotless * ES|QL: Add PRESENT ES|QL function Add unit tests and documentation for the PRESENT function. Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function Clean-up of the PRESENT function. Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function - Change intermediate state for using boolean - Add unit tests for PresentAggregatorFunctionTests and PresentGroupingAggregatorFunctionTests Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function - Add VerifierTests Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function - Add union_types csv tests Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function - Fix unit tests Part of elastic#131069 * [CI] Auto commit changes from spotless * ES|QL: Add PRESENT_OVER_TIME ES|QL function - Comment out TestLogging on CsvTests - Add missing DataTypes to the function Part of elastic#131069 * [CI] Auto commit changes from spotless * ES|QL: Add PRESENT_OVER_TIME ES|QL function - Improve documentation Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function - Optimize AggregatorFunctions Part of elastic#131069 * [CI] Auto commit changes from spotless * ES|QL: Add PRESENT ES|QL function - Fix Rest Tests Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function Optimize AggregatorFunction Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function Optimize PresentGroupingAggregatorFunction Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function Add PresentErrorTests Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function Add docs Part of elastic#131069 * ES|QL: Add PRESENT ES|QL function Add docs Part of elastic#131069 --------- Co-authored-by: elasticsearchmachine <[email protected]>

ES|QL: Add PRESENT ES|QL function

0ba57ea

Add a new ES|QL function that checks for the presence of a field in the output result. Presence means that the input expression yields any non-null value. Part of elastic#131069

leontyevdv requested a review from dnhatn September 2, 2025 13:47

leontyevdv self-assigned this Sep 2, 2025

elasticsearchmachine added the v9.2.0 label Sep 2, 2025

elasticsearchmachine and others added 3 commits September 2, 2025 13:53

[CI] Auto commit changes from spotless

8bbf92c

ES|QL: Add PRESENT ES|QL function

004542b

Add unit tests and documentation for the PRESENT function. Part of elastic#131069

Merge remote-tracking branch 'origin/feature/esql-present-function' i…

134929b

…nto feature/esql-present-function

leontyevdv added 2 commits September 2, 2025 17:36

ES|QL: Add PRESENT ES|QL function

da9774f

Clean-up of the PRESENT function. Part of elastic#131069

Merge branch 'main' into feature/esql-present-function

06632d3

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

dnhatn reviewed Sep 2, 2025

View reviewed changes

...e/src/main/java/org/elasticsearch/compute/aggregation/PresentGroupingAggregatorFunction.java Outdated Show resolved Hide resolved

leontyevdv added 4 commits September 3, 2025 15:05

ES|QL: Add PRESENT ES|QL function

7f8d913

- Change intermediate state for using boolean - Add unit tests for PresentAggregatorFunctionTests and PresentGroupingAggregatorFunctionTests Part of elastic#131069

ES|QL: Add PRESENT ES|QL function

59d6b3b

- Add VerifierTests Part of elastic#131069

ES|QL: Add PRESENT ES|QL function

f537b8b

- Add union_types csv tests Part of elastic#131069

Merge branch 'main' into feature/esql-present-function

4c38aaa

leontyevdv marked this pull request as ready for review September 3, 2025 17:56

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Sep 3, 2025

leontyevdv and others added 2 commits September 4, 2025 11:45

ES|QL: Add PRESENT ES|QL function

0a5721f

- Fix unit tests Part of elastic#131069

[CI] Auto commit changes from spotless

bb54a7b

leontyevdv requested a review from dnhatn September 4, 2025 10:06

leontyevdv and others added 3 commits September 4, 2025 13:50

ES|QL: Add PRESENT_OVER_TIME ES|QL function

c2a7edb

- Comment out TestLogging on CsvTests - Add missing DataTypes to the function Part of elastic#131069

[CI] Auto commit changes from spotless

63fa255

ES|QL: Add PRESENT_OVER_TIME ES|QL function

ec176ee

- Improve documentation Part of elastic#131069

dnhatn reviewed Sep 5, 2025

View reviewed changes

leontyevdv and others added 3 commits September 5, 2025 12:47

ES|QL: Add PRESENT ES|QL function

bf0a02c

- Optimize AggregatorFunctions Part of elastic#131069

Merge branch 'main' into feature/esql-present-function

7b4d4cf

[CI] Auto commit changes from spotless

bbd5a18

leontyevdv requested a review from dnhatn September 5, 2025 11:46

ES|QL: Add PRESENT ES|QL function

d916601

- Fix Rest Tests Part of elastic#131069

leontyevdv mentioned this pull request Sep 5, 2025

ESQL PresentOverTime function leontyevdv/elasticsearch#1

Closed

ES|QL: Add PRESENT ES|QL function

55dcb0b

Optimize AggregatorFunction Part of elastic#131069

leontyevdv requested a review from not-napoleon September 5, 2025 15:30

dnhatn approved these changes Sep 5, 2025

View reviewed changes

leontyevdv added 5 commits September 8, 2025 12:42

ES|QL: Add PRESENT ES|QL function

23ccfe7

Optimize PresentGroupingAggregatorFunction Part of elastic#131069

Merge branch 'main' into feature/esql-present-function

236267b

ES|QL: Add PRESENT ES|QL function

3f79391

Add PresentErrorTests Part of elastic#131069

ES|QL: Add PRESENT ES|QL function

7cec7c3

Add docs Part of elastic#131069

Merge branch 'main' into feature/esql-present-function

3df69dc

leontyevdv added the test-release Trigger CI checks against release build label Sep 8, 2025

leontyevdv added 3 commits September 8, 2025 19:22

Merge branch 'main' into feature/esql-present-function

6e5cabd

Merge branch 'main' into feature/esql-present-function

20e4b1e

# Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/AggregateWritables.java

ES|QL: Add PRESENT ES|QL function

c6c62b5

Add docs Part of elastic#131069

leontyevdv merged commit cce52dd into elastic:main Sep 9, 2025
33 of 36 checks passed

leontyevdv mentioned this pull request Sep 9, 2025

ES|QL PresentOverTime function #134355

Merged

ES|QL: Add PRESENT ES|QL function #133986

ES|QL: Add PRESENT ES|QL function #133986

Uh oh!

Conversation

leontyevdv commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions bot commented Sep 2, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

Uh oh!

elasticsearchmachine commented Sep 4, 2025

Uh oh!

elasticsearchmachine commented Sep 4, 2025

Uh oh!

dnhatn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dnhatn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leontyevdv commented Sep 2, 2025 •

edited

Loading

github-actions bot commented Sep 2, 2025 •

edited

Loading