ESQL: Update mechanism for under construction data types #135697

alex-spies · 2025-09-30T11:53:07Z

Follow-up to #135215

Fix the test expectations for our AllSupportedFieldsITs according to #135215 (comment).

Additionally, clean up our mechanism and docs for "under construction" data types:

For SNAPSHOT builds, go back to the old status quo where all types are considered supported and we just skip tests based on capabilities.
Re-brand the CreatedVersion for data types to SupportedVersion. For release builds, it doesn't matter when serialization code for a type was introduced; it matters only when it was actually released, and this is generally (much) later than when the serialization code for the type was written.
Make the SupportedVersion non-optional in the data type builder, so that we don't accidentally mark a new data type as "supported on all versions" in the future.
Remove the old feature-flag based system for marking types as "under construction" as we don't plan to use feature flags, but minimum transport versions to determine if a remote node/cluster supports a type.

Point 1. prevents having to update existing capabilities when a new type is released (which was required for #135604 and #135213) and thus gives a little bit of extra safety by keeping bwc tests intact (even if they technically test versions that don't officially support a type, yet).

Point 4. is lining up our data type system for #131108.

Some context on under-construction data types

We have data types that are considered under construction until they are released. Older nodes may not know these types at all (not able to deserialize), or they may know them but not actually support them.

Previously, we had a feature flag-based mechanism that would determine if a type is supported or not. Unfortunately, we do not check the feature flags on remote nodes and clusters, so this mechanism alone doesn't work; moreover, it leads to broken queries when the coordinator considers a type supported, but a remote node can't even deserialize it. The remote node will then not participate in simple queries like FROM logs-* and be counted as partial failure, even though it participated just fine in the past, when the data type was considered unsupported.

That was fixed for the aggregate_metric_double and dense_vector types, specifically, in the workaround #135215; these types now require the query to contain functions or commands that are only available when the types are actually released, otherwise we still consider these types as unsupported.

Of course, that's not a long term solution. Long term, we want to consider types as supported if the minimum transport version of all participating nodes is at least the version when the type was declared released. This is #131108. (See the ongoing work in #135633).

All the new types added in 9.2 will be considered supported by a remote node if its transport version is at least `INDEX_SOURCE`. We cannot add a new transport version and use that because it would break some queries already working on Serverless.

Rely on the SupportedVersion class, instead.

elasticsearchmachine · 2025-09-30T12:17:17Z

Hi @alex-spies, I've created a changelog YAML for you.

…_version_for_datatypes

elasticsearchmachine · 2025-10-01T11:03:07Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

alex-spies · 2025-10-01T11:33:21Z

.../plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/SupportedVersion.java

+        return new SupportedVersion() {
+            @Override
+            public boolean supports(TransportVersion version) {
+                return version.supports(supportedVersion) || Build.current().isSnapshot();


This is a subtle change that considers released types supported on snapshot builds, always.

Similarly, under construction types are also always considered supported on snapshot builds.

I thought about making a distinction between "created at version" and "supported from version", but that wouldn't really add anything as we already have capabilities to control the behavior of bwc tests in snapshot builds.

I'm good with just having the supported from version.

It does feel weird to say that released types are always supported on snapshots though.

I'll add a comment to explain that this is so that we don't break a bunch of existing tests when finally enabling a type.

Added some javadoc.

👍

This is an interesting compromise that makes tests flow more nicely. We don't discard any backwards compatibility tests that we wrote while working on the new type. That's good. But those tests might be guaranteeing more backwards compatibility than we actually offer in production. That's ok. And it makes it explicit when we disable that backward compatibility. All good.

nik9000 · 2025-10-01T12:30:20Z

.../plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/SupportedVersion.java

+        return new SupportedVersion() {
+            @Override
+            public boolean supports(TransportVersion version) {
+                return version.supports(supportedVersion) || Build.current().isSnapshot();


I'm good with just having the supported from version.

It does feel weird to say that released types are always supported on snapshots though.

nik9000 · 2025-10-01T12:35:18Z

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

    private static String[] removeUnderConstruction(String[] types) {
-        for (Map.Entry<DataType, FeatureFlag> underConstruction : DataType.UNDER_CONSTRUCTION.entrySet()) {
-            if (underConstruction.getValue().isEnabled() == false) {
-                types = Arrays.stream(types).filter(t -> underConstruction.getKey().typeName().equals(t) == false).toArray(String[]::new);


I suppose you could pass the feature flag into the under construction SupportVersion variant and use the enabled/disabled-ness of the feature flag. I'm wondering whether a createdVersion AND a featureFlag might be the right thing here. It's more work when you make a new data type, but it feels a little more true.

I guess you could have a feature-flag based override to enable under-construction types outside of SNAPSHOT; e.g. so that folks toying around with in-development data types could enable them on their local cluster (and live with the fact that queries might break in unexpected ways, esp. if connected to a remote cluster that may not enable the same feature flag).

That said, we currently have no under construction types anymore, so that would complicate the system for no gain. But I could leave a comment in place so that future-us will know how to do that.

Added a comment.

nik9000 · 2025-10-01T15:33:48Z

x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DataType.java

 *         EsqlDataTypeConverter#commonType, individual function type checking, the
 *         verifier rules, or other places. We suggest starting with CSV tests and
 *         seeing where they fail.</li>
+


Oh right.

I think it's fine to remove just that and not wait on the whole CI to run again before merging - that's what I'm gonna do.

nik9000 · 2025-10-01T15:40:40Z

.../plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/SupportedVersion.java

+        return new SupportedVersion() {
+            @Override
+            public boolean supports(TransportVersion version) {
+                return version.supports(supportedVersion) || Build.current().isSnapshot();


👍

This is an interesting compromise that makes tests flow more nicely. We don't discard any backwards compatibility tests that we wrote while working on the new type. That's good. But those tests might be guaranteeing more backwards compatibility than we actually offer in production. That's ok. And it makes it explicit when we disable that backward compatibility. All good.

alex-spies added 2 commits September 30, 2025 12:02

Choose a minimum transport version

9296d5c

All the new types added in 9.2 will be considered supported by a remote node if its transport version is at least `INDEX_SOURCE`. We cannot add a new transport version and use that because it would break some queries already working on Serverless.

Remove the UNDER_CONSTRUCTION feature flag usage

37fa745

Rely on the SupportedVersion class, instead.

elasticsearchmachine added the v9.2.0 label Sep 30, 2025

alex-spies added >bug :Analytics/ES|QL AKA ESQL labels Sep 30, 2025

Update docs/changelog/135697.yaml

b33037c

alex-spies added 4 commits October 1, 2025 10:55

Merge remote-tracking branch 'upstream/main' into introduce_supported…

2df04e7

…_version_for_datatypes

Update AllSupportedFieldsTestCase

1e2b18e

Fix some more tests

56614ae

Update comments on _tsid

95c5c54

alex-spies marked this pull request as ready for review October 1, 2025 11:02

alex-spies added test-full-bwc Trigger full BWC version matrix tests test-release Trigger CI checks against release build labels Oct 1, 2025

alex-spies requested a review from nik9000 October 1, 2025 11:02

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Oct 1, 2025

alex-spies commented Oct 1, 2025

View reviewed changes

alex-spies added >refactoring and removed >bug labels Oct 1, 2025

Delete docs/changelog/135697.yaml

c79630a

nik9000 reviewed Oct 1, 2025

View reviewed changes

Move comment

98ab058

elastic deleted a comment Oct 1, 2025

alex-spies added 2 commits October 1, 2025 15:48

Moar comments

fd1afad

Comment formatting

dc4b58f

elastic deleted a comment Oct 1, 2025

nik9000 approved these changes Oct 1, 2025

View reviewed changes

Remove extra line in javadoc

c297ffb

alex-spies merged commit d0d4339 into elastic:main Oct 1, 2025
7 of 30 checks passed

alex-spies deleted the introduce_supported_version_for_datatypes branch October 1, 2025 15:46

ESQL: Update mechanism for under construction data types #135697

ESQL: Update mechanism for under construction data types #135697

Uh oh!

Conversation

alex-spies commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Some context on under-construction data types

Uh oh!

elasticsearchmachine commented Sep 30, 2025

Uh oh!

elasticsearchmachine commented Oct 1, 2025

Uh oh!

alex-spies Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alex-spies commented Sep 30, 2025 •

edited

Loading

alex-spies Oct 1, 2025 •

edited

Loading