Skip to content

Conversation

fang-xing-esql
Copy link
Member

@fang-xing-esql fang-xing-esql commented Aug 22, 2025

This PR takes #127797 out of snapshot. It was gated behind snapshot by #130026 .

This was suggested originally by this issue - #110009.

Just want to provide some background information here again.
The goal of #127797 is to support union typed fields that reference both date and date_nanos fields without explicit casting. Union typed(mixed with date and date_nanos) fields are casted to date_nanos during Analyzer phase automatically. numeric or string typed fields are not within the scope of this feature.

Without this feature(release mode), queries that reference union(mixed with date and date_nanos) typed fields like below either return nulls or fail like the examples below, because they reference union typed fields without explicit casting.

mapping
curl -u elastic:password -X PUT "localhost:9200/sample_data_1?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "client.ip": {"type": "ip"},
      "message": {"type": "text"},
      "test_date": {"type": "date"},
      "event.duration": {"type": "long"}
    }
  }
}
'
curl -u elastic:password -X PUT "localhost:9200/sample_data_2?pretty" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "client.ip": {"type": "ip"},
      "message": {"type": "keyword"},
      "test_date": {"type": "date_nanos"},
      "event.duration": {"type": "integer"}
    }
  }
}
'
queries
+ curl -u elastic:password -X POST 'localhost:9200/_query?format=txt&pretty' -H 'Content-Type: application/json' '-d
{
  "query": "FROM sample_data*"
}
'
       @timestamp       |   client.ip   |event.duration |    message    |   test_date   
------------------------+---------------+---------------+---------------+---------------
2023-10-23T12:15:03.360Z|172.21.2.162   |null           |null           |null           
2023-10-23T12:27:28.948Z|172.21.2.113   |null           |null           |null           
2023-10-23T13:33:34.937Z|172.21.0.5     |null           |null           |null           
2023-10-23T13:51:54.732Z|172.21.3.15    |null           |null           |null           
2023-10-23T13:52:55.015Z|172.21.3.15    |null           |null           |null           
2023-10-23T13:53:55.832Z|172.21.3.15    |null           |null           |null           
2023-10-23T13:55:01.543Z|172.21.3.15    |null           |null           |null           
2023-10-23T12:15:03.360Z|172.22.2.162   |null           |null           |null           
2023-10-23T12:27:28.948Z|172.22.2.113   |null           |null           |null           
2023-10-23T13:33:34.937Z|172.22.0.5     |null           |null           |null           
2023-10-23T13:51:54.732Z|172.22.3.15    |null           |null           |null           
2023-10-23T13:52:55.015Z|172.22.3.15    |null           |null           |null           
2023-10-23T13:53:55.832Z|172.22.3.15    |null           |null           |null           
2023-10-23T13:55:01.543Z|172.22.3.15    |null           |null           |null           

+ curl -u elastic:password -X POST 'localhost:9200/_query?format=txt&pretty' -H 'Content-Type: application/json' '-d
{
  "query": "FROM sample_data* | sort test_date"
}
'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "verification_exception",
        "reason" : "Found 1 problem\nline 1:26: Cannot use field [test_date] due to ambiguities being mapped as [2] incompatible types: [date_nanos] in [sample_data_2], [datetime] in [sample_data_1]"
      }
    ],
    "type" : "verification_exception",
    "reason" : "Found 1 problem\nline 1:26: Cannot use field [test_date] due to ambiguities being mapped as [2] incompatible types: [date_nanos] in [sample_data_2], [datetime] in [sample_data_1]"
  },
  "status" : 400
}

With this feature(already behind snapshot), they should not fail or return null for values within date_nanos's valid ranges.

@fang-xing-esql fang-xing-esql requested a review from Copilot August 22, 2025 03:06
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR removes the snapshot requirement for the implicit casting functionality between date and date_nanos types in ES|QL, making it available in production builds. The feature was previously gated behind a snapshot-only capability flag.

  • Removes snapshot gating from the IMPLICIT_CASTING_DATE_AND_DATE_NANOS capability
  • Removes conditional test assumptions that required snapshot builds
  • Simplifies the DateMillisToNanosInEsRelation analyzer rule by removing snapshot checks

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
160_union_types.yml Adds aggregate_metric_double capability to test requirements
QueryTranslatorTests.java Removes snapshot assumption from date_nanos test
LocalPhysicalPlanOptimizerTests.java Removes snapshot assumption from date_nanos pushdown test
AnalyzerTests.java Removes snapshot assumption from implicit casting test
Analyzer.java Simplifies DateMillisToNanosInEsRelation rule by removing snapshot conditionals
EsqlCapabilities.java Removes snapshot requirement from IMPLICIT_CASTING_DATE_AND_DATE_NANOS capability
RestEsqlIT.java Removes snapshot check from suggested cast test logic

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@fang-xing-esql fang-xing-esql added :Analytics/ES|QL AKA ESQL >enhancement test-release Trigger CI checks against release build labels Aug 22, 2025
@elasticsearchmachine
Copy link
Collaborator

Hi @fang-xing-esql, I've created a changelog YAML for you.

Copy link
Contributor

github-actions bot commented Aug 28, 2025

🔍 Preview links for changed docs

@fang-xing-esql fang-xing-esql marked this pull request as ready for review August 28, 2025 03:38
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Aug 28, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@fang-xing-esql
Copy link
Member Author

The failed release tests are not related to this PR.

@alex-spies Could you help review this? This feature is ready to be taken out of snapshot according to the last few comments here
@leemthompo Could you help review the doc changes?

Thank you!

Copy link
Contributor

@alex-spies alex-spies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hiya, this looks good. Some additional suggestions for the docs, but as long as the (corresponding) release tests pass this should be good to go :)

@alex-spies
Copy link
Contributor

Also, sorry, qualifiers caused a conflict for you :/

Copy link
Contributor

@leemthompo leemthompo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice docs! Just couple very minor suggestions from me :)

Copy link
Contributor

github-actions bot commented Sep 2, 2025

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

@fang-xing-esql fang-xing-esql removed the test-release Trigger CI checks against release build label Sep 2, 2025
Copy link
Contributor

@leemthompo leemthompo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to version tag the docs nowadays :)

@fang-xing-esql
Copy link
Member Author

release tests passed locally. The test-release label was removed because there are irrelevant failures that are not caused by this PR.

@fang-xing-esql
Copy link
Member Author

Thanks so much for reviewing @alex-spies @leemthompo !

@fang-xing-esql fang-xing-esql merged commit 190a3f1 into elastic:main Sep 3, 2025
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >enhancement Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v9.2.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants