Support geohash, geotile and geohex grid types #129581

craigtaverner · 2025-06-17T17:13:39Z

The PR at #125143 added support for ST_GEOHASH, ST_GEOTILE and ST_GEOHEX. However, since it used long as the internal type for the grid id, there was need for many additional functions for converting the long to and from keyword as well as generating geo_shape cell bounds for map display. Each pf these involved many more files (for the functions, their docs and the generated evaluators). With inspiration from PostGIS we decided to take a different direction, and instead use a new internal type for each grid:

geohash for the ST_GEOHASH function, created from literal using either TO_GEOHASH(hash) or hash::geohash
geotile for the ST_GEOTILE function, created from literal using either TO_GEOTILE(tile) or tile::geotile
geohex for the ST_GEOHEX function, created from literal using either TO_GEOHEX(h3) or h3::geohex

This also leads to much stricter type checking as we can no longer use the long as a plain long in all functions that accept longs, or inadvertently using a geohash in a geotile function. However, the addition of new types involves a lot of boilerplate, especially considering the large number of functions that operate on all types, and need to be informed of the existence of these three new types.

One of the main goals of this work was to also support the concept of a geogrid search. This means "find all documents that intersect a grid cell", similar to the Query DSL version. This is achieved by enabling the use of these three new types in the ST_INTERSECTS function, but that will be done in a followup PR.

Checklist

Move all new functions to SNAPSHOT
Move spatial search capabilities into a separate PR (ST_INTERSECTS and related) - Support geohash, geotile and geohex grid types in ST_INTERSECTS #133546
Make PR to move out of SNAPSHOT and link docs
Add issues for missing pieces:
- Support for shapes
- Lucene pushdown
- Missing evaluators (literals vs fields, separate evaluators for grid types, etc.)

elasticsearchmachine · 2025-06-17T17:14:04Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-06-17T17:14:05Z

Hi @craigtaverner, I've created a changelog YAML for you.

…search into esql_geogrid_types

Added EsqlCapabily to block older BWC Changed license IT to new API

iverase · 2025-06-19T10:00:40Z

.../org/elasticsearch/xpack/esql/expression/function/scalar/spatial/SpatialRelatesFunction.java

+                long geoGridId = getGridId(point, gridId, gridType);
+                return gridId == geoGridId;
+            } else {
+                throw new IllegalArgumentException(


This should be easy to implement though. It is ok to do it in a follow up

More over, I think we should do it generic, no point in distinguish a point from another geometry, we have code that that this already.

iverase

I like a lot this approach as a grid cell cannot be really represented by a geometry. I am +1 to this approach as far as ESQL folks are good with it.

alex-spies

~~We need to carefully consider how to deal with the breaking change that this proposes, because we remove some functions altogether.~~ Update: Breaking change itself was performed in #129839, this PR isn't breaking anymore but adds functionality.

Additionally, the new functions added by this aren't snapshot-only, either, and will be made available immediately. It's better to have them bake a little before un-snapshotting them in a dedicated PR.

Finally, this adds multiple data types and wires them immediately - I much prefer we do that in a dedicated PR, this is a big change and can easily introduce bugs.

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java

#125143 added 9 spatial grid functions and released them into Serverless. We think this is not the best long-term approach and the functions in #129581 are likely better. As a first step, rmove the spatial grid functions added in #125143 from release builds so they don't get released into 8.19/9.1. --------- Co-authored-by: Craig Taverner <[email protected]>

elastic#125143 added 9 spatial grid functions and released them into Serverless. We think this is not the best long-term approach and the functions in elastic#129581 are likely better. As a first step, rmove the spatial grid functions added in elastic#125143 from release builds so they don't get released into 8.19/9.1. --------- Co-authored-by: Craig Taverner <[email protected]> (cherry picked from commit efb1397) # Conflicts: # docs/reference/query-languages/esql/_snippets/lists/spatial-functions.md # docs/reference/query-languages/esql/functions-operators/spatial-functions.md

The simplification favoured placing the error on the first parameter, but we'd rather place it on the second.

x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DataType.java

alex-spies

This is lovely. Thanks for splitting this @craigtaverner, much much easier to digest this way.

I found one inconsistency in the behavior of the conversion functions that should be addressed: you cannot convert a geotile type to itself, in contrast to our other data types.

Otherwise I only have minor test suggestions.

Please proceed at your own discretion and feel free to address anything in follow-up PRs if it's easier to get this chonky PR merged, first; I imagine the merge conflicts are not fun to deal with.

alex-spies · 2025-08-29T09:04:53Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Count.java

Do we plan to support count_distinct in a follow-up? It currently raises a verification exception when used with geohash.

I investigated and it appears CountDistinct uses isExact to ensure that only actual fields (not reference attributes) are ever used. Not sure why this distinction is there, but it does block all types that are not fields (and the grid types are never es fields). So if we ever want to support this in future it is a bigger lift because it means supported non-fields types.

Sorry, my mistake, I read the code wrong. It does not require fields, but does have explicit support for each type, unlike Count. However, I can see that none of the spatial types are supported. It would be easy to support all spatial types as BytesRef, and grid types as long, but I think we should do that in a separate PR.

alex-spies · 2025-08-29T09:10:20Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/aggregate/Count.java

Since we add support for this, let's also add a csv test?

We could index some documents with keywords representing geohashes, sometimes being null. Then from idx | eval x = hash_as_keyword::geohash | stats count(x) should return the non-null count. And similarly for hex and tiles.

Also applies to the other agg functions that get geotile/hash/hex support in this PR.

Done. Updated the existing tests to use COUNT(grid) instead of COUNT(*).

alex-spies · 2025-08-29T09:17:31Z

.../src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/conditional/Case.java

I'd add a csv test that uses this with geotiles/hex/hashes.

alex-spies · 2025-08-29T09:33:11Z

...src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/convert/ToGeohash.java

The grid converters are missing the trivial conversion for the same type to itself. Our other converters support this, but for instance row x = (\"u3bu\"::geohash)::geohash raises a verification exception.

alex-spies · 2025-08-29T09:49:37Z

...c/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/multivalue/MvAppend.java

Let's also add csv tests with the MV_... functions and the new data types.

Done. Added an MV test for each testing a few MV functions, including the new MV_CONTAINS.

alex-spies · 2025-08-29T09:59:26Z

...ql/src/main/java/org/elasticsearch/xpack/esql/expression/function/scalar/nulls/Coalesce.java

Also here, a csv test to confirm this working e2e would be nice.

alex-spies · 2025-08-29T10:15:21Z

.../src/main/java/org/elasticsearch/xpack/esql/expression/predicate/operator/comparison/In.java

I think this one doesn't have a corresponding test update, neither in union tests nor in csv tests.

Oh, dear, this one is a bit of a can of worms, as there seems to be wider support in the operator than in covered by the tests, outside of grid support. Notably datetime. I think a separate issue to resolve all inconsistencies is needed here.

The PR at #129581 did not carefully check that all tests also work as release tests. This lead to some CI failures, which we are fixing here.

) In the work at #129581 we added new geo-grid types: `GEOHASH`, `GEOTILE` and `GEOHEX`, as well as support functions for creating these from `geo_point`, `long` and `keyword`. However, one of the key use cases we wish to replicate from the Query DSL is the grid search, and this involves the need to include the grid id inside a search predicate, in particular the `ST_INTERSECTS` function. For example: ``` FROM airports | WHERE ST_INTERSECTS(location, TO_GEOTILE("3/4/3")) | STATS count = COUNT(*), centroid = ST_CENTROID_AGG(location) ``` Since ST_INTERSECTS and ST_DISJOINT are converse of each other, we also added this support to that function.

The PR at #129581 added three new types: `geohash`, `geotile` and `geohex`, and support functions for creating these from strings, longs and geo_point fields. However all this was done under SNAPSHOT. Now it's time to move it into `tech-preview`.

Support geohash, geotile and geohex grid types

bf5c145

craigtaverner added >enhancement :Analytics/Geo Indexing, search aggregations of geo points and shapes Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :Analytics/ES|QL AKA ESQL v9.1.0 labels Jun 17, 2025

Update docs/changelog/129581.yaml

9963d1d

github-actions bot deployed to docs-preview June 17, 2025 17:14 View deployment

craigtaverner added 3 commits June 17, 2025 19:14

Use type-casting syntax to be sure it works

b84befe

Merge branch 'esql_geogrid_types' of github.com:craigtaverner/elastic…

98cc504

…search into esql_geogrid_types

Fix changelog yaml

a095433

github-actions bot deployed to docs-preview June 17, 2025 17:20 View deployment

craigtaverner requested review from ivancea, iverase and nik9000 June 17, 2025 17:22

craigtaverner added 3 commits June 18, 2025 13:56

New capability due to behaviour change

996d1ed

Fix failing tests

71ca7bb

Added EsqlCapabily to block older BWC Changed license IT to new API

Merge remote-tracking branch 'origin/main' into esql_geogrid_types

252f539

github-actions bot deployed to docs-preview June 18, 2025 12:16 View deployment

craigtaverner added 2 commits June 18, 2025 22:18

Disable broken lucene pushdown

625d918

Merge branch 'main' into esql_geogrid_types

5f3d5aa

iverase reviewed Jun 19, 2025

View reviewed changes

Fix failing tests

53ea500

alex-spies requested changes Jun 20, 2025

View reviewed changes

...sql/src/main/java/org/elasticsearch/xpack/esql/expression/function/EsqlFunctionRegistry.java Outdated Show resolved Hide resolved

alex-spies mentioned this pull request Jun 23, 2025

ESQL: Hide spatial grid functions behind SNAPSHOT #129839

Merged

craigtaverner added 8 commits August 26, 2025 10:54

Try absolute path

681dc30

Merge branch 'main' into esql_geogrid_types

ed4b6a3

Fixed failing tests after type resolution simplification

1c4ee72

The simplification favoured placing the error on the first parameter, but we'd rather place it on the second.

Return to single-line redirects

f12945e

Rename function to improve clarity

355c18d

Merge remote-tracking branch 'origin/main' into esql_geogrid_types

00ad3ce

Remove support for geogrid in ST_INTERSECTS

2d700d9

Bring back function supporting grid types in other tests

7f831c8

craigtaverner mentioned this pull request Aug 26, 2025

Support geohash, geotile and geohex grid types in ST_INTERSECTS #133546

Merged

craigtaverner added 3 commits August 26, 2025 16:32

Merge branch 'main' into esql_geogrid_types

534524c

Merge remote-tracking branch 'origin/main' into esql_geogrid_types

3de0b72

Fix evaluators after merging in main

32db040

alex-spies reviewed Aug 28, 2025

View reviewed changes

x-pack/plugin/esql-core/src/main/java/org/elasticsearch/xpack/esql/core/type/DataType.java Show resolved Hide resolved

alex-spies approved these changes Aug 29, 2025

View reviewed changes

craigtaverner added 9 commits August 30, 2025 12:22

Support self conversions for the three new types

8ee2891

Add csv tests for count(grid)

df81c7c

Added self-conversions to csv-spec tests

484d87e

Added CASE example to csv-tests

46eca05

Added MV functions to csv-tests

07b730d

Merge remote-tracking branch 'origin/main' into esql_geogrid_types

385fcce

Support MV_CONTAINS

e44c824

Add csv-spec tests for COALESCE

995724d

Updated docs after test updates

dfabb96

craigtaverner merged commit f7dd604 into elastic:main Aug 30, 2025
33 checks passed

craigtaverner mentioned this pull request Sep 1, 2025

Fix release tests after merging geogrid types #133929

Merged

craigtaverner added a commit that referenced this pull request Sep 1, 2025

Fix release tests after merging geogrid types (#133929)

f406f02

The PR at #129581 did not carefully check that all tests also work as release tests. This lead to some CI failures, which we are fixing here.

craigtaverner mentioned this pull request Sep 2, 2025

ES|QL Functions to generate geo_shape geometries from grid cells #128239

Closed

craigtaverner mentioned this pull request Sep 5, 2025

Move geo-grid functions out of SNAPSHOT and into tech-preview #134228

Merged

Support geohash, geotile and geohex grid types #129581

Support geohash, geotile and geohex grid types #129581

Uh oh!

Conversation

craigtaverner commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

elasticsearchmachine commented Jun 17, 2025

Uh oh!

elasticsearchmachine commented Jun 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iverase left a comment

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

craigtaverner commented Jun 17, 2025 •

edited

Loading

alex-spies left a comment •

edited

Loading