Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
27941a7
ES|QL: Add TBUCKET function
leontyevdv Jul 17, 2025
fe6dd2b
ES|QL: Add TBUCKET function
leontyevdv Jul 22, 2025
a5fe0fa
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Jul 22, 2025
45be0fe
ES|QL: Add TBUCKET function
leontyevdv Jul 24, 2025
e72467f
ES|QL: Add TBUCKET function
leontyevdv Jul 24, 2025
5bd6f36
ES|QL: Add TBUCKET function
leontyevdv Jul 24, 2025
e4a2e04
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Jul 24, 2025
cc16a42
ES|QL: Add TBUCKET function
leontyevdv Jul 24, 2025
84717ba
ES|QL: Add TBUCKET function
leontyevdv Jul 25, 2025
b378d10
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Jul 25, 2025
466e563
ES|QL: Add test for TBucket into the AnalyzerTests
leontyevdv Jul 25, 2025
06cd95a
ES|QL: Address PR review comments
leontyevdv Jul 28, 2025
6626682
ES|QL: Address PR review comments
leontyevdv Jul 29, 2025
8cfca3c
[CI] Auto commit changes from spotless
Jul 29, 2025
ad7fba6
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Jul 30, 2025
be418fc
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Jul 31, 2025
b0d5599
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Jul 31, 2025
f934369
ES|QL: Address PR review comments
leontyevdv Jul 31, 2025
28f0bea
ES|QL: Address PR review comments
leontyevdv Jul 31, 2025
d255687
ES|QL: Fix test
leontyevdv Jul 31, 2025
7216d45
ES|QL: Fix test
leontyevdv Jul 31, 2025
de606d1
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Jul 31, 2025
a57929e
ES|QL: Polish the code.
leontyevdv Aug 1, 2025
2eceb51
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 1, 2025
2fbd13e
ES|QL: Polish the code.
leontyevdv Aug 1, 2025
0dff032
ES|QL: Polish the code.
leontyevdv Aug 12, 2025
36063f0
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 12, 2025
8a90391
ES|QL: Add timestamp injection
leontyevdv Aug 14, 2025
a28a651
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 14, 2025
bfb23c3
ES|QL: Add timestamp injection
leontyevdv Aug 14, 2025
117eaac
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 18, 2025
03f45d9
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 20, 2025
cbe757b
ES|QL: Refactor tests to
leontyevdv Aug 20, 2025
bc23cdb
ES|QL: Refactor tests to
leontyevdv Aug 20, 2025
014fd83
ES|QL: Refactor tests to
leontyevdv Aug 20, 2025
8bf1c07
ES|QL: Refactor tests to
leontyevdv Aug 20, 2025
91d75a0
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 20, 2025
86b8d06
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 21, 2025
83bfd2a
ES|QL: Refactor tests to
leontyevdv Aug 22, 2025
ed92d11
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 22, 2025
92b02d1
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 25, 2025
2bf9111
ES|QL: Improve tests
leontyevdv Aug 25, 2025
24e8870
ES|QL: Improve tests
leontyevdv Aug 25, 2025
2e2ea88
ES|QL: Improve TBucket
leontyevdv Aug 26, 2025
ee5bfc0
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 26, 2025
3d52fbb
Merge branch 'main' into feature/esql-tbucket-function
leontyevdv Aug 26, 2025
5603fae
ES|QL: Improve tests
leontyevdv Aug 26, 2025
e74f6ac
ES|QL: Improve tests
leontyevdv Aug 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,10 @@ The [`STATS`](/reference/query-languages/esql/commands/processing-commands.md#es
:::{include} ../_snippets/functions/layout/bucket.md
:::

:::{note}
:::{include} ../_snippets/functions/layout/tbucket.md
:::

:::{note}
The `CATEGORIZE` function requires a [platinum license](https://www.elastic.co/subscriptions).
:::

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

This file was deleted.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be good to also include a test or two for TBUCKET in an eval; something like | EVAL key = TBUCKET(1 hour) | STATS minimum = MIN(whatever) BY key

Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
// TBUCKET-specific tests
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bucket can also be referenced outside of the BY clause in the STATS command, can we have some similar tests for TBucket to validate the transformation happens correctly?

reuseGroupingFunction#[skip:-8.13.99, reason:BUCKET renamed in 8.14]
FROM employees
| STATS sum = 1 + BUCKET(salary, 1000.) BY b1k = BUCKET(salary, 1000.)
| SORT sum
| LIMIT 4
;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm unable to write such a test because BUCKET on a numeric field returns a numeric value, which can legally participate in arithmetic in the STATS select list, even if the same BUCKET expression is also used as the grouping key. That is what the test mentioned is testing.

In case of TBUCKET, the error occurs because a grouping function that appears in the BY clause is treated as a grouping key and cannot simultaneously be used as an aggregate expression in the same STATS statement unless it’s referenced exactly the same way or via its alias from the BY list. ES|QL treats BUCKET as a grouping function; once it’s in BY, it can’t be re-used as an aggregate expression unless it’s just referenced as the exact same expression or via the alias. That triggers the verification exception about a grouping function being used as an aggregate after being declared in BY.

To confirm this I rewrote the test with BUCKET which led to the same verification exception:

FROM samle_data
| STATS sum = 1 + BUCKET(@timestamp, 1h) BY bh = BUCKET(@timestamp, 1h)
| SORT sum
| LIMIT 4

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you try this query to see if Bucket works for a similar pattern like the one below?

from sample_data
| stats x = 1 year + bucket(@timestamp, 1 day)  by  bucket(@timestamp, 1 day)

Copy link
Contributor Author

@leontyevdv leontyevdv Aug 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to test it missing the "year" after "1" 🤦🏻‍♂️ My bad. Your example works in both cases - tbucket and bucket. The test has been added to tbucket.csv-spec. Thanks for that!

Copy link
Contributor Author

@leontyevdv leontyevdv Aug 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It still didn't work due to the implicit nature of the @timestamp field and inability to resolve such fields. I found a workaround for that by creating a set of functions that require implicit timestamps, then recursively moving through the plan, checking if the functions require a timestamp and adding it to the nameList. See the changes in the class FieldNameUtils. Please, let me know what you think about this change and if there are better ways to implement this.


docsTBucketByTimeDuration#[skip:-9.1.99,reason:new grouping function added in 9.2]
required_capability: implicit_casting_string_literal_to_temporal_amount

// tag::docsTBucketByTimeDuration[]
FROM sample_data
| STATS min = MAX(@timestamp), max = MAX(@timestamp) BY bucket = TBUCKET(30 minutes)
| SORT min
// end::docsTBucketByTimeDuration[]
;

// tag::docsTBucketByTimeDuration-result[]
min:date | max:date | bucket:date
2023-10-23T12:27:28.948Z|2023-10-23T12:27:28.948Z|2023-10-23T12:00:00.000Z
2023-10-23T13:55:01.543Z|2023-10-23T13:55:01.543Z|2023-10-23T13:30:00.000Z
// end::docsTBucketByTimeDuration-result[]
;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we expect tbucket to support mixed/union typed fields? If so, there are some examples in union_types.csv-spec to verify how bucket works with union typed fields, and they can be used as a references to add more tbucket tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is expected. I've added a test to the union_types.csv-spec. Thank you!

Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@
import org.elasticsearch.xpack.esql.expression.function.fulltext.Term;
import org.elasticsearch.xpack.esql.expression.function.grouping.Bucket;
import org.elasticsearch.xpack.esql.expression.function.grouping.Categorize;
import org.elasticsearch.xpack.esql.expression.function.grouping.TBucket;
import org.elasticsearch.xpack.esql.expression.function.scalar.conditional.Case;
import org.elasticsearch.xpack.esql.expression.function.scalar.conditional.Greatest;
import org.elasticsearch.xpack.esql.expression.function.scalar.conditional.Least;
Expand Down Expand Up @@ -305,7 +306,8 @@ private static FunctionDefinition[][] functions() {
// grouping functions
new FunctionDefinition[] {
def(Bucket.class, Bucket::new, "bucket", "bin"),
def(Categorize.class, Categorize::new, "categorize") },
def(Categorize.class, Categorize::new, "categorize"),
def(TBucket.class, uni(TBucket::new), "tbucket") },
// aggregate functions
// since they declare two public constructors - one with filter (for nested where) and one without
// use casting to disambiguate between the two
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,6 @@
public class GroupingWritables {

public static List<NamedWriteableRegistry.Entry> getNamedWriteables() {
return List.of(Bucket.ENTRY, Categorize.ENTRY);
return List.of(Bucket.ENTRY, Categorize.ENTRY, TBucket.ENTRY);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed if tbucket is on coordinator node only?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not needed. Removed it from here and replaced code in the methods writeTo and getWriteableName to throw exceptions similarly to ToIp. Thank you!

}
}
Loading
Loading