Skip to content

Use standard tokenizer in ES|QL categorize #129642

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 19, 2025
Merged

Use standard tokenizer in ES|QL categorize #129642

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
595674b
Use standard tokenizer in ES|QL categorize
jan-elastic Jun 18, 2025
e96dcaf
rename capability
jan-elastic Jun 18, 2025
f5e317e
remove unused param
jan-elastic Jun 18, 2025
8806f80
[CI] Auto commit changes from spotless
Jun 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions ...rc/main/java/org/elasticsearch/xpack/core/ml/job/config/CategorizationAnalyzerConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,23 @@ public static CategorizationAnalyzerConfig buildStandardCategorizationAnalyzer(L
.build();
}

/**
* Create a <code>categorization_analyzer</code> that will be used by the ES|QL categorize function.
* The only difference from the DSL analyzer is the tokenizer (standard instead of ml_standard).
* This means the results are slightly different from the categorize text aggregation and the ML job,
* however you can use these tokens for looking up messages in indices generated with the standard
* tokenizer. The latter is considered more important.
Copy link
Member

@davidkyle davidkyle Jun 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

*/
public static CategorizationAnalyzerConfig buildStandardEsqlCategorizationAnalyzer(List<String> categorizationFilters) {

return new CategorizationAnalyzerConfig.Builder().addCharFilter("first_line_with_letters")
.addCategorizationFilters(categorizationFilters)
.setTokenizer("standard")
.addDateWordsTokenFilter()
.addLimitFilter()
.build();
}

private final String analyzer;
private final List<NameOrDefinition> charFilters;
private final NameOrDefinition tokenizer;
Expand Down
5 changes: 2 additions & 3 deletions ...te/src/main/java/org/elasticsearch/compute/aggregation/blockhash/CategorizeBlockHash.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,8 @@
*/
public class CategorizeBlockHash extends BlockHash {

private static final CategorizationAnalyzerConfig ANALYZER_CONFIG = CategorizationAnalyzerConfig.buildStandardCategorizationAnalyzer(
List.of()
);
private static final CategorizationAnalyzerConfig ANALYZER_CONFIG = CategorizationAnalyzerConfig
.buildStandardEsqlCategorizationAnalyzer(List.of());
private static final int NULL_ORD = 0;

private final int channel;
Expand Down
14 changes: 7 additions & 7 deletions x-pack/plugin/esql/qa/testFixtures/src/main/resources/categorize.csv-spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,9 +256,9 @@ FROM hosts
;

COUNT():long | category:keyword
2 | .*?DB.+?servers.*?
2 | .*?Gateway.+?instances.*?
5 | .*?Kubernetes.+?cluster.*?
2 | .*?servers.*?
1 | null

// Note: DB is removed from "DB servers", because the ml_standard
Expand Down Expand Up @@ -302,9 +302,9 @@ FROM sample_data
;

COUNT():long | category:keyword
3 | .*?Connected.+?to.*?
3 | .*?Connection.+?error.*?
1 | .*?Disconnected.*?
3 | .*?Connected.+?to.+?👍🏽.+?😊.*?
3 | .*?Connection.+?error.+?👍🏽.+?😊.*?
1 | .*?Disconnected.+?👍🏽.+?😊.*?
;

on REVERSE(CONCAT())
Expand All @@ -316,9 +316,9 @@ FROM sample_data
;

COUNT():long | category:keyword
1 | .*?detcennocsiD.*?
3 | .*?ot.+?detcennoC.*?
3 | .*?rorre.+?noitcennoC.*?
1 | .*?😊.+?👍🏽.+?detcennocsiD.*?
3 | .*?😊.+?👍🏽.+?ot.+?detcennoC.*?
3 | .*?😊.+?👍🏽.+?rorre.+?noitcennoC.*?
;

and then TO_LOWER
Expand Down