[ES|QL] Add automated script to sync operators and add support for MATCH operators #205565

qn895 · 2025-01-04T21:43:08Z

Closes #188978
Closes #199092

This PR addresses #199092 and adds automated script to sync operators and adds support for MATCH and its equivalent : match operator.

Screen.Recording.2025-01-17.at.15.36.30.mov

Screen.Recording.2025-01-17.at.15.36.11.mov

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

See some risk examples
...

# Conflicts: # packages/kbn-esql-validation-autocomplete/src/definitions/types.ts

# Conflicts: # src/platform/packages/shared/kbn-esql-validation-autocomplete/src/definitions/generated/operators.ts

qn895 · 2025-01-04T21:44:55Z

/ci

qn895 · 2025-01-05T23:20:54Z

/ci

stratoula

Thanx Quynh! I didnt test it but I have some questions to understand better the code. Also why do I see changes in the validation tests?

stratoula · 2025-01-08T12:31:17Z

src/platform/packages/shared/kbn-esql-validation-autocomplete/src/autocomplete/helper.ts

  }));
 }

+const leftHandSideParamName = new Set(['left', 'field', 'lhs']);


Haven't we mapped in the script above as left? Why do we need them here?

Good point! This was a left over. I removed it here 9f938ef (#205565)

stratoula · 2025-01-08T12:34:15Z

src/platform/packages/shared/kbn-esql-validation-autocomplete/src/definitions/types.ts

  str !== 'unsupported' &&
  (dataTypes.includes(str as SupportedDataType) || str === 'unknown' || str === 'any');

+export interface Signature {


stratoula · 2025-01-08T12:37:05Z

...kages/shared/kbn-esql-validation-autocomplete/src/validation/esql_validation_meta_tests.json

      "query": "row 1 years + 1 year",
      "error": [
-        "Argument of [+] must be [date], found value [1 years] type [duration]"
+        "Argument of [+] must be [date], found value [1 year] type [duration]"


So now we start the validation from the end and not from the beginning?

@qn895 can you explain to me these changes here?

stratoula · 2025-01-08T12:43:48Z

...rm/packages/shared/kbn-esql-validation-autocomplete/scripts/generate_function_definitions.ts

+    extraSignatures?: Signature[];
+  }
+> = {
+  add: {


So every time a new operator is being added we need to add an entry here? Can this info be given by ES? Have we talked with them?

Have you discussed this with Nik?

stratoula · 2025-01-08T12:46:06Z

src/platform/packages/shared/kbn-esql-validation-autocomplete/src/definitions/builtin.ts

 ].map((op): FunctionDefinition => createComparisonDefinition(op));

-const likeFunctions: FunctionDefinition[] = [
+const notLikeFunctions: FunctionDefinition[] = [


I dont understand this change, the not_like is not being returned from ES?

Yes, I left this notLikeFunctions here cause I didn't see an equivalent from Elasticsearch.

Can we ask them to add them ? (if you haven't done already)

qn895 · 2025-01-13T23:27:19Z

/ci

stratoula · 2025-01-23T08:45:51Z

@qn895 lets run the script when this PR gets merged elastic/elasticsearch#120504 which addresses the like and rlike operators.

This reverts commit 4e2bcd9.

stratoula

Impressive work, there are only 2 things I dont really get:

qn895 · 2025-01-28T16:04:37Z

For #205565 (files), in this case, this is coming from one of the Elasticsearch-generated signatures for add, which is more robust than what we had previously:


    {
      params: [
        {
          name: 'left',
          type: 'time_duration',
          optional: false,
        },
        {
          name: 'right',
          type: 'date_nanos',
          optional: false,
        },
      ],
      returnType: 'date_nanos',
    },

The validation goes from left to right, meaning it depends on what was previously typed before the operator. In this case, after 1 years +, it expects date type. I think it makes sense this way and instead of the previous way?

For #205565 (files), I'll investigate a bit more after FF (if possible) how to remove this meta completely by adding more info on the ES side. Some of the changes we need from ES are straightforward, but there's a few pieces in the current implementation that requires some rewiring.

# Conflicts: # src/platform/packages/shared/kbn-esql-validation-autocomplete/src/shared/constants.ts

elasticmachine · 2025-01-28T18:11:41Z

💚 Build Succeeded

Buildkite Build
Commit: f4b4229

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`cloudSecurityPosture`	774	775	+1
`controls`	472	473	+1
`dashboard`	746	747	+1
`data`	607	608	+1
`dataVisualizer`	816	817	+1
`discover`	1070	1071	+1
`esql`	210	211	+1
`esqlDataGrid`	499	500	+1
`eventAnnotationListing`	739	740	+1
`infra`	1301	1302	+1
`investigateApp`	666	667	+1
`lens`	1796	1797	+1
`lists`	430	431	+1
`logsExplorer`	696	697	+1
`maps`	1334	1335	+1
`ml`	2201	2202	+1
`observability`	1351	1352	+1
`observabilityAIAssistantApp`	504	505	+1
`securitySolution`	6593	6594	+1
`slo`	983	984	+1
`stackAlerts`	270	271	+1
`unifiedHistogram`	282	283	+1
`unifiedSearch`	392	393	+1
total			+23

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/esql-validation-autocomplete`	204	205	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	21.3MB	21.3MB	+42.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`kbnUiSharedDeps-srcJs`	3.5MB	3.6MB	+43.3KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/esql-validation-autocomplete`	216	217	+1

History

💚 Build #269829 succeeded 73dd937
💔 Build #269411 failed 4e2bcd9
💛 Build #268246 was flaky c4b47fd
💛 Build #267525 was flaky 261a7ac
💛 Build #266790 was flaky 8bec12f
💔 Build #266763 failed 9f938ef0c4364a1e386daefdb9b6a9cf25c27a23

cc @qn895

stratoula · 2025-01-29T07:44:01Z

@qn895

The validation goes from left to right, meaning it depends on what was previously typed before the operator. In this case, after 1 years +, it expects date type. I think it makes sense this way and instead of the previous way?

Ah I see now, this is much better yes 👍

I'll investigate a bit more after FF (if possible) how to remove this meta completely by adding more info on the ES side. Some of the changes we need from ES are straightforward, but there's a few pieces in the current implementation that requires some rewiring.

Sure it will be cool if we can remove this section. 🙏

kibanamachine · 2025-01-29T07:45:53Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/13026734780

…TCH operators (elastic#205565) Closes elastic#188978 Closes elastic#199092 This PR addresses elastic#199092 and adds automated script to sync operators and adds support for MATCH and its equivalent `:` match operator. https://github.com/user-attachments/assets/8f78345d-f1a0-4be4-8eba-2aae35225842 https://github.com/user-attachments/assets/eb9e63e3-0086-45a5-b782-199225895afc ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 6fed083)

kibanamachine · 2025-01-29T07:51:19Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…for MATCH operators (#205565) (#208678) # Backport This will backport the following commits from `main` to `8.x`: - [[ES|QL] Add automated script to sync operators and add support for MATCH operators (#205565)](#205565)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Quynh Nguyen (Quinn) <[email protected]>

## Summary in #205565 we accidentally removed the behavior where we don't show `NOT LIKE` and `NOT RLIKE` in the suggestions list. We want to show these eventually but right now, it's broken behavior: https://github.com/user-attachments/assets/0a7f860d-3a12-49cf-94b3-668848cff6ae --------- Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

## Summary in elastic#205565 we accidentally removed the behavior where we don't show `NOT LIKE` and `NOT RLIKE` in the suggestions list. We want to show these eventually but right now, it's broken behavior: https://github.com/user-attachments/assets/0a7f860d-3a12-49cf-94b3-668848cff6ae --------- Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 4970bb9)

## Summary in elastic#205565 we accidentally removed the behavior where we don't show `NOT LIKE` and `NOT RLIKE` in the suggestions list. We want to show these eventually but right now, it's broken behavior: https://github.com/user-attachments/assets/0a7f860d-3a12-49cf-94b3-668848cff6ae --------- Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 4970bb9) # Conflicts: # src/platform/packages/shared/kbn-esql-validation-autocomplete/src/definitions/generated/operators.ts

…216533) # Backport This will backport the following commits from `main` to `8.x`: - [[ES|QL] Hide "not" operators from suggestions menu (#216355)](#216355)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Drew Tate <[email protected]>

## Summary in elastic#205565 we accidentally removed the behavior where we don't show `NOT LIKE` and `NOT RLIKE` in the suggestions list. We want to show these eventually but right now, it's broken behavior: https://github.com/user-attachments/assets/0a7f860d-3a12-49cf-94b3-668848cff6ae --------- Co-authored-by: Stratoula Kalafateli <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

…216548) # Backport This will backport the following commits from `main` to `8.18`: - [[ES|QL] Hide "not" operators from suggestions menu (#216355)](#216355)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…216544) # Backport This will backport the following commits from `main` to `9.0`: - [[ES|QL] Hide "not" operators from suggestions menu (#216355)](#216355)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

qn895 added 14 commits November 13, 2024 18:09

Update script for operators

f8e9e16

Merge remote-tracking branch 'upstream/main' into esql-sync-operators

ccb590e

# Conflicts: # packages/kbn-esql-validation-autocomplete/src/definitions/types.ts

Update script to automatically sync operators

0918b0a

Update missing not_in, not_like, not_rlike

6a7cdde

Update script

41f7ea3

Update scripts

4f1e647

Update types

ebf0211

Update math function failures

4f6c9e9

Update tests

de6225e

Merge branch 'esql-sync-operators' into sync-esql-operators

6a8113b

# Conflicts: # src/platform/packages/shared/kbn-esql-validation-autocomplete/src/definitions/generated/operators.ts

Update tests

44749aa

Clean up

860878e

Pull operators latest elasticsearch snapshots

6b5c866

Update types

28388a6

Fix translations

1610c0a

qn895 self-assigned this Jan 6, 2025

qn895 added v9.0.0 Team:ESQL ES|QL related features in Kibana t// labels Jan 6, 2025

qn895 mentioned this pull request Jan 6, 2025

[ES|QL] Update script to sync operators automatically #200115

Closed

9 tasks

Update tests

847457d

qn895 requested a review from stratoula January 7, 2025 15:37

Merge branch 'main' into sync-esql-operators

acc0e79

stratoula reviewed Jan 8, 2025

View reviewed changes

qn895 added 3 commits January 13, 2025 15:51

Merge remote-tracking branch 'upstream/main' into sync-esql-operators

0f2776e

Clean up

93d1853

Update to latest, fix types

3073872

Clean up getSupportedTypesForBinaryOperators

599994b

qn895 added the v8.18.0 label Jan 23, 2025

qn895 added 5 commits January 23, 2025 13:13

Revert "Update suggest"

27d36cd

This reverts commit 4e2bcd9.

Fix previous commit deleting content

e144c1a

Update with latest from elasticsearch

951f39a

Merge remote-tracking branch 'upstream/main' into sync-esql-operators

a6fd992

Update latest from es again

73dd937

stratoula reviewed Jan 24, 2025

View reviewed changes

stratoula added release_note:skip Skip the PR/issue when compiling release notes and removed release_note:feature Makes this part of the condensed release notes labels Jan 28, 2025

Merge remote-tracking branch 'upstream/main' into sync-esql-operators

f4b4229

# Conflicts: # src/platform/packages/shared/kbn-esql-validation-autocomplete/src/shared/constants.ts

stratoula approved these changes Jan 29, 2025

View reviewed changes

stratoula merged commit 6fed083 into elastic:main Jan 29, 2025
9 checks passed

kibanamachine mentioned this pull request Jan 29, 2025

[8.x] [ES|QL] Add automated script to sync operators and add support for MATCH operators (#205565) #208678

Merged

drewdaemon mentioned this pull request Mar 28, 2025

[ES|QL] Hide "not" operators from suggestions menu #216355

Merged

[ES|QL] Add automated script to sync operators and add support for MATCH operators #205565

[ES|QL] Add automated script to sync operators and add support for MATCH operators #205565

Uh oh!

Conversation

qn895 commented Jan 4, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Identify risks

Uh oh!

qn895 commented Jan 4, 2025

Uh oh!

qn895 commented Jan 5, 2025

Uh oh!

stratoula left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qn895 commented Jan 13, 2025

Uh oh!

stratoula commented Jan 23, 2025

Uh oh!

stratoula left a comment

Choose a reason for hiding this comment

Uh oh!

qn895 commented Jan 28, 2025

Uh oh!

elasticmachine commented Jan 28, 2025

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

History

Uh oh!

stratoula commented Jan 29, 2025

Uh oh!

Uh oh!

kibanamachine commented Jan 29, 2025

Uh oh!

kibanamachine commented Jan 29, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

qn895 commented Jan 4, 2025 •

edited by kibanamachine

Loading