[Security][9.x] Add 'search.allow_expensive_queries' to detections-requirements.md #3543
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔍 Preview links for changed docs |
Member
Author
|
Ready to review. Thanks! 😁 🙏 |
7 tasks
sdesalas
changed the title
banderror
reviewed
Oct 20, 2025
Contributor
banderror
left a comment
banderror
left a comment
There was a problem hiding this comment.
Thanks @sdesalas for the important observation and opening this documentation improvement! I left a small suggestion.
Let's defer to @nastasha-solomon for a final review and approval.
sdesalas
force-pushed
the
main
branch
3 times, most recently
from
1316de5 to
7104b04
Compare
Co-authored-by: Georgii Gorbachev <[email protected]>
Contributor
nastasha-solomon
left a comment
nastasha-solomon
left a comment
There was a problem hiding this comment.
Left some suggestions for your consideration and a few questions as well. Thanks!
Co-authored-by: Nastasha Solomon <[email protected]>
Co-authored-by: Nastasha Solomon <[email protected]>
nastasha-solomon
approved these changes
Oct 21, 2025
Contributor
nastasha-solomon
left a comment
nastasha-solomon
left a comment
There was a problem hiding this comment.
Suggested changing the numbers to list bullets because these tasks don't need to be completed in a specific order. They just need to be completed in the same file (the elasticsearch.yml file).
Co-authored-by: Nastasha Solomon <[email protected]>
Co-authored-by: Nastasha Solomon <[email protected]>
nastasha-solomon
changed the title
Co-authored-by: Nastasha Solomon <[email protected]>
sdesalas
force-pushed
the
main
branch
2 times, most recently
from
809c4f9 to
f880825
Compare
sdesalas
force-pushed
the
main
branch
3 times, most recently
from
32cd1be to
24eac20
Compare
rhr323
pushed a commit
to rhr323/docs-content
that referenced
this pull request
Oct 27, 2025
…quirements.md (elastic#3543) ## Summary De-facto requirement of Security Detections, (and much of Kibana) that has not been documented properly. ``` # elasticsearch.yml search.allow_expensive_queries=false # <=== causes errors across Kibana 🤯 ``` Errors go as far back as `v8.18` (but likely much further), and not only affect Security app, but also many `Server Management` features such as Fleet, Saved Objects, Tags, etc. For more info: elastic/kibana#237496 -> Section **`Risks that were found acceptable`** We're documenting it [under the correct location](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) to make sure that users know about it. There is an [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469) for discussion. Note also that this is documented as a [requirement of the 'alerting' plugin](https://www.elastic.co/docs/explore-analyze/alerts-cases/alerts/alerting-setup#alerting-prerequisites) that we're using under the hood. <img width="800" alt="image" src="https://github.com/user-attachments/assets/512b57ed-d8e3-454b-a132-1776826d4a86" /> --------- Co-authored-by: Georgii Gorbachev <[email protected]> Co-authored-by: Nastasha Solomon <[email protected]>
sdesalas
added a commit
to elastic/kibana
that referenced
this pull request
Oct 28, 2025
… installed rules. (#237496) **Fixes: #237278** **Fixes: #97094** **Fixes: #194066** ## Summary When a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes to find all rules matching a partial string on the rule name (like `win` to get all rules about `Windows`) they receive 0 matches. This is in contrast to the behavior they experience when searching available prebuilt "Elastic Rules", where partial name searches are available. This PR fixes this UX inconsistency by modifying the KQL output generated by the search query. Whereas previously a search for `"win"` would have generated the following KQL: ```sql (alert.attributes.name: "win" OR alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` It now treats the `alert.attributes.name` differently, allowing it to match on partial terms (ie `*win*` instead of `"win"`) and using `.keyword` index for better special character support. ```sql (alert.attributes.name.keyword: *win* OR # <-- here alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` ### 🕵️ BUT! .. We only do this for single term searches! Please note that this approach only applies to single term searches. For multiple term searches we maintain the "old" way of searching with quotations `"windows 10 patch"` instead of `*windows 10 patch*` or even `*windows* *10* *patch*`. The reasoning here is that since search results are NOT sorted by score, we want to avoid returning too many matches (wildcard searches match on _any_ combination of the terms, ie those with `windows` or `10` or `patch`) which would confuse the user just as they are trying to narrow down their search results!! (ie 🤔 why am I getting `Linux patch` rules when I'm asking for `windows 10 patch`??). ## How to test: To test this PR please checkout the relevant branch and run an instance of kibana/elasticsearch locally. 1. `Security App` > `Rules` > `Detection Rules (SIEM)` 2. A table of installed rules should appear. Remove all installed Elastic rules. 3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules` > `Bulk actions` > `Delete` > `Delete` 4. Go to `Add Elastic Rules` 5. `Search rules by name` > `win` > `Enter` 6. You should see 5 pages of results (~84 rules) 7. `Install All` 8. All Elastic rules have been installed > `Go back to installed Elastic rules` 9. Search by rule name > `win` > `Enter` 10. You should see 5 pages of results (~84 rules - same as in Step 6.) Please feel free to try some additional search queries. <details> <summary> ### 👉 Click for additional testing ideas </summary> Here are some single term searches: - `goog` - `proc` - `sql` (should include `Postgresql` and `MSSQL`) - `shell` (should include `Powershell`) - `inject` (should include `injection`) - `git` (should include `GitHub`) - `.exe` - `pub/sub` (exact matches only) - `user-agent` (exact matches only) - `/bin` - `CVE-2025` (should include partial matches) - `-` (should return matches with dash in the name like `user-agent`) - `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`) - `|` (should return no matches - no elastic rules with this) And behavior for multiple term searches: - `AWS` (check the count), then `AWS IAM` (there should be less results for `AWS IAM`) - `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub top` partial match has no matches!) - `root` then `root cert` then `root certificate` (the second has no results, `root certificate` should only return exact matches) - `proc`, then `process`, then `process injection`, then `potential process injection` (each should return less results) </details> ## Screenshots  ## Special character support Note that by switching to wildcard searches (ie `*win*`) on the `.keyword` index and fully escaping special characters in KQL we'll **_ALSO_** be allowing special character searches on single search terms. For example, searching by `user-agent` will return results that only match `User-Agent` but not `user` or `agent` individually. Some other useful example of these approach are searches for: `Pub/Sub`, `CVE-2025`, `/bin`, `_`, `.exe`. This support, in addition for escaping the backslash `\` character will allow us to close the next TWO ISSUES in this epic 🥳 wohoo!! 👉 #97094 (special chars in rule name) and, 👉 #194066 (special chars in tags) <img width="800" alt="image" src="https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7" /> ### Testing special character support. In addition to the steps above (installing all elastic rules), we can: 1. Create a rule with special characters: ie `Rule with special chars (&, *, #, $, ?, >, @, \, /, ", ‘, {, [, ;)` 2. Try some single term matches: - `@` - `&` (additional elastic rules with this term should appear). - `*` - `"` - `:` - `>` - `{` - `\` (backslash, this used to break the search under #97094) ## Risks These are some of the risks that could be identified by using this approach. ### 1. Dependencies that reuse query logic Note that the `searchTerm -> KQL` conversion for searching for rules is used a few places. - Installed Rules - Rule Monitoring - Bulk actions **Note**: All these paths are tested manually, and form part of the automated tests. ### 2. Performance (ie `allowLeadingWildcards`) We're using wildcard searches before and after the search term (ie `*win*`) in order to replicate the behavior across both 'prebuilt' and 'installed' rules tables. The wildcard AFTER the term (`win*` => matching terms like `Windows`) is no problem but the one BEFORE (`*win` => matching terms like `Darwin`) could create an issue. Our [KQL documentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards) warns that Kibana UI Advanced Settings have `query:allowLeadingWildcards` turned off by default. This is for performance reasons as a leading wildcard can have a large impact when searching indexes that have millions of terms associated with them. <img width="2784" height="2120" alt="image" src="https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde" /> > By default, leading wildcards are not allowed for performance reasons. You can modify this with the [query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards) advanced setting. Please note that the [Query DSL docs also warn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard) about avoiding this approach. There is also more info and additional warnings under [Lucene API docs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html). The risk here is two pronged: 1. Users with a lot (millions?) of detection rules will likely have a degraded search experience as queries will take longer to execute. 2. Leading wildcards appear to be working by default (in contrast to what is stated in the documentation). But the mere _existence_ of various settings to avoid them ([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards), [`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard), [`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)) is a risk, because some users may have a special setup we've not been able to anticipate in our testing, leading to potential issues like this one: #57828 ### Mitigating factors: 1. Please note that in the case of security detection rules, the prebuilt rules package is _only ~1500 rules_! Users do not manage millions of rules, they generally manage low thousands or even hundreds. We've tested 100K rules successfully and there was [_no perceivable difference in terms of search performance_](#237496 (comment)). And this seems to be a realistic test when checking actual usage stats (our 10 largest users in Sep'25 had between 60-160K installed). Also, due to current limits on pagination and bulk action logic we would likely hit different kinds of problems here _before_ search performance becomes an issue. 2. We've tested all the documented ways to disable leading wildcards, including disabling them manually (under `Server Management > Advanced Settings`) and explicitly in the `kibana.yml`. None of them seemed to affect searches carried out on Saved Objects. This is because our solution uses the [`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting) plugin under the hood which [converts the filter to KQL without referring to any UI Advanced Settings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23). And since there is no explicit setting for `allowLeadingWildcards` the default setting of `true` gets applied instead [inside the `grammar.peggy` file](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12). ### Risks that were found acceptable In the search for any possible settings that might affect the rollout of changes under this PR [we did find a setting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7) inside `elasticsearch.yml` that causes problems with the proposed solution. **TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯 <img width="600" alt="image" src="https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44" /> However we also found bigger problems, _this setting also prevents the installation of prebuilt rules_ (see below), which are a cornerstone of our security solution. <img width="600" alt="image" src="https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1" /> In other words, the setting `search.allow_expensive_queries=true` has become _a de-facto requirement of Detection Rules_, that has not yet been documented. Hence we including it to the [documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) as part of this PR. Note also that the errors here are not limited to detection rules. We found that the setting _also compromised or outright broke a lot of functionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved Object search, Tags, Server Monitoring etc). More about this is [documented in this internal document](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0). And in this [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469). So we're assuming that most if not all of our users will have it set to `true`. ## Checklist Check the PR satisfies following conditions. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Follow the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. - [x] Changes have been socialized with the PM and rest of the team. - [x] All identified Risks have been properly documented and investigated. ([internal investigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0)) - [x] New requirements added to the technical docs (PR [#3543](elastic/docs-content#3543), see [here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 28, 2025
… installed rules. (elastic#237496) **Fixes: elastic#237278** **Fixes: elastic#97094** **Fixes: elastic#194066** ## Summary When a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes to find all rules matching a partial string on the rule name (like `win` to get all rules about `Windows`) they receive 0 matches. This is in contrast to the behavior they experience when searching available prebuilt "Elastic Rules", where partial name searches are available. This PR fixes this UX inconsistency by modifying the KQL output generated by the search query. Whereas previously a search for `"win"` would have generated the following KQL: ```sql (alert.attributes.name: "win" OR alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` It now treats the `alert.attributes.name` differently, allowing it to match on partial terms (ie `*win*` instead of `"win"`) and using `.keyword` index for better special character support. ```sql (alert.attributes.name.keyword: *win* OR # <-- here alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` ### 🕵️ BUT! .. We only do this for single term searches! Please note that this approach only applies to single term searches. For multiple term searches we maintain the "old" way of searching with quotations `"windows 10 patch"` instead of `*windows 10 patch*` or even `*windows* *10* *patch*`. The reasoning here is that since search results are NOT sorted by score, we want to avoid returning too many matches (wildcard searches match on _any_ combination of the terms, ie those with `windows` or `10` or `patch`) which would confuse the user just as they are trying to narrow down their search results!! (ie 🤔 why am I getting `Linux patch` rules when I'm asking for `windows 10 patch`??). ## How to test: To test this PR please checkout the relevant branch and run an instance of kibana/elasticsearch locally. 1. `Security App` > `Rules` > `Detection Rules (SIEM)` 2. A table of installed rules should appear. Remove all installed Elastic rules. 3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules` > `Bulk actions` > `Delete` > `Delete` 4. Go to `Add Elastic Rules` 5. `Search rules by name` > `win` > `Enter` 6. You should see 5 pages of results (~84 rules) 7. `Install All` 8. All Elastic rules have been installed > `Go back to installed Elastic rules` 9. Search by rule name > `win` > `Enter` 10. You should see 5 pages of results (~84 rules - same as in Step 6.) Please feel free to try some additional search queries. <details> <summary> ### 👉 Click for additional testing ideas </summary> Here are some single term searches: - `goog` - `proc` - `sql` (should include `Postgresql` and `MSSQL`) - `shell` (should include `Powershell`) - `inject` (should include `injection`) - `git` (should include `GitHub`) - `.exe` - `pub/sub` (exact matches only) - `user-agent` (exact matches only) - `/bin` - `CVE-2025` (should include partial matches) - `-` (should return matches with dash in the name like `user-agent`) - `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`) - `|` (should return no matches - no elastic rules with this) And behavior for multiple term searches: - `AWS` (check the count), then `AWS IAM` (there should be less results for `AWS IAM`) - `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub top` partial match has no matches!) - `root` then `root cert` then `root certificate` (the second has no results, `root certificate` should only return exact matches) - `proc`, then `process`, then `process injection`, then `potential process injection` (each should return less results) </details> ## Screenshots  ## Special character support Note that by switching to wildcard searches (ie `*win*`) on the `.keyword` index and fully escaping special characters in KQL we'll **_ALSO_** be allowing special character searches on single search terms. For example, searching by `user-agent` will return results that only match `User-Agent` but not `user` or `agent` individually. Some other useful example of these approach are searches for: `Pub/Sub`, `CVE-2025`, `/bin`, `_`, `.exe`. This support, in addition for escaping the backslash `\` character will allow us to close the next TWO ISSUES in this epic 🥳 wohoo!! 👉 elastic#97094 (special chars in rule name) and, 👉 elastic#194066 (special chars in tags) <img width="800" alt="image" src="https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7" /> ### Testing special character support. In addition to the steps above (installing all elastic rules), we can: 1. Create a rule with special characters: ie `Rule with special chars (&, *, #, $, ?, >, @, \, /, ", ‘, {, [, ;)` 2. Try some single term matches: - `@` - `&` (additional elastic rules with this term should appear). - `*` - `"` - `:` - `>` - `{` - `\` (backslash, this used to break the search under elastic#97094) ## Risks These are some of the risks that could be identified by using this approach. ### 1. Dependencies that reuse query logic Note that the `searchTerm -> KQL` conversion for searching for rules is used a few places. - Installed Rules - Rule Monitoring - Bulk actions **Note**: All these paths are tested manually, and form part of the automated tests. ### 2. Performance (ie `allowLeadingWildcards`) We're using wildcard searches before and after the search term (ie `*win*`) in order to replicate the behavior across both 'prebuilt' and 'installed' rules tables. The wildcard AFTER the term (`win*` => matching terms like `Windows`) is no problem but the one BEFORE (`*win` => matching terms like `Darwin`) could create an issue. Our [KQL documentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards) warns that Kibana UI Advanced Settings have `query:allowLeadingWildcards` turned off by default. This is for performance reasons as a leading wildcard can have a large impact when searching indexes that have millions of terms associated with them. <img width="2784" height="2120" alt="image" src="https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde" /> > By default, leading wildcards are not allowed for performance reasons. You can modify this with the [query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards) advanced setting. Please note that the [Query DSL docs also warn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard) about avoiding this approach. There is also more info and additional warnings under [Lucene API docs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html). The risk here is two pronged: 1. Users with a lot (millions?) of detection rules will likely have a degraded search experience as queries will take longer to execute. 2. Leading wildcards appear to be working by default (in contrast to what is stated in the documentation). But the mere _existence_ of various settings to avoid them ([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards), [`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard), [`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)) is a risk, because some users may have a special setup we've not been able to anticipate in our testing, leading to potential issues like this one: elastic#57828 ### Mitigating factors: 1. Please note that in the case of security detection rules, the prebuilt rules package is _only ~1500 rules_! Users do not manage millions of rules, they generally manage low thousands or even hundreds. We've tested 100K rules successfully and there was [_no perceivable difference in terms of search performance_](elastic#237496 (comment)). And this seems to be a realistic test when checking actual usage stats (our 10 largest users in Sep'25 had between 60-160K installed). Also, due to current limits on pagination and bulk action logic we would likely hit different kinds of problems here _before_ search performance becomes an issue. 2. We've tested all the documented ways to disable leading wildcards, including disabling them manually (under `Server Management > Advanced Settings`) and explicitly in the `kibana.yml`. None of them seemed to affect searches carried out on Saved Objects. This is because our solution uses the [`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting) plugin under the hood which [converts the filter to KQL without referring to any UI Advanced Settings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23). And since there is no explicit setting for `allowLeadingWildcards` the default setting of `true` gets applied instead [inside the `grammar.peggy` file](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12). ### Risks that were found acceptable In the search for any possible settings that might affect the rollout of changes under this PR [we did find a setting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7) inside `elasticsearch.yml` that causes problems with the proposed solution. **TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯 <img width="600" alt="image" src="https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44" /> However we also found bigger problems, _this setting also prevents the installation of prebuilt rules_ (see below), which are a cornerstone of our security solution. <img width="600" alt="image" src="https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1" /> In other words, the setting `search.allow_expensive_queries=true` has become _a de-facto requirement of Detection Rules_, that has not yet been documented. Hence we including it to the [documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) as part of this PR. Note also that the errors here are not limited to detection rules. We found that the setting _also compromised or outright broke a lot of functionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved Object search, Tags, Server Monitoring etc). More about this is [documented in this internal document](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0). And in this [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469). So we're assuming that most if not all of our users will have it set to `true`. ## Checklist Check the PR satisfies following conditions. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Follow the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. - [x] Changes have been socialized with the PM and rest of the team. - [x] All identified Risks have been properly documented and investigated. ([internal investigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0)) - [x] New requirements added to the technical docs (PR [elastic#3543](elastic/docs-content#3543), see [here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)) (cherry picked from commit 433902b)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 28, 2025
… installed rules. (elastic#237496) **Fixes: elastic#237278** **Fixes: elastic#97094** **Fixes: elastic#194066** ## Summary When a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes to find all rules matching a partial string on the rule name (like `win` to get all rules about `Windows`) they receive 0 matches. This is in contrast to the behavior they experience when searching available prebuilt "Elastic Rules", where partial name searches are available. This PR fixes this UX inconsistency by modifying the KQL output generated by the search query. Whereas previously a search for `"win"` would have generated the following KQL: ```sql (alert.attributes.name: "win" OR alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` It now treats the `alert.attributes.name` differently, allowing it to match on partial terms (ie `*win*` instead of `"win"`) and using `.keyword` index for better special character support. ```sql (alert.attributes.name.keyword: *win* OR # <-- here alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` ### 🕵️ BUT! .. We only do this for single term searches! Please note that this approach only applies to single term searches. For multiple term searches we maintain the "old" way of searching with quotations `"windows 10 patch"` instead of `*windows 10 patch*` or even `*windows* *10* *patch*`. The reasoning here is that since search results are NOT sorted by score, we want to avoid returning too many matches (wildcard searches match on _any_ combination of the terms, ie those with `windows` or `10` or `patch`) which would confuse the user just as they are trying to narrow down their search results!! (ie 🤔 why am I getting `Linux patch` rules when I'm asking for `windows 10 patch`??). ## How to test: To test this PR please checkout the relevant branch and run an instance of kibana/elasticsearch locally. 1. `Security App` > `Rules` > `Detection Rules (SIEM)` 2. A table of installed rules should appear. Remove all installed Elastic rules. 3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules` > `Bulk actions` > `Delete` > `Delete` 4. Go to `Add Elastic Rules` 5. `Search rules by name` > `win` > `Enter` 6. You should see 5 pages of results (~84 rules) 7. `Install All` 8. All Elastic rules have been installed > `Go back to installed Elastic rules` 9. Search by rule name > `win` > `Enter` 10. You should see 5 pages of results (~84 rules - same as in Step 6.) Please feel free to try some additional search queries. <details> <summary> ### 👉 Click for additional testing ideas </summary> Here are some single term searches: - `goog` - `proc` - `sql` (should include `Postgresql` and `MSSQL`) - `shell` (should include `Powershell`) - `inject` (should include `injection`) - `git` (should include `GitHub`) - `.exe` - `pub/sub` (exact matches only) - `user-agent` (exact matches only) - `/bin` - `CVE-2025` (should include partial matches) - `-` (should return matches with dash in the name like `user-agent`) - `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`) - `|` (should return no matches - no elastic rules with this) And behavior for multiple term searches: - `AWS` (check the count), then `AWS IAM` (there should be less results for `AWS IAM`) - `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub top` partial match has no matches!) - `root` then `root cert` then `root certificate` (the second has no results, `root certificate` should only return exact matches) - `proc`, then `process`, then `process injection`, then `potential process injection` (each should return less results) </details> ## Screenshots  ## Special character support Note that by switching to wildcard searches (ie `*win*`) on the `.keyword` index and fully escaping special characters in KQL we'll **_ALSO_** be allowing special character searches on single search terms. For example, searching by `user-agent` will return results that only match `User-Agent` but not `user` or `agent` individually. Some other useful example of these approach are searches for: `Pub/Sub`, `CVE-2025`, `/bin`, `_`, `.exe`. This support, in addition for escaping the backslash `\` character will allow us to close the next TWO ISSUES in this epic 🥳 wohoo!! 👉 elastic#97094 (special chars in rule name) and, 👉 elastic#194066 (special chars in tags) <img width="800" alt="image" src="https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7" /> ### Testing special character support. In addition to the steps above (installing all elastic rules), we can: 1. Create a rule with special characters: ie `Rule with special chars (&, *, #, $, ?, >, @, \, /, ", ‘, {, [, ;)` 2. Try some single term matches: - `@` - `&` (additional elastic rules with this term should appear). - `*` - `"` - `:` - `>` - `{` - `\` (backslash, this used to break the search under elastic#97094) ## Risks These are some of the risks that could be identified by using this approach. ### 1. Dependencies that reuse query logic Note that the `searchTerm -> KQL` conversion for searching for rules is used a few places. - Installed Rules - Rule Monitoring - Bulk actions **Note**: All these paths are tested manually, and form part of the automated tests. ### 2. Performance (ie `allowLeadingWildcards`) We're using wildcard searches before and after the search term (ie `*win*`) in order to replicate the behavior across both 'prebuilt' and 'installed' rules tables. The wildcard AFTER the term (`win*` => matching terms like `Windows`) is no problem but the one BEFORE (`*win` => matching terms like `Darwin`) could create an issue. Our [KQL documentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards) warns that Kibana UI Advanced Settings have `query:allowLeadingWildcards` turned off by default. This is for performance reasons as a leading wildcard can have a large impact when searching indexes that have millions of terms associated with them. <img width="2784" height="2120" alt="image" src="https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde" /> > By default, leading wildcards are not allowed for performance reasons. You can modify this with the [query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards) advanced setting. Please note that the [Query DSL docs also warn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard) about avoiding this approach. There is also more info and additional warnings under [Lucene API docs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html). The risk here is two pronged: 1. Users with a lot (millions?) of detection rules will likely have a degraded search experience as queries will take longer to execute. 2. Leading wildcards appear to be working by default (in contrast to what is stated in the documentation). But the mere _existence_ of various settings to avoid them ([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards), [`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard), [`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)) is a risk, because some users may have a special setup we've not been able to anticipate in our testing, leading to potential issues like this one: elastic#57828 ### Mitigating factors: 1. Please note that in the case of security detection rules, the prebuilt rules package is _only ~1500 rules_! Users do not manage millions of rules, they generally manage low thousands or even hundreds. We've tested 100K rules successfully and there was [_no perceivable difference in terms of search performance_](elastic#237496 (comment)). And this seems to be a realistic test when checking actual usage stats (our 10 largest users in Sep'25 had between 60-160K installed). Also, due to current limits on pagination and bulk action logic we would likely hit different kinds of problems here _before_ search performance becomes an issue. 2. We've tested all the documented ways to disable leading wildcards, including disabling them manually (under `Server Management > Advanced Settings`) and explicitly in the `kibana.yml`. None of them seemed to affect searches carried out on Saved Objects. This is because our solution uses the [`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting) plugin under the hood which [converts the filter to KQL without referring to any UI Advanced Settings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23). And since there is no explicit setting for `allowLeadingWildcards` the default setting of `true` gets applied instead [inside the `grammar.peggy` file](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12). ### Risks that were found acceptable In the search for any possible settings that might affect the rollout of changes under this PR [we did find a setting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7) inside `elasticsearch.yml` that causes problems with the proposed solution. **TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯 <img width="600" alt="image" src="https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44" /> However we also found bigger problems, _this setting also prevents the installation of prebuilt rules_ (see below), which are a cornerstone of our security solution. <img width="600" alt="image" src="https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1" /> In other words, the setting `search.allow_expensive_queries=true` has become _a de-facto requirement of Detection Rules_, that has not yet been documented. Hence we including it to the [documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) as part of this PR. Note also that the errors here are not limited to detection rules. We found that the setting _also compromised or outright broke a lot of functionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved Object search, Tags, Server Monitoring etc). More about this is [documented in this internal document](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0). And in this [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469). So we're assuming that most if not all of our users will have it set to `true`. ## Checklist Check the PR satisfies following conditions. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Follow the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. - [x] Changes have been socialized with the PM and rest of the team. - [x] All identified Risks have been properly documented and investigated. ([internal investigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0)) - [x] New requirements added to the technical docs (PR [elastic#3543](elastic/docs-content#3543), see [here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)) (cherry picked from commit 433902b)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 28, 2025
… installed rules. (elastic#237496) **Fixes: elastic#237278** **Fixes: elastic#97094** **Fixes: elastic#194066** ## Summary When a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes to find all rules matching a partial string on the rule name (like `win` to get all rules about `Windows`) they receive 0 matches. This is in contrast to the behavior they experience when searching available prebuilt "Elastic Rules", where partial name searches are available. This PR fixes this UX inconsistency by modifying the KQL output generated by the search query. Whereas previously a search for `"win"` would have generated the following KQL: ```sql (alert.attributes.name: "win" OR alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` It now treats the `alert.attributes.name` differently, allowing it to match on partial terms (ie `*win*` instead of `"win"`) and using `.keyword` index for better special character support. ```sql (alert.attributes.name.keyword: *win* OR # <-- here alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` ### 🕵️ BUT! .. We only do this for single term searches! Please note that this approach only applies to single term searches. For multiple term searches we maintain the "old" way of searching with quotations `"windows 10 patch"` instead of `*windows 10 patch*` or even `*windows* *10* *patch*`. The reasoning here is that since search results are NOT sorted by score, we want to avoid returning too many matches (wildcard searches match on _any_ combination of the terms, ie those with `windows` or `10` or `patch`) which would confuse the user just as they are trying to narrow down their search results!! (ie 🤔 why am I getting `Linux patch` rules when I'm asking for `windows 10 patch`??). ## How to test: To test this PR please checkout the relevant branch and run an instance of kibana/elasticsearch locally. 1. `Security App` > `Rules` > `Detection Rules (SIEM)` 2. A table of installed rules should appear. Remove all installed Elastic rules. 3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules` > `Bulk actions` > `Delete` > `Delete` 4. Go to `Add Elastic Rules` 5. `Search rules by name` > `win` > `Enter` 6. You should see 5 pages of results (~84 rules) 7. `Install All` 8. All Elastic rules have been installed > `Go back to installed Elastic rules` 9. Search by rule name > `win` > `Enter` 10. You should see 5 pages of results (~84 rules - same as in Step 6.) Please feel free to try some additional search queries. <details> <summary> ### 👉 Click for additional testing ideas </summary> Here are some single term searches: - `goog` - `proc` - `sql` (should include `Postgresql` and `MSSQL`) - `shell` (should include `Powershell`) - `inject` (should include `injection`) - `git` (should include `GitHub`) - `.exe` - `pub/sub` (exact matches only) - `user-agent` (exact matches only) - `/bin` - `CVE-2025` (should include partial matches) - `-` (should return matches with dash in the name like `user-agent`) - `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`) - `|` (should return no matches - no elastic rules with this) And behavior for multiple term searches: - `AWS` (check the count), then `AWS IAM` (there should be less results for `AWS IAM`) - `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub top` partial match has no matches!) - `root` then `root cert` then `root certificate` (the second has no results, `root certificate` should only return exact matches) - `proc`, then `process`, then `process injection`, then `potential process injection` (each should return less results) </details> ## Screenshots  ## Special character support Note that by switching to wildcard searches (ie `*win*`) on the `.keyword` index and fully escaping special characters in KQL we'll **_ALSO_** be allowing special character searches on single search terms. For example, searching by `user-agent` will return results that only match `User-Agent` but not `user` or `agent` individually. Some other useful example of these approach are searches for: `Pub/Sub`, `CVE-2025`, `/bin`, `_`, `.exe`. This support, in addition for escaping the backslash `\` character will allow us to close the next TWO ISSUES in this epic 🥳 wohoo!! 👉 elastic#97094 (special chars in rule name) and, 👉 elastic#194066 (special chars in tags) <img width="800" alt="image" src="https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7" /> ### Testing special character support. In addition to the steps above (installing all elastic rules), we can: 1. Create a rule with special characters: ie `Rule with special chars (&, *, #, $, ?, >, @, \, /, ", ‘, {, [, ;)` 2. Try some single term matches: - `@` - `&` (additional elastic rules with this term should appear). - `*` - `"` - `:` - `>` - `{` - `\` (backslash, this used to break the search under elastic#97094) ## Risks These are some of the risks that could be identified by using this approach. ### 1. Dependencies that reuse query logic Note that the `searchTerm -> KQL` conversion for searching for rules is used a few places. - Installed Rules - Rule Monitoring - Bulk actions **Note**: All these paths are tested manually, and form part of the automated tests. ### 2. Performance (ie `allowLeadingWildcards`) We're using wildcard searches before and after the search term (ie `*win*`) in order to replicate the behavior across both 'prebuilt' and 'installed' rules tables. The wildcard AFTER the term (`win*` => matching terms like `Windows`) is no problem but the one BEFORE (`*win` => matching terms like `Darwin`) could create an issue. Our [KQL documentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards) warns that Kibana UI Advanced Settings have `query:allowLeadingWildcards` turned off by default. This is for performance reasons as a leading wildcard can have a large impact when searching indexes that have millions of terms associated with them. <img width="2784" height="2120" alt="image" src="https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde" /> > By default, leading wildcards are not allowed for performance reasons. You can modify this with the [query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards) advanced setting. Please note that the [Query DSL docs also warn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard) about avoiding this approach. There is also more info and additional warnings under [Lucene API docs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html). The risk here is two pronged: 1. Users with a lot (millions?) of detection rules will likely have a degraded search experience as queries will take longer to execute. 2. Leading wildcards appear to be working by default (in contrast to what is stated in the documentation). But the mere _existence_ of various settings to avoid them ([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards), [`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard), [`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)) is a risk, because some users may have a special setup we've not been able to anticipate in our testing, leading to potential issues like this one: elastic#57828 ### Mitigating factors: 1. Please note that in the case of security detection rules, the prebuilt rules package is _only ~1500 rules_! Users do not manage millions of rules, they generally manage low thousands or even hundreds. We've tested 100K rules successfully and there was [_no perceivable difference in terms of search performance_](elastic#237496 (comment)). And this seems to be a realistic test when checking actual usage stats (our 10 largest users in Sep'25 had between 60-160K installed). Also, due to current limits on pagination and bulk action logic we would likely hit different kinds of problems here _before_ search performance becomes an issue. 2. We've tested all the documented ways to disable leading wildcards, including disabling them manually (under `Server Management > Advanced Settings`) and explicitly in the `kibana.yml`. None of them seemed to affect searches carried out on Saved Objects. This is because our solution uses the [`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting) plugin under the hood which [converts the filter to KQL without referring to any UI Advanced Settings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23). And since there is no explicit setting for `allowLeadingWildcards` the default setting of `true` gets applied instead [inside the `grammar.peggy` file](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12). ### Risks that were found acceptable In the search for any possible settings that might affect the rollout of changes under this PR [we did find a setting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7) inside `elasticsearch.yml` that causes problems with the proposed solution. **TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯 <img width="600" alt="image" src="https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44" /> However we also found bigger problems, _this setting also prevents the installation of prebuilt rules_ (see below), which are a cornerstone of our security solution. <img width="600" alt="image" src="https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1" /> In other words, the setting `search.allow_expensive_queries=true` has become _a de-facto requirement of Detection Rules_, that has not yet been documented. Hence we including it to the [documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) as part of this PR. Note also that the errors here are not limited to detection rules. We found that the setting _also compromised or outright broke a lot of functionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved Object search, Tags, Server Monitoring etc). More about this is [documented in this internal document](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0). And in this [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469). So we're assuming that most if not all of our users will have it set to `true`. ## Checklist Check the PR satisfies following conditions. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Follow the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. - [x] Changes have been socialized with the PM and rest of the team. - [x] All identified Risks have been properly documented and investigated. ([internal investigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0)) - [x] New requirements added to the technical docs (PR [elastic#3543](elastic/docs-content#3543), see [here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)) (cherry picked from commit 433902b)
kibanamachine
added a commit
to elastic/kibana
that referenced
this pull request
Oct 28, 2025
…rching installed rules. (#237496) (#240942) # Backport This will backport the following commits from `main` to `9.2`: - [[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)](#237496) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Steven de Salas","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-10-28T08:25:18Z","message":"[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)\n\n**Fixes: #237278**\n**Fixes: #97094**\n**Fixes: #194066**\n\n## Summary\n\nWhen a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes\nto find all rules matching a partial string on the rule name (like `win`\nto get all rules about `Windows`) they receive 0 matches. This is in\ncontrast to the behavior they experience when searching available\nprebuilt \"Elastic Rules\", where partial name searches are available.\n\nThis PR fixes this UX inconsistency by modifying the KQL output\ngenerated by the search query.\n\nWhereas previously a search for `\"win\"` would have generated the\nfollowing KQL:\n\n```sql\n(alert.attributes.name: \"win\" OR\n alert.attributes.params.index: \"win\" OR\n alert.attributes.params.threat.tactic.id: \"win\" OR\n ...\n```\n\nIt now treats the `alert.attributes.name` differently, allowing it to\nmatch on partial terms (ie `*win*` instead of `\"win\"`) and using\n`.keyword` index for better special character support.\n\n```sql\n(alert.attributes.name.keyword: *win* OR # <-- here\n alert.attributes.params.index: \"win\" OR \n alert.attributes.params.threat.tactic.id: \"win\" OR \n ...\n```\n\n### 🕵️ BUT! .. We only do this for single term searches!\n\nPlease note that this approach only applies to single term searches. For\nmultiple term searches we maintain the \"old\" way of searching with\nquotations `\"windows 10 patch\"` instead of `*windows 10 patch*` or even\n`*windows* *10* *patch*`.\n\nThe reasoning here is that since search results are NOT sorted by score,\nwe want to avoid returning too many matches (wildcard searches match on\n_any_ combination of the terms, ie those with `windows` or `10` or\n`patch`) which would confuse the user just as they are trying to narrow\ndown their search results!! (ie 🤔 why am I getting `Linux patch` rules\nwhen I'm asking for `windows 10 patch`??).\n\n## How to test:\n\nTo test this PR please checkout the relevant branch and run an instance\nof kibana/elasticsearch locally.\n\n1. `Security App` > `Rules` > `Detection Rules (SIEM)` \n2. A table of installed rules should appear. Remove all installed\nElastic rules.\n3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules`\n> `Bulk actions` > `Delete` > `Delete`\n4. Go to `Add Elastic Rules`\n5. `Search rules by name` > `win` > `Enter`\n6. You should see 5 pages of results (~84 rules)\n7. `Install All`\n8. All Elastic rules have been installed > `Go back to installed Elastic\nrules`\n9. Search by rule name > `win` > `Enter`\n10. You should see 5 pages of results (~84 rules - same as in Step 6.)\n\nPlease feel free to try some additional search queries. \n\n<details>\n<summary>\n\n### 👉 Click for additional testing ideas\n\n</summary>\n\nHere are some single term searches:\n - `goog`\n - `proc`\n - `sql` (should include `Postgresql` and `MSSQL`)\n - `shell` (should include `Powershell`)\n - `inject` (should include `injection`)\n - `git` (should include `GitHub`)\n - `.exe`\n - `pub/sub` (exact matches only)\n - `user-agent` (exact matches only)\n - `/bin`\n - `CVE-2025` (should include partial matches)\n - `-` (should return matches with dash in the name like `user-agent`)\n- `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`)\n - `|` (should return no matches - no elastic rules with this)\n\nAnd behavior for multiple term searches:\n- `AWS` (check the count), then `AWS IAM` (there should be less results\nfor `AWS IAM`)\n- `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub\ntop` partial match has no matches!)\n- `root` then `root cert` then `root certificate` (the second has no\nresults, `root certificate` should only return exact matches)\n- `proc`, then `process`, then `process injection`, then `potential\nprocess injection` (each should return less results)\n\n\n</details>\n\n## Screenshots\n\n\n\n\n\n\n## Special character support\n\nNote that by switching to wildcard searches (ie `*win*`) on the\n`.keyword` index and fully escaping special characters in KQL we'll\n**_ALSO_** be allowing special character searches on single search\nterms.\n\nFor example, searching by `user-agent` will return results that only\nmatch `User-Agent` but not `user` or `agent` individually.\n\nSome other useful example of these approach are searches for: `Pub/Sub`,\n`CVE-2025`, `/bin`, `_`, `.exe`.\n\nThis support, in addition for escaping the backslash `\\` character will\nallow us to close the next TWO ISSUES in this epic 🥳 wohoo!!\n\n👉 #97094 (special chars in rule name) and,\n👉 #194066 (special chars in tags)\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c\"\n/>\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7\"\n/>\n\n\n\n### Testing special character support.\n\nIn addition to the steps above (installing all elastic rules), we can:\n\n1. Create a rule with special characters: ie `Rule with special chars\n(&, *, #, $, ?, >, @, \\, /, \", ‘, {, [, ;)`\n2. Try some single term matches:\n - `@`\n - `&` (additional elastic rules with this term should appear).\n - `*` \n - `\"`\n - `:`\n - `>`\n - `{`\n - `\\` (backslash, this used to break the search under #97094)\n\n## Risks\n\nThese are some of the risks that could be identified by using this\napproach.\n\n### 1. Dependencies that reuse query logic\n\nNote that the `searchTerm -> KQL` conversion for searching for rules is\nused a few places.\n\n- Installed Rules\n- Rule Monitoring\n- Bulk actions\n\n**Note**: All these paths are tested manually, and form part of the\nautomated tests.\n\n### 2. Performance (ie `allowLeadingWildcards`)\n\nWe're using wildcard searches before and after the search term (ie\n`*win*`) in order to replicate the behavior across both 'prebuilt' and\n'installed' rules tables. The wildcard AFTER the term (`win*` =>\nmatching terms like `Windows`) is no problem but the one BEFORE (`*win`\n=> matching terms like `Darwin`) could create an issue.\n\nOur [KQL\ndocumentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards)\nwarns that Kibana UI Advanced Settings have\n`query:allowLeadingWildcards` turned off by default. This is for\nperformance reasons as a leading wildcard can have a large impact when\nsearching indexes that have millions of terms associated with them.\n\n<img width=\"2784\" height=\"2120\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde\"\n/>\n\n> By default, leading wildcards are not allowed for performance reasons.\nYou can modify this with the\n[query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards)\nadvanced setting.\n\nPlease note that the [Query DSL docs also\nwarn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)\nabout avoiding this approach.\n\nThere is also more info and additional warnings under [Lucene API\ndocs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html).\n\nThe risk here is two pronged:\n\n1. Users with a lot (millions?) of detection rules will likely have a\ndegraded search experience as queries will take longer to execute.\n\n2. Leading wildcards appear to be working by default (in contrast to\nwhat is stated in the documentation). But the mere _existence_ of\nvarious settings to avoid them\n([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards),\n[`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard),\n[`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard))\nis a risk, because some users may have a special setup we've not been\nable to anticipate in our testing, leading to potential issues like this\none: https://github.com/elastic/kibana/issues/57828\n\n### Mitigating factors:\n\n1. Please note that in the case of security detection rules, the\nprebuilt rules package is _only ~1500 rules_! Users do not manage\nmillions of rules, they generally manage low thousands or even hundreds.\nWe've tested 100K rules successfully and there was [_no perceivable\ndifference in terms of search\nperformance_](https://github.com/elastic/kibana/pull/237496#issuecomment-3389956730).\nAnd this seems to be a realistic test when checking actual usage stats\n(our 10 largest users in Sep'25 had between 60-160K installed). Also,\ndue to current limits on pagination and bulk action logic we would\nlikely hit different kinds of problems here _before_ search performance\nbecomes an issue.\n\n2. We've tested all the documented ways to disable leading wildcards,\nincluding disabling them manually (under `Server Management > Advanced\nSettings`) and explicitly in the `kibana.yml`. None of them seemed to\naffect searches carried out on Saved Objects. This is because our\nsolution uses the\n[`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting)\nplugin under the hood which [converts the filter to KQL without\nreferring to any UI Advanced\nSettings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23).\nAnd since there is no explicit setting for `allowLeadingWildcards` the\ndefault setting of `true` gets applied instead [inside the\n`grammar.peggy`\nfile](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12).\n\n### Risks that were found acceptable\n\nIn the search for any possible settings that might affect the rollout of\nchanges under this PR [we did find a\nsetting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7)\ninside `elasticsearch.yml` that causes problems with the proposed\nsolution.\n\n**TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44\"\n/>\n\nHowever we also found bigger problems, _this setting also prevents the\ninstallation of prebuilt rules_ (see below), which are a cornerstone of\nour security solution.\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1\"\n/>\n\nIn other words, the setting `search.allow_expensive_queries=true` has\nbecome _a de-facto requirement of Detection Rules_, that has not yet\nbeen documented. Hence we including it to the\n[documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)\nas part of this PR.\n\nNote also that the errors here are not limited to detection rules.\n\nWe found that the setting _also compromised or outright broke a lot of\nfunctionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved\nObject search, Tags, Server Monitoring etc). More about this is\n[documented in this internal\ndocument](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0).\nAnd in this [internal slack\nthread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469).\nSo we're assuming that most if not all of our users will have it set to\n`true`.\n\n## Checklist\n\nCheck the PR satisfies following conditions. \n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Follow the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n- [x] Changes have been socialized with the PM and rest of the team.\n- [x] All identified Risks have been properly documented and\ninvestigated. ([internal\ninvestigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0))\n- [x] New requirements added to the technical docs (PR\n[#3543](elastic/docs-content#3543), see\n[here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))","sha":"433902b9b40c9fb3f4db5346008a61efaed3df31","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","Team:Detections and Resp","Team: SecuritySolution","Feature:Rule Management","Team:Detection Rule Management","ci:cloud-deploy","ci:project-deploy-security","backport:version","v9.3.0","v8.19.7","v9.1.7","v9.2.1"],"title":"[Security Solution] Allow partial matches on rule name when searching installed rules.","number":237496,"url":"https://github.com/elastic/kibana/pull/237496","mergeCommit":{"message":"[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)\n\n**Fixes: #237278**\n**Fixes: #97094**\n**Fixes: #194066**\n\n## Summary\n\nWhen a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes\nto find all rules matching a partial string on the rule name (like `win`\nto get all rules about `Windows`) they receive 0 matches. This is in\ncontrast to the behavior they experience when searching available\nprebuilt \"Elastic Rules\", where partial name searches are available.\n\nThis PR fixes this UX inconsistency by modifying the KQL output\ngenerated by the search query.\n\nWhereas previously a search for `\"win\"` would have generated the\nfollowing KQL:\n\n```sql\n(alert.attributes.name: \"win\" OR\n alert.attributes.params.index: \"win\" OR\n alert.attributes.params.threat.tactic.id: \"win\" OR\n ...\n```\n\nIt now treats the `alert.attributes.name` differently, allowing it to\nmatch on partial terms (ie `*win*` instead of `\"win\"`) and using\n`.keyword` index for better special character support.\n\n```sql\n(alert.attributes.name.keyword: *win* OR # <-- here\n alert.attributes.params.index: \"win\" OR \n alert.attributes.params.threat.tactic.id: \"win\" OR \n ...\n```\n\n### 🕵️ BUT! .. We only do this for single term searches!\n\nPlease note that this approach only applies to single term searches. For\nmultiple term searches we maintain the \"old\" way of searching with\nquotations `\"windows 10 patch\"` instead of `*windows 10 patch*` or even\n`*windows* *10* *patch*`.\n\nThe reasoning here is that since search results are NOT sorted by score,\nwe want to avoid returning too many matches (wildcard searches match on\n_any_ combination of the terms, ie those with `windows` or `10` or\n`patch`) which would confuse the user just as they are trying to narrow\ndown their search results!! (ie 🤔 why am I getting `Linux patch` rules\nwhen I'm asking for `windows 10 patch`??).\n\n## How to test:\n\nTo test this PR please checkout the relevant branch and run an instance\nof kibana/elasticsearch locally.\n\n1. `Security App` > `Rules` > `Detection Rules (SIEM)` \n2. A table of installed rules should appear. Remove all installed\nElastic rules.\n3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules`\n> `Bulk actions` > `Delete` > `Delete`\n4. Go to `Add Elastic Rules`\n5. `Search rules by name` > `win` > `Enter`\n6. You should see 5 pages of results (~84 rules)\n7. `Install All`\n8. All Elastic rules have been installed > `Go back to installed Elastic\nrules`\n9. Search by rule name > `win` > `Enter`\n10. You should see 5 pages of results (~84 rules - same as in Step 6.)\n\nPlease feel free to try some additional search queries. \n\n<details>\n<summary>\n\n### 👉 Click for additional testing ideas\n\n</summary>\n\nHere are some single term searches:\n - `goog`\n - `proc`\n - `sql` (should include `Postgresql` and `MSSQL`)\n - `shell` (should include `Powershell`)\n - `inject` (should include `injection`)\n - `git` (should include `GitHub`)\n - `.exe`\n - `pub/sub` (exact matches only)\n - `user-agent` (exact matches only)\n - `/bin`\n - `CVE-2025` (should include partial matches)\n - `-` (should return matches with dash in the name like `user-agent`)\n- `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`)\n - `|` (should return no matches - no elastic rules with this)\n\nAnd behavior for multiple term searches:\n- `AWS` (check the count), then `AWS IAM` (there should be less results\nfor `AWS IAM`)\n- `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub\ntop` partial match has no matches!)\n- `root` then `root cert` then `root certificate` (the second has no\nresults, `root certificate` should only return exact matches)\n- `proc`, then `process`, then `process injection`, then `potential\nprocess injection` (each should return less results)\n\n\n</details>\n\n## Screenshots\n\n\n\n\n\n\n## Special character support\n\nNote that by switching to wildcard searches (ie `*win*`) on the\n`.keyword` index and fully escaping special characters in KQL we'll\n**_ALSO_** be allowing special character searches on single search\nterms.\n\nFor example, searching by `user-agent` will return results that only\nmatch `User-Agent` but not `user` or `agent` individually.\n\nSome other useful example of these approach are searches for: `Pub/Sub`,\n`CVE-2025`, `/bin`, `_`, `.exe`.\n\nThis support, in addition for escaping the backslash `\\` character will\nallow us to close the next TWO ISSUES in this epic 🥳 wohoo!!\n\n👉 #97094 (special chars in rule name) and,\n👉 #194066 (special chars in tags)\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c\"\n/>\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7\"\n/>\n\n\n\n### Testing special character support.\n\nIn addition to the steps above (installing all elastic rules), we can:\n\n1. Create a rule with special characters: ie `Rule with special chars\n(&, *, #, $, ?, >, @, \\, /, \", ‘, {, [, ;)`\n2. Try some single term matches:\n - `@`\n - `&` (additional elastic rules with this term should appear).\n - `*` \n - `\"`\n - `:`\n - `>`\n - `{`\n - `\\` (backslash, this used to break the search under #97094)\n\n## Risks\n\nThese are some of the risks that could be identified by using this\napproach.\n\n### 1. Dependencies that reuse query logic\n\nNote that the `searchTerm -> KQL` conversion for searching for rules is\nused a few places.\n\n- Installed Rules\n- Rule Monitoring\n- Bulk actions\n\n**Note**: All these paths are tested manually, and form part of the\nautomated tests.\n\n### 2. Performance (ie `allowLeadingWildcards`)\n\nWe're using wildcard searches before and after the search term (ie\n`*win*`) in order to replicate the behavior across both 'prebuilt' and\n'installed' rules tables. The wildcard AFTER the term (`win*` =>\nmatching terms like `Windows`) is no problem but the one BEFORE (`*win`\n=> matching terms like `Darwin`) could create an issue.\n\nOur [KQL\ndocumentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards)\nwarns that Kibana UI Advanced Settings have\n`query:allowLeadingWildcards` turned off by default. This is for\nperformance reasons as a leading wildcard can have a large impact when\nsearching indexes that have millions of terms associated with them.\n\n<img width=\"2784\" height=\"2120\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde\"\n/>\n\n> By default, leading wildcards are not allowed for performance reasons.\nYou can modify this with the\n[query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards)\nadvanced setting.\n\nPlease note that the [Query DSL docs also\nwarn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)\nabout avoiding this approach.\n\nThere is also more info and additional warnings under [Lucene API\ndocs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html).\n\nThe risk here is two pronged:\n\n1. Users with a lot (millions?) of detection rules will likely have a\ndegraded search experience as queries will take longer to execute.\n\n2. Leading wildcards appear to be working by default (in contrast to\nwhat is stated in the documentation). But the mere _existence_ of\nvarious settings to avoid them\n([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards),\n[`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard),\n[`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard))\nis a risk, because some users may have a special setup we've not been\nable to anticipate in our testing, leading to potential issues like this\none: https://github.com/elastic/kibana/issues/57828\n\n### Mitigating factors:\n\n1. Please note that in the case of security detection rules, the\nprebuilt rules package is _only ~1500 rules_! Users do not manage\nmillions of rules, they generally manage low thousands or even hundreds.\nWe've tested 100K rules successfully and there was [_no perceivable\ndifference in terms of search\nperformance_](https://github.com/elastic/kibana/pull/237496#issuecomment-3389956730).\nAnd this seems to be a realistic test when checking actual usage stats\n(our 10 largest users in Sep'25 had between 60-160K installed). Also,\ndue to current limits on pagination and bulk action logic we would\nlikely hit different kinds of problems here _before_ search performance\nbecomes an issue.\n\n2. We've tested all the documented ways to disable leading wildcards,\nincluding disabling them manually (under `Server Management > Advanced\nSettings`) and explicitly in the `kibana.yml`. None of them seemed to\naffect searches carried out on Saved Objects. This is because our\nsolution uses the\n[`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting)\nplugin under the hood which [converts the filter to KQL without\nreferring to any UI Advanced\nSettings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23).\nAnd since there is no explicit setting for `allowLeadingWildcards` the\ndefault setting of `true` gets applied instead [inside the\n`grammar.peggy`\nfile](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12).\n\n### Risks that were found acceptable\n\nIn the search for any possible settings that might affect the rollout of\nchanges under this PR [we did find a\nsetting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7)\ninside `elasticsearch.yml` that causes problems with the proposed\nsolution.\n\n**TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44\"\n/>\n\nHowever we also found bigger problems, _this setting also prevents the\ninstallation of prebuilt rules_ (see below), which are a cornerstone of\nour security solution.\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1\"\n/>\n\nIn other words, the setting `search.allow_expensive_queries=true` has\nbecome _a de-facto requirement of Detection Rules_, that has not yet\nbeen documented. Hence we including it to the\n[documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)\nas part of this PR.\n\nNote also that the errors here are not limited to detection rules.\n\nWe found that the setting _also compromised or outright broke a lot of\nfunctionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved\nObject search, Tags, Server Monitoring etc). More about this is\n[documented in this internal\ndocument](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0).\nAnd in this [internal slack\nthread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469).\nSo we're assuming that most if not all of our users will have it set to\n`true`.\n\n## Checklist\n\nCheck the PR satisfies following conditions. \n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Follow the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n- [x] Changes have been socialized with the PM and rest of the team.\n- [x] All identified Risks have been properly documented and\ninvestigated. ([internal\ninvestigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0))\n- [x] New requirements added to the technical docs (PR\n[#3543](elastic/docs-content#3543), see\n[here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))","sha":"433902b9b40c9fb3f4db5346008a61efaed3df31"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.1","9.2"],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/237496","number":237496,"mergeCommit":{"message":"[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)\n\n**Fixes: #237278**\n**Fixes: #97094**\n**Fixes: #194066**\n\n## Summary\n\nWhen a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes\nto find all rules matching a partial string on the rule name (like `win`\nto get all rules about `Windows`) they receive 0 matches. This is in\ncontrast to the behavior they experience when searching available\nprebuilt \"Elastic Rules\", where partial name searches are available.\n\nThis PR fixes this UX inconsistency by modifying the KQL output\ngenerated by the search query.\n\nWhereas previously a search for `\"win\"` would have generated the\nfollowing KQL:\n\n```sql\n(alert.attributes.name: \"win\" OR\n alert.attributes.params.index: \"win\" OR\n alert.attributes.params.threat.tactic.id: \"win\" OR\n ...\n```\n\nIt now treats the `alert.attributes.name` differently, allowing it to\nmatch on partial terms (ie `*win*` instead of `\"win\"`) and using\n`.keyword` index for better special character support.\n\n```sql\n(alert.attributes.name.keyword: *win* OR # <-- here\n alert.attributes.params.index: \"win\" OR \n alert.attributes.params.threat.tactic.id: \"win\" OR \n ...\n```\n\n### 🕵️ BUT! .. We only do this for single term searches!\n\nPlease note that this approach only applies to single term searches. For\nmultiple term searches we maintain the \"old\" way of searching with\nquotations `\"windows 10 patch\"` instead of `*windows 10 patch*` or even\n`*windows* *10* *patch*`.\n\nThe reasoning here is that since search results are NOT sorted by score,\nwe want to avoid returning too many matches (wildcard searches match on\n_any_ combination of the terms, ie those with `windows` or `10` or\n`patch`) which would confuse the user just as they are trying to narrow\ndown their search results!! (ie 🤔 why am I getting `Linux patch` rules\nwhen I'm asking for `windows 10 patch`??).\n\n## How to test:\n\nTo test this PR please checkout the relevant branch and run an instance\nof kibana/elasticsearch locally.\n\n1. `Security App` > `Rules` > `Detection Rules (SIEM)` \n2. A table of installed rules should appear. Remove all installed\nElastic rules.\n3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules`\n> `Bulk actions` > `Delete` > `Delete`\n4. Go to `Add Elastic Rules`\n5. `Search rules by name` > `win` > `Enter`\n6. You should see 5 pages of results (~84 rules)\n7. `Install All`\n8. All Elastic rules have been installed > `Go back to installed Elastic\nrules`\n9. Search by rule name > `win` > `Enter`\n10. You should see 5 pages of results (~84 rules - same as in Step 6.)\n\nPlease feel free to try some additional search queries. \n\n<details>\n<summary>\n\n### 👉 Click for additional testing ideas\n\n</summary>\n\nHere are some single term searches:\n - `goog`\n - `proc`\n - `sql` (should include `Postgresql` and `MSSQL`)\n - `shell` (should include `Powershell`)\n - `inject` (should include `injection`)\n - `git` (should include `GitHub`)\n - `.exe`\n - `pub/sub` (exact matches only)\n - `user-agent` (exact matches only)\n - `/bin`\n - `CVE-2025` (should include partial matches)\n - `-` (should return matches with dash in the name like `user-agent`)\n- `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`)\n - `|` (should return no matches - no elastic rules with this)\n\nAnd behavior for multiple term searches:\n- `AWS` (check the count), then `AWS IAM` (there should be less results\nfor `AWS IAM`)\n- `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub\ntop` partial match has no matches!)\n- `root` then `root cert` then `root certificate` (the second has no\nresults, `root certificate` should only return exact matches)\n- `proc`, then `process`, then `process injection`, then `potential\nprocess injection` (each should return less results)\n\n\n</details>\n\n## Screenshots\n\n\n\n\n\n\n## Special character support\n\nNote that by switching to wildcard searches (ie `*win*`) on the\n`.keyword` index and fully escaping special characters in KQL we'll\n**_ALSO_** be allowing special character searches on single search\nterms.\n\nFor example, searching by `user-agent` will return results that only\nmatch `User-Agent` but not `user` or `agent` individually.\n\nSome other useful example of these approach are searches for: `Pub/Sub`,\n`CVE-2025`, `/bin`, `_`, `.exe`.\n\nThis support, in addition for escaping the backslash `\\` character will\nallow us to close the next TWO ISSUES in this epic 🥳 wohoo!!\n\n👉 #97094 (special chars in rule name) and,\n👉 #194066 (special chars in tags)\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c\"\n/>\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7\"\n/>\n\n\n\n### Testing special character support.\n\nIn addition to the steps above (installing all elastic rules), we can:\n\n1. Create a rule with special characters: ie `Rule with special chars\n(&, *, #, $, ?, >, @, \\, /, \", ‘, {, [, ;)`\n2. Try some single term matches:\n - `@`\n - `&` (additional elastic rules with this term should appear).\n - `*` \n - `\"`\n - `:`\n - `>`\n - `{`\n - `\\` (backslash, this used to break the search under #97094)\n\n## Risks\n\nThese are some of the risks that could be identified by using this\napproach.\n\n### 1. Dependencies that reuse query logic\n\nNote that the `searchTerm -> KQL` conversion for searching for rules is\nused a few places.\n\n- Installed Rules\n- Rule Monitoring\n- Bulk actions\n\n**Note**: All these paths are tested manually, and form part of the\nautomated tests.\n\n### 2. Performance (ie `allowLeadingWildcards`)\n\nWe're using wildcard searches before and after the search term (ie\n`*win*`) in order to replicate the behavior across both 'prebuilt' and\n'installed' rules tables. The wildcard AFTER the term (`win*` =>\nmatching terms like `Windows`) is no problem but the one BEFORE (`*win`\n=> matching terms like `Darwin`) could create an issue.\n\nOur [KQL\ndocumentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards)\nwarns that Kibana UI Advanced Settings have\n`query:allowLeadingWildcards` turned off by default. This is for\nperformance reasons as a leading wildcard can have a large impact when\nsearching indexes that have millions of terms associated with them.\n\n<img width=\"2784\" height=\"2120\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde\"\n/>\n\n> By default, leading wildcards are not allowed for performance reasons.\nYou can modify this with the\n[query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards)\nadvanced setting.\n\nPlease note that the [Query DSL docs also\nwarn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)\nabout avoiding this approach.\n\nThere is also more info and additional warnings under [Lucene API\ndocs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html).\n\nThe risk here is two pronged:\n\n1. Users with a lot (millions?) of detection rules will likely have a\ndegraded search experience as queries will take longer to execute.\n\n2. Leading wildcards appear to be working by default (in contrast to\nwhat is stated in the documentation). But the mere _existence_ of\nvarious settings to avoid them\n([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards),\n[`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard),\n[`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard))\nis a risk, because some users may have a special setup we've not been\nable to anticipate in our testing, leading to potential issues like this\none: https://github.com/elastic/kibana/issues/57828\n\n### Mitigating factors:\n\n1. Please note that in the case of security detection rules, the\nprebuilt rules package is _only ~1500 rules_! Users do not manage\nmillions of rules, they generally manage low thousands or even hundreds.\nWe've tested 100K rules successfully and there was [_no perceivable\ndifference in terms of search\nperformance_](https://github.com/elastic/kibana/pull/237496#issuecomment-3389956730).\nAnd this seems to be a realistic test when checking actual usage stats\n(our 10 largest users in Sep'25 had between 60-160K installed). Also,\ndue to current limits on pagination and bulk action logic we would\nlikely hit different kinds of problems here _before_ search performance\nbecomes an issue.\n\n2. We've tested all the documented ways to disable leading wildcards,\nincluding disabling them manually (under `Server Management > Advanced\nSettings`) and explicitly in the `kibana.yml`. None of them seemed to\naffect searches carried out on Saved Objects. This is because our\nsolution uses the\n[`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting)\nplugin under the hood which [converts the filter to KQL without\nreferring to any UI Advanced\nSettings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23).\nAnd since there is no explicit setting for `allowLeadingWildcards` the\ndefault setting of `true` gets applied instead [inside the\n`grammar.peggy`\nfile](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12).\n\n### Risks that were found acceptable\n\nIn the search for any possible settings that might affect the rollout of\nchanges under this PR [we did find a\nsetting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7)\ninside `elasticsearch.yml` that causes problems with the proposed\nsolution.\n\n**TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44\"\n/>\n\nHowever we also found bigger problems, _this setting also prevents the\ninstallation of prebuilt rules_ (see below), which are a cornerstone of\nour security solution.\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1\"\n/>\n\nIn other words, the setting `search.allow_expensive_queries=true` has\nbecome _a de-facto requirement of Detection Rules_, that has not yet\nbeen documented. Hence we including it to the\n[documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)\nas part of this PR.\n\nNote also that the errors here are not limited to detection rules.\n\nWe found that the setting _also compromised or outright broke a lot of\nfunctionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved\nObject search, Tags, Server Monitoring etc). More about this is\n[documented in this internal\ndocument](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0).\nAnd in this [internal slack\nthread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469).\nSo we're assuming that most if not all of our users will have it set to\n`true`.\n\n## Checklist\n\nCheck the PR satisfies following conditions. \n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Follow the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n- [x] Changes have been socialized with the PM and rest of the team.\n- [x] All identified Risks have been properly documented and\ninvestigated. ([internal\ninvestigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0))\n- [x] New requirements added to the technical docs (PR\n[#3543](elastic/docs-content#3543), see\n[here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))","sha":"433902b9b40c9fb3f4db5346008a61efaed3df31"}},{"branch":"8.19","label":"v8.19.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.2","label":"v9.2.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Steven de Salas <[email protected]>
sdesalas
added a commit
to sdesalas/kibana
that referenced
this pull request
Oct 28, 2025
… installed rules. (elastic#237496) **Fixes: elastic#237278** **Fixes: elastic#97094** **Fixes: elastic#194066** ## Summary When a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes to find all rules matching a partial string on the rule name (like `win` to get all rules about `Windows`) they receive 0 matches. This is in contrast to the behavior they experience when searching available prebuilt "Elastic Rules", where partial name searches are available. This PR fixes this UX inconsistency by modifying the KQL output generated by the search query. Whereas previously a search for `"win"` would have generated the following KQL: ```sql (alert.attributes.name: "win" OR alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` It now treats the `alert.attributes.name` differently, allowing it to match on partial terms (ie `*win*` instead of `"win"`) and using `.keyword` index for better special character support. ```sql (alert.attributes.name.keyword: *win* OR # <-- here alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` ### 🕵️ BUT! .. We only do this for single term searches! Please note that this approach only applies to single term searches. For multiple term searches we maintain the "old" way of searching with quotations `"windows 10 patch"` instead of `*windows 10 patch*` or even `*windows* *10* *patch*`. The reasoning here is that since search results are NOT sorted by score, we want to avoid returning too many matches (wildcard searches match on _any_ combination of the terms, ie those with `windows` or `10` or `patch`) which would confuse the user just as they are trying to narrow down their search results!! (ie 🤔 why am I getting `Linux patch` rules when I'm asking for `windows 10 patch`??). ## How to test: To test this PR please checkout the relevant branch and run an instance of kibana/elasticsearch locally. 1. `Security App` > `Rules` > `Detection Rules (SIEM)` 2. A table of installed rules should appear. Remove all installed Elastic rules. 3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules` > `Bulk actions` > `Delete` > `Delete` 4. Go to `Add Elastic Rules` 5. `Search rules by name` > `win` > `Enter` 6. You should see 5 pages of results (~84 rules) 7. `Install All` 8. All Elastic rules have been installed > `Go back to installed Elastic rules` 9. Search by rule name > `win` > `Enter` 10. You should see 5 pages of results (~84 rules - same as in Step 6.) Please feel free to try some additional search queries. <details> <summary> ### 👉 Click for additional testing ideas </summary> Here are some single term searches: - `goog` - `proc` - `sql` (should include `Postgresql` and `MSSQL`) - `shell` (should include `Powershell`) - `inject` (should include `injection`) - `git` (should include `GitHub`) - `.exe` - `pub/sub` (exact matches only) - `user-agent` (exact matches only) - `/bin` - `CVE-2025` (should include partial matches) - `-` (should return matches with dash in the name like `user-agent`) - `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`) - `|` (should return no matches - no elastic rules with this) And behavior for multiple term searches: - `AWS` (check the count), then `AWS IAM` (there should be less results for `AWS IAM`) - `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub top` partial match has no matches!) - `root` then `root cert` then `root certificate` (the second has no results, `root certificate` should only return exact matches) - `proc`, then `process`, then `process injection`, then `potential process injection` (each should return less results) </details> ## Screenshots  ## Special character support Note that by switching to wildcard searches (ie `*win*`) on the `.keyword` index and fully escaping special characters in KQL we'll **_ALSO_** be allowing special character searches on single search terms. For example, searching by `user-agent` will return results that only match `User-Agent` but not `user` or `agent` individually. Some other useful example of these approach are searches for: `Pub/Sub`, `CVE-2025`, `/bin`, `_`, `.exe`. This support, in addition for escaping the backslash `\` character will allow us to close the next TWO ISSUES in this epic 🥳 wohoo!! 👉 elastic#97094 (special chars in rule name) and, 👉 elastic#194066 (special chars in tags) <img width="800" alt="image" src="https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7" /> ### Testing special character support. In addition to the steps above (installing all elastic rules), we can: 1. Create a rule with special characters: ie `Rule with special chars (&, *, #, $, ?, >, @, \, /, ", ‘, {, [, ;)` 2. Try some single term matches: - `@` - `&` (additional elastic rules with this term should appear). - `*` - `"` - `:` - `>` - `{` - `\` (backslash, this used to break the search under elastic#97094) ## Risks These are some of the risks that could be identified by using this approach. ### 1. Dependencies that reuse query logic Note that the `searchTerm -> KQL` conversion for searching for rules is used a few places. - Installed Rules - Rule Monitoring - Bulk actions **Note**: All these paths are tested manually, and form part of the automated tests. ### 2. Performance (ie `allowLeadingWildcards`) We're using wildcard searches before and after the search term (ie `*win*`) in order to replicate the behavior across both 'prebuilt' and 'installed' rules tables. The wildcard AFTER the term (`win*` => matching terms like `Windows`) is no problem but the one BEFORE (`*win` => matching terms like `Darwin`) could create an issue. Our [KQL documentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards) warns that Kibana UI Advanced Settings have `query:allowLeadingWildcards` turned off by default. This is for performance reasons as a leading wildcard can have a large impact when searching indexes that have millions of terms associated with them. <img width="2784" height="2120" alt="image" src="https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde" /> > By default, leading wildcards are not allowed for performance reasons. You can modify this with the [query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards) advanced setting. Please note that the [Query DSL docs also warn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard) about avoiding this approach. There is also more info and additional warnings under [Lucene API docs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html). The risk here is two pronged: 1. Users with a lot (millions?) of detection rules will likely have a degraded search experience as queries will take longer to execute. 2. Leading wildcards appear to be working by default (in contrast to what is stated in the documentation). But the mere _existence_ of various settings to avoid them ([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards), [`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard), [`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)) is a risk, because some users may have a special setup we've not been able to anticipate in our testing, leading to potential issues like this one: elastic#57828 ### Mitigating factors: 1. Please note that in the case of security detection rules, the prebuilt rules package is _only ~1500 rules_! Users do not manage millions of rules, they generally manage low thousands or even hundreds. We've tested 100K rules successfully and there was [_no perceivable difference in terms of search performance_](elastic#237496 (comment)). And this seems to be a realistic test when checking actual usage stats (our 10 largest users in Sep'25 had between 60-160K installed). Also, due to current limits on pagination and bulk action logic we would likely hit different kinds of problems here _before_ search performance becomes an issue. 2. We've tested all the documented ways to disable leading wildcards, including disabling them manually (under `Server Management > Advanced Settings`) and explicitly in the `kibana.yml`. None of them seemed to affect searches carried out on Saved Objects. This is because our solution uses the [`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting) plugin under the hood which [converts the filter to KQL without referring to any UI Advanced Settings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23). And since there is no explicit setting for `allowLeadingWildcards` the default setting of `true` gets applied instead [inside the `grammar.peggy` file](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12). ### Risks that were found acceptable In the search for any possible settings that might affect the rollout of changes under this PR [we did find a setting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7) inside `elasticsearch.yml` that causes problems with the proposed solution. **TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯 <img width="600" alt="image" src="https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44" /> However we also found bigger problems, _this setting also prevents the installation of prebuilt rules_ (see below), which are a cornerstone of our security solution. <img width="600" alt="image" src="https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1" /> In other words, the setting `search.allow_expensive_queries=true` has become _a de-facto requirement of Detection Rules_, that has not yet been documented. Hence we including it to the [documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) as part of this PR. Note also that the errors here are not limited to detection rules. We found that the setting _also compromised or outright broke a lot of functionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved Object search, Tags, Server Monitoring etc). More about this is [documented in this internal document](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0). And in this [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469). So we're assuming that most if not all of our users will have it set to `true`. ## Checklist Check the PR satisfies following conditions. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Follow the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. - [x] Changes have been socialized with the PM and rest of the team. - [x] All identified Risks have been properly documented and investigated. ([internal investigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0)) - [x] New requirements added to the technical docs (PR [elastic#3543](elastic/docs-content#3543), see [here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)) (cherry picked from commit 433902b)
kibanamachine
added a commit
to elastic/kibana
that referenced
this pull request
Oct 28, 2025
…arching installed rules. (#237496) (#240940) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)](#237496) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Steven de Salas","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-10-28T08:25:18Z","message":"[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)\n\n**Fixes: #237278**\n**Fixes: #97094**\n**Fixes: #194066**\n\n## Summary\n\nWhen a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes\nto find all rules matching a partial string on the rule name (like `win`\nto get all rules about `Windows`) they receive 0 matches. This is in\ncontrast to the behavior they experience when searching available\nprebuilt \"Elastic Rules\", where partial name searches are available.\n\nThis PR fixes this UX inconsistency by modifying the KQL output\ngenerated by the search query.\n\nWhereas previously a search for `\"win\"` would have generated the\nfollowing KQL:\n\n```sql\n(alert.attributes.name: \"win\" OR\n alert.attributes.params.index: \"win\" OR\n alert.attributes.params.threat.tactic.id: \"win\" OR\n ...\n```\n\nIt now treats the `alert.attributes.name` differently, allowing it to\nmatch on partial terms (ie `*win*` instead of `\"win\"`) and using\n`.keyword` index for better special character support.\n\n```sql\n(alert.attributes.name.keyword: *win* OR # <-- here\n alert.attributes.params.index: \"win\" OR \n alert.attributes.params.threat.tactic.id: \"win\" OR \n ...\n```\n\n### 🕵️ BUT! .. We only do this for single term searches!\n\nPlease note that this approach only applies to single term searches. For\nmultiple term searches we maintain the \"old\" way of searching with\nquotations `\"windows 10 patch\"` instead of `*windows 10 patch*` or even\n`*windows* *10* *patch*`.\n\nThe reasoning here is that since search results are NOT sorted by score,\nwe want to avoid returning too many matches (wildcard searches match on\n_any_ combination of the terms, ie those with `windows` or `10` or\n`patch`) which would confuse the user just as they are trying to narrow\ndown their search results!! (ie 🤔 why am I getting `Linux patch` rules\nwhen I'm asking for `windows 10 patch`??).\n\n## How to test:\n\nTo test this PR please checkout the relevant branch and run an instance\nof kibana/elasticsearch locally.\n\n1. `Security App` > `Rules` > `Detection Rules (SIEM)` \n2. A table of installed rules should appear. Remove all installed\nElastic rules.\n3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules`\n> `Bulk actions` > `Delete` > `Delete`\n4. Go to `Add Elastic Rules`\n5. `Search rules by name` > `win` > `Enter`\n6. You should see 5 pages of results (~84 rules)\n7. `Install All`\n8. All Elastic rules have been installed > `Go back to installed Elastic\nrules`\n9. Search by rule name > `win` > `Enter`\n10. You should see 5 pages of results (~84 rules - same as in Step 6.)\n\nPlease feel free to try some additional search queries. \n\n<details>\n<summary>\n\n### 👉 Click for additional testing ideas\n\n</summary>\n\nHere are some single term searches:\n - `goog`\n - `proc`\n - `sql` (should include `Postgresql` and `MSSQL`)\n - `shell` (should include `Powershell`)\n - `inject` (should include `injection`)\n - `git` (should include `GitHub`)\n - `.exe`\n - `pub/sub` (exact matches only)\n - `user-agent` (exact matches only)\n - `/bin`\n - `CVE-2025` (should include partial matches)\n - `-` (should return matches with dash in the name like `user-agent`)\n- `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`)\n - `|` (should return no matches - no elastic rules with this)\n\nAnd behavior for multiple term searches:\n- `AWS` (check the count), then `AWS IAM` (there should be less results\nfor `AWS IAM`)\n- `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub\ntop` partial match has no matches!)\n- `root` then `root cert` then `root certificate` (the second has no\nresults, `root certificate` should only return exact matches)\n- `proc`, then `process`, then `process injection`, then `potential\nprocess injection` (each should return less results)\n\n\n</details>\n\n## Screenshots\n\n\n\n\n\n\n## Special character support\n\nNote that by switching to wildcard searches (ie `*win*`) on the\n`.keyword` index and fully escaping special characters in KQL we'll\n**_ALSO_** be allowing special character searches on single search\nterms.\n\nFor example, searching by `user-agent` will return results that only\nmatch `User-Agent` but not `user` or `agent` individually.\n\nSome other useful example of these approach are searches for: `Pub/Sub`,\n`CVE-2025`, `/bin`, `_`, `.exe`.\n\nThis support, in addition for escaping the backslash `\\` character will\nallow us to close the next TWO ISSUES in this epic 🥳 wohoo!!\n\n👉 #97094 (special chars in rule name) and,\n👉 #194066 (special chars in tags)\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c\"\n/>\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7\"\n/>\n\n\n\n### Testing special character support.\n\nIn addition to the steps above (installing all elastic rules), we can:\n\n1. Create a rule with special characters: ie `Rule with special chars\n(&, *, #, $, ?, >, @, \\, /, \", ‘, {, [, ;)`\n2. Try some single term matches:\n - `@`\n - `&` (additional elastic rules with this term should appear).\n - `*` \n - `\"`\n - `:`\n - `>`\n - `{`\n - `\\` (backslash, this used to break the search under #97094)\n\n## Risks\n\nThese are some of the risks that could be identified by using this\napproach.\n\n### 1. Dependencies that reuse query logic\n\nNote that the `searchTerm -> KQL` conversion for searching for rules is\nused a few places.\n\n- Installed Rules\n- Rule Monitoring\n- Bulk actions\n\n**Note**: All these paths are tested manually, and form part of the\nautomated tests.\n\n### 2. Performance (ie `allowLeadingWildcards`)\n\nWe're using wildcard searches before and after the search term (ie\n`*win*`) in order to replicate the behavior across both 'prebuilt' and\n'installed' rules tables. The wildcard AFTER the term (`win*` =>\nmatching terms like `Windows`) is no problem but the one BEFORE (`*win`\n=> matching terms like `Darwin`) could create an issue.\n\nOur [KQL\ndocumentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards)\nwarns that Kibana UI Advanced Settings have\n`query:allowLeadingWildcards` turned off by default. This is for\nperformance reasons as a leading wildcard can have a large impact when\nsearching indexes that have millions of terms associated with them.\n\n<img width=\"2784\" height=\"2120\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde\"\n/>\n\n> By default, leading wildcards are not allowed for performance reasons.\nYou can modify this with the\n[query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards)\nadvanced setting.\n\nPlease note that the [Query DSL docs also\nwarn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)\nabout avoiding this approach.\n\nThere is also more info and additional warnings under [Lucene API\ndocs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html).\n\nThe risk here is two pronged:\n\n1. Users with a lot (millions?) of detection rules will likely have a\ndegraded search experience as queries will take longer to execute.\n\n2. Leading wildcards appear to be working by default (in contrast to\nwhat is stated in the documentation). But the mere _existence_ of\nvarious settings to avoid them\n([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards),\n[`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard),\n[`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard))\nis a risk, because some users may have a special setup we've not been\nable to anticipate in our testing, leading to potential issues like this\none: https://github.com/elastic/kibana/issues/57828\n\n### Mitigating factors:\n\n1. Please note that in the case of security detection rules, the\nprebuilt rules package is _only ~1500 rules_! Users do not manage\nmillions of rules, they generally manage low thousands or even hundreds.\nWe've tested 100K rules successfully and there was [_no perceivable\ndifference in terms of search\nperformance_](https://github.com/elastic/kibana/pull/237496#issuecomment-3389956730).\nAnd this seems to be a realistic test when checking actual usage stats\n(our 10 largest users in Sep'25 had between 60-160K installed). Also,\ndue to current limits on pagination and bulk action logic we would\nlikely hit different kinds of problems here _before_ search performance\nbecomes an issue.\n\n2. We've tested all the documented ways to disable leading wildcards,\nincluding disabling them manually (under `Server Management > Advanced\nSettings`) and explicitly in the `kibana.yml`. None of them seemed to\naffect searches carried out on Saved Objects. This is because our\nsolution uses the\n[`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting)\nplugin under the hood which [converts the filter to KQL without\nreferring to any UI Advanced\nSettings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23).\nAnd since there is no explicit setting for `allowLeadingWildcards` the\ndefault setting of `true` gets applied instead [inside the\n`grammar.peggy`\nfile](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12).\n\n### Risks that were found acceptable\n\nIn the search for any possible settings that might affect the rollout of\nchanges under this PR [we did find a\nsetting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7)\ninside `elasticsearch.yml` that causes problems with the proposed\nsolution.\n\n**TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44\"\n/>\n\nHowever we also found bigger problems, _this setting also prevents the\ninstallation of prebuilt rules_ (see below), which are a cornerstone of\nour security solution.\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1\"\n/>\n\nIn other words, the setting `search.allow_expensive_queries=true` has\nbecome _a de-facto requirement of Detection Rules_, that has not yet\nbeen documented. Hence we including it to the\n[documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)\nas part of this PR.\n\nNote also that the errors here are not limited to detection rules.\n\nWe found that the setting _also compromised or outright broke a lot of\nfunctionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved\nObject search, Tags, Server Monitoring etc). More about this is\n[documented in this internal\ndocument](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0).\nAnd in this [internal slack\nthread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469).\nSo we're assuming that most if not all of our users will have it set to\n`true`.\n\n## Checklist\n\nCheck the PR satisfies following conditions. \n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Follow the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n- [x] Changes have been socialized with the PM and rest of the team.\n- [x] All identified Risks have been properly documented and\ninvestigated. ([internal\ninvestigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0))\n- [x] New requirements added to the technical docs (PR\n[#3543](elastic/docs-content#3543), see\n[here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))","sha":"433902b9b40c9fb3f4db5346008a61efaed3df31","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","Team:Detections and Resp","Team: SecuritySolution","Feature:Rule Management","Team:Detection Rule Management","ci:cloud-deploy","ci:project-deploy-security","backport:version","v9.3.0","v8.19.7","v9.1.7","v9.2.1"],"title":"[Security Solution] Allow partial matches on rule name when searching installed rules.","number":237496,"url":"https://github.com/elastic/kibana/pull/237496","mergeCommit":{"message":"[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)\n\n**Fixes: #237278**\n**Fixes: #97094**\n**Fixes: #194066**\n\n## Summary\n\nWhen a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes\nto find all rules matching a partial string on the rule name (like `win`\nto get all rules about `Windows`) they receive 0 matches. This is in\ncontrast to the behavior they experience when searching available\nprebuilt \"Elastic Rules\", where partial name searches are available.\n\nThis PR fixes this UX inconsistency by modifying the KQL output\ngenerated by the search query.\n\nWhereas previously a search for `\"win\"` would have generated the\nfollowing KQL:\n\n```sql\n(alert.attributes.name: \"win\" OR\n alert.attributes.params.index: \"win\" OR\n alert.attributes.params.threat.tactic.id: \"win\" OR\n ...\n```\n\nIt now treats the `alert.attributes.name` differently, allowing it to\nmatch on partial terms (ie `*win*` instead of `\"win\"`) and using\n`.keyword` index for better special character support.\n\n```sql\n(alert.attributes.name.keyword: *win* OR # <-- here\n alert.attributes.params.index: \"win\" OR \n alert.attributes.params.threat.tactic.id: \"win\" OR \n ...\n```\n\n### 🕵️ BUT! .. We only do this for single term searches!\n\nPlease note that this approach only applies to single term searches. For\nmultiple term searches we maintain the \"old\" way of searching with\nquotations `\"windows 10 patch\"` instead of `*windows 10 patch*` or even\n`*windows* *10* *patch*`.\n\nThe reasoning here is that since search results are NOT sorted by score,\nwe want to avoid returning too many matches (wildcard searches match on\n_any_ combination of the terms, ie those with `windows` or `10` or\n`patch`) which would confuse the user just as they are trying to narrow\ndown their search results!! (ie 🤔 why am I getting `Linux patch` rules\nwhen I'm asking for `windows 10 patch`??).\n\n## How to test:\n\nTo test this PR please checkout the relevant branch and run an instance\nof kibana/elasticsearch locally.\n\n1. `Security App` > `Rules` > `Detection Rules (SIEM)` \n2. A table of installed rules should appear. Remove all installed\nElastic rules.\n3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules`\n> `Bulk actions` > `Delete` > `Delete`\n4. Go to `Add Elastic Rules`\n5. `Search rules by name` > `win` > `Enter`\n6. You should see 5 pages of results (~84 rules)\n7. `Install All`\n8. All Elastic rules have been installed > `Go back to installed Elastic\nrules`\n9. Search by rule name > `win` > `Enter`\n10. You should see 5 pages of results (~84 rules - same as in Step 6.)\n\nPlease feel free to try some additional search queries. \n\n<details>\n<summary>\n\n### 👉 Click for additional testing ideas\n\n</summary>\n\nHere are some single term searches:\n - `goog`\n - `proc`\n - `sql` (should include `Postgresql` and `MSSQL`)\n - `shell` (should include `Powershell`)\n - `inject` (should include `injection`)\n - `git` (should include `GitHub`)\n - `.exe`\n - `pub/sub` (exact matches only)\n - `user-agent` (exact matches only)\n - `/bin`\n - `CVE-2025` (should include partial matches)\n - `-` (should return matches with dash in the name like `user-agent`)\n- `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`)\n - `|` (should return no matches - no elastic rules with this)\n\nAnd behavior for multiple term searches:\n- `AWS` (check the count), then `AWS IAM` (there should be less results\nfor `AWS IAM`)\n- `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub\ntop` partial match has no matches!)\n- `root` then `root cert` then `root certificate` (the second has no\nresults, `root certificate` should only return exact matches)\n- `proc`, then `process`, then `process injection`, then `potential\nprocess injection` (each should return less results)\n\n\n</details>\n\n## Screenshots\n\n\n\n\n\n\n## Special character support\n\nNote that by switching to wildcard searches (ie `*win*`) on the\n`.keyword` index and fully escaping special characters in KQL we'll\n**_ALSO_** be allowing special character searches on single search\nterms.\n\nFor example, searching by `user-agent` will return results that only\nmatch `User-Agent` but not `user` or `agent` individually.\n\nSome other useful example of these approach are searches for: `Pub/Sub`,\n`CVE-2025`, `/bin`, `_`, `.exe`.\n\nThis support, in addition for escaping the backslash `\\` character will\nallow us to close the next TWO ISSUES in this epic 🥳 wohoo!!\n\n👉 #97094 (special chars in rule name) and,\n👉 #194066 (special chars in tags)\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c\"\n/>\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7\"\n/>\n\n\n\n### Testing special character support.\n\nIn addition to the steps above (installing all elastic rules), we can:\n\n1. Create a rule with special characters: ie `Rule with special chars\n(&, *, #, $, ?, >, @, \\, /, \", ‘, {, [, ;)`\n2. Try some single term matches:\n - `@`\n - `&` (additional elastic rules with this term should appear).\n - `*` \n - `\"`\n - `:`\n - `>`\n - `{`\n - `\\` (backslash, this used to break the search under #97094)\n\n## Risks\n\nThese are some of the risks that could be identified by using this\napproach.\n\n### 1. Dependencies that reuse query logic\n\nNote that the `searchTerm -> KQL` conversion for searching for rules is\nused a few places.\n\n- Installed Rules\n- Rule Monitoring\n- Bulk actions\n\n**Note**: All these paths are tested manually, and form part of the\nautomated tests.\n\n### 2. Performance (ie `allowLeadingWildcards`)\n\nWe're using wildcard searches before and after the search term (ie\n`*win*`) in order to replicate the behavior across both 'prebuilt' and\n'installed' rules tables. The wildcard AFTER the term (`win*` =>\nmatching terms like `Windows`) is no problem but the one BEFORE (`*win`\n=> matching terms like `Darwin`) could create an issue.\n\nOur [KQL\ndocumentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards)\nwarns that Kibana UI Advanced Settings have\n`query:allowLeadingWildcards` turned off by default. This is for\nperformance reasons as a leading wildcard can have a large impact when\nsearching indexes that have millions of terms associated with them.\n\n<img width=\"2784\" height=\"2120\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde\"\n/>\n\n> By default, leading wildcards are not allowed for performance reasons.\nYou can modify this with the\n[query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards)\nadvanced setting.\n\nPlease note that the [Query DSL docs also\nwarn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)\nabout avoiding this approach.\n\nThere is also more info and additional warnings under [Lucene API\ndocs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html).\n\nThe risk here is two pronged:\n\n1. Users with a lot (millions?) of detection rules will likely have a\ndegraded search experience as queries will take longer to execute.\n\n2. Leading wildcards appear to be working by default (in contrast to\nwhat is stated in the documentation). But the mere _existence_ of\nvarious settings to avoid them\n([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards),\n[`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard),\n[`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard))\nis a risk, because some users may have a special setup we've not been\nable to anticipate in our testing, leading to potential issues like this\none: https://github.com/elastic/kibana/issues/57828\n\n### Mitigating factors:\n\n1. Please note that in the case of security detection rules, the\nprebuilt rules package is _only ~1500 rules_! Users do not manage\nmillions of rules, they generally manage low thousands or even hundreds.\nWe've tested 100K rules successfully and there was [_no perceivable\ndifference in terms of search\nperformance_](https://github.com/elastic/kibana/pull/237496#issuecomment-3389956730).\nAnd this seems to be a realistic test when checking actual usage stats\n(our 10 largest users in Sep'25 had between 60-160K installed). Also,\ndue to current limits on pagination and bulk action logic we would\nlikely hit different kinds of problems here _before_ search performance\nbecomes an issue.\n\n2. We've tested all the documented ways to disable leading wildcards,\nincluding disabling them manually (under `Server Management > Advanced\nSettings`) and explicitly in the `kibana.yml`. None of them seemed to\naffect searches carried out on Saved Objects. This is because our\nsolution uses the\n[`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting)\nplugin under the hood which [converts the filter to KQL without\nreferring to any UI Advanced\nSettings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23).\nAnd since there is no explicit setting for `allowLeadingWildcards` the\ndefault setting of `true` gets applied instead [inside the\n`grammar.peggy`\nfile](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12).\n\n### Risks that were found acceptable\n\nIn the search for any possible settings that might affect the rollout of\nchanges under this PR [we did find a\nsetting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7)\ninside `elasticsearch.yml` that causes problems with the proposed\nsolution.\n\n**TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44\"\n/>\n\nHowever we also found bigger problems, _this setting also prevents the\ninstallation of prebuilt rules_ (see below), which are a cornerstone of\nour security solution.\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1\"\n/>\n\nIn other words, the setting `search.allow_expensive_queries=true` has\nbecome _a de-facto requirement of Detection Rules_, that has not yet\nbeen documented. Hence we including it to the\n[documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)\nas part of this PR.\n\nNote also that the errors here are not limited to detection rules.\n\nWe found that the setting _also compromised or outright broke a lot of\nfunctionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved\nObject search, Tags, Server Monitoring etc). More about this is\n[documented in this internal\ndocument](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0).\nAnd in this [internal slack\nthread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469).\nSo we're assuming that most if not all of our users will have it set to\n`true`.\n\n## Checklist\n\nCheck the PR satisfies following conditions. \n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Follow the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n- [x] Changes have been socialized with the PM and rest of the team.\n- [x] All identified Risks have been properly documented and\ninvestigated. ([internal\ninvestigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0))\n- [x] New requirements added to the technical docs (PR\n[#3543](elastic/docs-content#3543), see\n[here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))","sha":"433902b9b40c9fb3f4db5346008a61efaed3df31"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.1","9.2"],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/237496","number":237496,"mergeCommit":{"message":"[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)\n\n**Fixes: #237278**\n**Fixes: #97094**\n**Fixes: #194066**\n\n## Summary\n\nWhen a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes\nto find all rules matching a partial string on the rule name (like `win`\nto get all rules about `Windows`) they receive 0 matches. This is in\ncontrast to the behavior they experience when searching available\nprebuilt \"Elastic Rules\", where partial name searches are available.\n\nThis PR fixes this UX inconsistency by modifying the KQL output\ngenerated by the search query.\n\nWhereas previously a search for `\"win\"` would have generated the\nfollowing KQL:\n\n```sql\n(alert.attributes.name: \"win\" OR\n alert.attributes.params.index: \"win\" OR\n alert.attributes.params.threat.tactic.id: \"win\" OR\n ...\n```\n\nIt now treats the `alert.attributes.name` differently, allowing it to\nmatch on partial terms (ie `*win*` instead of `\"win\"`) and using\n`.keyword` index for better special character support.\n\n```sql\n(alert.attributes.name.keyword: *win* OR # <-- here\n alert.attributes.params.index: \"win\" OR \n alert.attributes.params.threat.tactic.id: \"win\" OR \n ...\n```\n\n### 🕵️ BUT! .. We only do this for single term searches!\n\nPlease note that this approach only applies to single term searches. For\nmultiple term searches we maintain the \"old\" way of searching with\nquotations `\"windows 10 patch\"` instead of `*windows 10 patch*` or even\n`*windows* *10* *patch*`.\n\nThe reasoning here is that since search results are NOT sorted by score,\nwe want to avoid returning too many matches (wildcard searches match on\n_any_ combination of the terms, ie those with `windows` or `10` or\n`patch`) which would confuse the user just as they are trying to narrow\ndown their search results!! (ie 🤔 why am I getting `Linux patch` rules\nwhen I'm asking for `windows 10 patch`??).\n\n## How to test:\n\nTo test this PR please checkout the relevant branch and run an instance\nof kibana/elasticsearch locally.\n\n1. `Security App` > `Rules` > `Detection Rules (SIEM)` \n2. A table of installed rules should appear. Remove all installed\nElastic rules.\n3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules`\n> `Bulk actions` > `Delete` > `Delete`\n4. Go to `Add Elastic Rules`\n5. `Search rules by name` > `win` > `Enter`\n6. You should see 5 pages of results (~84 rules)\n7. `Install All`\n8. All Elastic rules have been installed > `Go back to installed Elastic\nrules`\n9. Search by rule name > `win` > `Enter`\n10. You should see 5 pages of results (~84 rules - same as in Step 6.)\n\nPlease feel free to try some additional search queries. \n\n<details>\n<summary>\n\n### 👉 Click for additional testing ideas\n\n</summary>\n\nHere are some single term searches:\n - `goog`\n - `proc`\n - `sql` (should include `Postgresql` and `MSSQL`)\n - `shell` (should include `Powershell`)\n - `inject` (should include `injection`)\n - `git` (should include `GitHub`)\n - `.exe`\n - `pub/sub` (exact matches only)\n - `user-agent` (exact matches only)\n - `/bin`\n - `CVE-2025` (should include partial matches)\n - `-` (should return matches with dash in the name like `user-agent`)\n- `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`)\n - `|` (should return no matches - no elastic rules with this)\n\nAnd behavior for multiple term searches:\n- `AWS` (check the count), then `AWS IAM` (there should be less results\nfor `AWS IAM`)\n- `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub\ntop` partial match has no matches!)\n- `root` then `root cert` then `root certificate` (the second has no\nresults, `root certificate` should only return exact matches)\n- `proc`, then `process`, then `process injection`, then `potential\nprocess injection` (each should return less results)\n\n\n</details>\n\n## Screenshots\n\n\n\n\n\n\n## Special character support\n\nNote that by switching to wildcard searches (ie `*win*`) on the\n`.keyword` index and fully escaping special characters in KQL we'll\n**_ALSO_** be allowing special character searches on single search\nterms.\n\nFor example, searching by `user-agent` will return results that only\nmatch `User-Agent` but not `user` or `agent` individually.\n\nSome other useful example of these approach are searches for: `Pub/Sub`,\n`CVE-2025`, `/bin`, `_`, `.exe`.\n\nThis support, in addition for escaping the backslash `\\` character will\nallow us to close the next TWO ISSUES in this epic 🥳 wohoo!!\n\n👉 #97094 (special chars in rule name) and,\n👉 #194066 (special chars in tags)\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c\"\n/>\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7\"\n/>\n\n\n\n### Testing special character support.\n\nIn addition to the steps above (installing all elastic rules), we can:\n\n1. Create a rule with special characters: ie `Rule with special chars\n(&, *, #, $, ?, >, @, \\, /, \", ‘, {, [, ;)`\n2. Try some single term matches:\n - `@`\n - `&` (additional elastic rules with this term should appear).\n - `*` \n - `\"`\n - `:`\n - `>`\n - `{`\n - `\\` (backslash, this used to break the search under #97094)\n\n## Risks\n\nThese are some of the risks that could be identified by using this\napproach.\n\n### 1. Dependencies that reuse query logic\n\nNote that the `searchTerm -> KQL` conversion for searching for rules is\nused a few places.\n\n- Installed Rules\n- Rule Monitoring\n- Bulk actions\n\n**Note**: All these paths are tested manually, and form part of the\nautomated tests.\n\n### 2. Performance (ie `allowLeadingWildcards`)\n\nWe're using wildcard searches before and after the search term (ie\n`*win*`) in order to replicate the behavior across both 'prebuilt' and\n'installed' rules tables. The wildcard AFTER the term (`win*` =>\nmatching terms like `Windows`) is no problem but the one BEFORE (`*win`\n=> matching terms like `Darwin`) could create an issue.\n\nOur [KQL\ndocumentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards)\nwarns that Kibana UI Advanced Settings have\n`query:allowLeadingWildcards` turned off by default. This is for\nperformance reasons as a leading wildcard can have a large impact when\nsearching indexes that have millions of terms associated with them.\n\n<img width=\"2784\" height=\"2120\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde\"\n/>\n\n> By default, leading wildcards are not allowed for performance reasons.\nYou can modify this with the\n[query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards)\nadvanced setting.\n\nPlease note that the [Query DSL docs also\nwarn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)\nabout avoiding this approach.\n\nThere is also more info and additional warnings under [Lucene API\ndocs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html).\n\nThe risk here is two pronged:\n\n1. Users with a lot (millions?) of detection rules will likely have a\ndegraded search experience as queries will take longer to execute.\n\n2. Leading wildcards appear to be working by default (in contrast to\nwhat is stated in the documentation). But the mere _existence_ of\nvarious settings to avoid them\n([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards),\n[`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard),\n[`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard))\nis a risk, because some users may have a special setup we've not been\nable to anticipate in our testing, leading to potential issues like this\none: https://github.com/elastic/kibana/issues/57828\n\n### Mitigating factors:\n\n1. Please note that in the case of security detection rules, the\nprebuilt rules package is _only ~1500 rules_! Users do not manage\nmillions of rules, they generally manage low thousands or even hundreds.\nWe've tested 100K rules successfully and there was [_no perceivable\ndifference in terms of search\nperformance_](https://github.com/elastic/kibana/pull/237496#issuecomment-3389956730).\nAnd this seems to be a realistic test when checking actual usage stats\n(our 10 largest users in Sep'25 had between 60-160K installed). Also,\ndue to current limits on pagination and bulk action logic we would\nlikely hit different kinds of problems here _before_ search performance\nbecomes an issue.\n\n2. We've tested all the documented ways to disable leading wildcards,\nincluding disabling them manually (under `Server Management > Advanced\nSettings`) and explicitly in the `kibana.yml`. None of them seemed to\naffect searches carried out on Saved Objects. This is because our\nsolution uses the\n[`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting)\nplugin under the hood which [converts the filter to KQL without\nreferring to any UI Advanced\nSettings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23).\nAnd since there is no explicit setting for `allowLeadingWildcards` the\ndefault setting of `true` gets applied instead [inside the\n`grammar.peggy`\nfile](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12).\n\n### Risks that were found acceptable\n\nIn the search for any possible settings that might affect the rollout of\nchanges under this PR [we did find a\nsetting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7)\ninside `elasticsearch.yml` that causes problems with the proposed\nsolution.\n\n**TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44\"\n/>\n\nHowever we also found bigger problems, _this setting also prevents the\ninstallation of prebuilt rules_ (see below), which are a cornerstone of\nour security solution.\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1\"\n/>\n\nIn other words, the setting `search.allow_expensive_queries=true` has\nbecome _a de-facto requirement of Detection Rules_, that has not yet\nbeen documented. Hence we including it to the\n[documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)\nas part of this PR.\n\nNote also that the errors here are not limited to detection rules.\n\nWe found that the setting _also compromised or outright broke a lot of\nfunctionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved\nObject search, Tags, Server Monitoring etc). More about this is\n[documented in this internal\ndocument](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0).\nAnd in this [internal slack\nthread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469).\nSo we're assuming that most if not all of our users will have it set to\n`true`.\n\n## Checklist\n\nCheck the PR satisfies following conditions. \n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Follow the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n- [x] Changes have been socialized with the PM and rest of the team.\n- [x] All identified Risks have been properly documented and\ninvestigated. ([internal\ninvestigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0))\n- [x] New requirements added to the technical docs (PR\n[#3543](elastic/docs-content#3543), see\n[here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))","sha":"433902b9b40c9fb3f4db5346008a61efaed3df31"}},{"branch":"8.19","label":"v8.19.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.2","label":"v9.2.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Steven de Salas <[email protected]>
naemono
pushed a commit
to naemono/docs-content
that referenced
this pull request
Oct 28, 2025
…quirements.md (elastic#3543) ## Summary De-facto requirement of Security Detections, (and much of Kibana) that has not been documented properly. ``` # elasticsearch.yml search.allow_expensive_queries=false # <=== causes errors across Kibana 🤯 ``` Errors go as far back as `v8.18` (but likely much further), and not only affect Security app, but also many `Server Management` features such as Fleet, Saved Objects, Tags, etc. For more info: elastic/kibana#237496 -> Section **`Risks that were found acceptable`** We're documenting it [under the correct location](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) to make sure that users know about it. There is an [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469) for discussion. Note also that this is documented as a [requirement of the 'alerting' plugin](https://www.elastic.co/docs/explore-analyze/alerts-cases/alerts/alerting-setup#alerting-prerequisites) that we're using under the hood. <img width="800" alt="image" src="https://github.com/user-attachments/assets/512b57ed-d8e3-454b-a132-1776826d4a86" /> --------- Co-authored-by: Georgii Gorbachev <[email protected]> Co-authored-by: Nastasha Solomon <[email protected]>
sdesalas
added a commit
to elastic/kibana
that referenced
this pull request
Oct 28, 2025
…rching installed rules. (#237496) (#240969) # Backport This will backport the following commits from `main` to `9.1`: - [[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)](#237496) <!--- Backport version: 10.1.0 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Steven de Salas","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-10-28T08:25:18Z","message":"[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)\n\n**Fixes: #237278**\n**Fixes: #97094**\n**Fixes: #194066**\n\n## Summary\n\nWhen a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes\nto find all rules matching a partial string on the rule name (like `win`\nto get all rules about `Windows`) they receive 0 matches. This is in\ncontrast to the behavior they experience when searching available\nprebuilt \"Elastic Rules\", where partial name searches are available.\n\nThis PR fixes this UX inconsistency by modifying the KQL output\ngenerated by the search query.\n\nWhereas previously a search for `\"win\"` would have generated the\nfollowing KQL:\n\n```sql\n(alert.attributes.name: \"win\" OR\n alert.attributes.params.index: \"win\" OR\n alert.attributes.params.threat.tactic.id: \"win\" OR\n ...\n```\n\nIt now treats the `alert.attributes.name` differently, allowing it to\nmatch on partial terms (ie `*win*` instead of `\"win\"`) and using\n`.keyword` index for better special character support.\n\n```sql\n(alert.attributes.name.keyword: *win* OR # <-- here\n alert.attributes.params.index: \"win\" OR \n alert.attributes.params.threat.tactic.id: \"win\" OR \n ...\n```\n\n### 🕵️ BUT! .. We only do this for single term searches!\n\nPlease note that this approach only applies to single term searches. For\nmultiple term searches we maintain the \"old\" way of searching with\nquotations `\"windows 10 patch\"` instead of `*windows 10 patch*` or even\n`*windows* *10* *patch*`.\n\nThe reasoning here is that since search results are NOT sorted by score,\nwe want to avoid returning too many matches (wildcard searches match on\n_any_ combination of the terms, ie those with `windows` or `10` or\n`patch`) which would confuse the user just as they are trying to narrow\ndown their search results!! (ie 🤔 why am I getting `Linux patch` rules\nwhen I'm asking for `windows 10 patch`??).\n\n## How to test:\n\nTo test this PR please checkout the relevant branch and run an instance\nof kibana/elasticsearch locally.\n\n1. `Security App` > `Rules` > `Detection Rules (SIEM)` \n2. A table of installed rules should appear. Remove all installed\nElastic rules.\n3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules`\n> `Bulk actions` > `Delete` > `Delete`\n4. Go to `Add Elastic Rules`\n5. `Search rules by name` > `win` > `Enter`\n6. You should see 5 pages of results (~84 rules)\n7. `Install All`\n8. All Elastic rules have been installed > `Go back to installed Elastic\nrules`\n9. Search by rule name > `win` > `Enter`\n10. You should see 5 pages of results (~84 rules - same as in Step 6.)\n\nPlease feel free to try some additional search queries. \n\n<details>\n<summary>\n\n### 👉 Click for additional testing ideas\n\n</summary>\n\nHere are some single term searches:\n - `goog`\n - `proc`\n - `sql` (should include `Postgresql` and `MSSQL`)\n - `shell` (should include `Powershell`)\n - `inject` (should include `injection`)\n - `git` (should include `GitHub`)\n - `.exe`\n - `pub/sub` (exact matches only)\n - `user-agent` (exact matches only)\n - `/bin`\n - `CVE-2025` (should include partial matches)\n - `-` (should return matches with dash in the name like `user-agent`)\n- `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`)\n - `|` (should return no matches - no elastic rules with this)\n\nAnd behavior for multiple term searches:\n- `AWS` (check the count), then `AWS IAM` (there should be less results\nfor `AWS IAM`)\n- `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub\ntop` partial match has no matches!)\n- `root` then `root cert` then `root certificate` (the second has no\nresults, `root certificate` should only return exact matches)\n- `proc`, then `process`, then `process injection`, then `potential\nprocess injection` (each should return less results)\n\n\n</details>\n\n## Screenshots\n\n\n\n\n\n\n## Special character support\n\nNote that by switching to wildcard searches (ie `*win*`) on the\n`.keyword` index and fully escaping special characters in KQL we'll\n**_ALSO_** be allowing special character searches on single search\nterms.\n\nFor example, searching by `user-agent` will return results that only\nmatch `User-Agent` but not `user` or `agent` individually.\n\nSome other useful example of these approach are searches for: `Pub/Sub`,\n`CVE-2025`, `/bin`, `_`, `.exe`.\n\nThis support, in addition for escaping the backslash `\\` character will\nallow us to close the next TWO ISSUES in this epic 🥳 wohoo!!\n\n👉 #97094 (special chars in rule name) and,\n👉 #194066 (special chars in tags)\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c\"\n/>\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7\"\n/>\n\n\n\n### Testing special character support.\n\nIn addition to the steps above (installing all elastic rules), we can:\n\n1. Create a rule with special characters: ie `Rule with special chars\n(&, *, #, $, ?, >, @, \\, /, \", ‘, {, [, ;)`\n2. Try some single term matches:\n - `@`\n - `&` (additional elastic rules with this term should appear).\n - `*` \n - `\"`\n - `:`\n - `>`\n - `{`\n - `\\` (backslash, this used to break the search under #97094)\n\n## Risks\n\nThese are some of the risks that could be identified by using this\napproach.\n\n### 1. Dependencies that reuse query logic\n\nNote that the `searchTerm -> KQL` conversion for searching for rules is\nused a few places.\n\n- Installed Rules\n- Rule Monitoring\n- Bulk actions\n\n**Note**: All these paths are tested manually, and form part of the\nautomated tests.\n\n### 2. Performance (ie `allowLeadingWildcards`)\n\nWe're using wildcard searches before and after the search term (ie\n`*win*`) in order to replicate the behavior across both 'prebuilt' and\n'installed' rules tables. The wildcard AFTER the term (`win*` =>\nmatching terms like `Windows`) is no problem but the one BEFORE (`*win`\n=> matching terms like `Darwin`) could create an issue.\n\nOur [KQL\ndocumentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards)\nwarns that Kibana UI Advanced Settings have\n`query:allowLeadingWildcards` turned off by default. This is for\nperformance reasons as a leading wildcard can have a large impact when\nsearching indexes that have millions of terms associated with them.\n\n<img width=\"2784\" height=\"2120\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde\"\n/>\n\n> By default, leading wildcards are not allowed for performance reasons.\nYou can modify this with the\n[query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards)\nadvanced setting.\n\nPlease note that the [Query DSL docs also\nwarn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)\nabout avoiding this approach.\n\nThere is also more info and additional warnings under [Lucene API\ndocs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html).\n\nThe risk here is two pronged:\n\n1. Users with a lot (millions?) of detection rules will likely have a\ndegraded search experience as queries will take longer to execute.\n\n2. Leading wildcards appear to be working by default (in contrast to\nwhat is stated in the documentation). But the mere _existence_ of\nvarious settings to avoid them\n([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards),\n[`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard),\n[`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard))\nis a risk, because some users may have a special setup we've not been\nable to anticipate in our testing, leading to potential issues like this\none: https://github.com/elastic/kibana/issues/57828\n\n### Mitigating factors:\n\n1. Please note that in the case of security detection rules, the\nprebuilt rules package is _only ~1500 rules_! Users do not manage\nmillions of rules, they generally manage low thousands or even hundreds.\nWe've tested 100K rules successfully and there was [_no perceivable\ndifference in terms of search\nperformance_](https://github.com/elastic/kibana/pull/237496#issuecomment-3389956730).\nAnd this seems to be a realistic test when checking actual usage stats\n(our 10 largest users in Sep'25 had between 60-160K installed). Also,\ndue to current limits on pagination and bulk action logic we would\nlikely hit different kinds of problems here _before_ search performance\nbecomes an issue.\n\n2. We've tested all the documented ways to disable leading wildcards,\nincluding disabling them manually (under `Server Management > Advanced\nSettings`) and explicitly in the `kibana.yml`. None of them seemed to\naffect searches carried out on Saved Objects. This is because our\nsolution uses the\n[`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting)\nplugin under the hood which [converts the filter to KQL without\nreferring to any UI Advanced\nSettings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23).\nAnd since there is no explicit setting for `allowLeadingWildcards` the\ndefault setting of `true` gets applied instead [inside the\n`grammar.peggy`\nfile](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12).\n\n### Risks that were found acceptable\n\nIn the search for any possible settings that might affect the rollout of\nchanges under this PR [we did find a\nsetting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7)\ninside `elasticsearch.yml` that causes problems with the proposed\nsolution.\n\n**TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44\"\n/>\n\nHowever we also found bigger problems, _this setting also prevents the\ninstallation of prebuilt rules_ (see below), which are a cornerstone of\nour security solution.\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1\"\n/>\n\nIn other words, the setting `search.allow_expensive_queries=true` has\nbecome _a de-facto requirement of Detection Rules_, that has not yet\nbeen documented. Hence we including it to the\n[documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)\nas part of this PR.\n\nNote also that the errors here are not limited to detection rules.\n\nWe found that the setting _also compromised or outright broke a lot of\nfunctionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved\nObject search, Tags, Server Monitoring etc). More about this is\n[documented in this internal\ndocument](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0).\nAnd in this [internal slack\nthread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469).\nSo we're assuming that most if not all of our users will have it set to\n`true`.\n\n## Checklist\n\nCheck the PR satisfies following conditions. \n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Follow the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n- [x] Changes have been socialized with the PM and rest of the team.\n- [x] All identified Risks have been properly documented and\ninvestigated. ([internal\ninvestigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0))\n- [x] New requirements added to the technical docs (PR\n[#3543](elastic/docs-content#3543), see\n[here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))","sha":"433902b9b40c9fb3f4db5346008a61efaed3df31","branchLabelMapping":{"^v9.3.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:fix","Team:Detections and Resp","Team: SecuritySolution","Feature:Rule Management","Team:Detection Rule Management","ci:cloud-deploy","ci:project-deploy-security","backport:version","v9.3.0","v8.19.7","v9.1.7","v9.2.1"],"title":"[Security Solution] Allow partial matches on rule name when searching installed rules.","number":237496,"url":"https://github.com/elastic/kibana/pull/237496","mergeCommit":{"message":"[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)\n\n**Fixes: #237278**\n**Fixes: #97094**\n**Fixes: #194066**\n\n## Summary\n\nWhen a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes\nto find all rules matching a partial string on the rule name (like `win`\nto get all rules about `Windows`) they receive 0 matches. This is in\ncontrast to the behavior they experience when searching available\nprebuilt \"Elastic Rules\", where partial name searches are available.\n\nThis PR fixes this UX inconsistency by modifying the KQL output\ngenerated by the search query.\n\nWhereas previously a search for `\"win\"` would have generated the\nfollowing KQL:\n\n```sql\n(alert.attributes.name: \"win\" OR\n alert.attributes.params.index: \"win\" OR\n alert.attributes.params.threat.tactic.id: \"win\" OR\n ...\n```\n\nIt now treats the `alert.attributes.name` differently, allowing it to\nmatch on partial terms (ie `*win*` instead of `\"win\"`) and using\n`.keyword` index for better special character support.\n\n```sql\n(alert.attributes.name.keyword: *win* OR # <-- here\n alert.attributes.params.index: \"win\" OR \n alert.attributes.params.threat.tactic.id: \"win\" OR \n ...\n```\n\n### 🕵️ BUT! .. We only do this for single term searches!\n\nPlease note that this approach only applies to single term searches. For\nmultiple term searches we maintain the \"old\" way of searching with\nquotations `\"windows 10 patch\"` instead of `*windows 10 patch*` or even\n`*windows* *10* *patch*`.\n\nThe reasoning here is that since search results are NOT sorted by score,\nwe want to avoid returning too many matches (wildcard searches match on\n_any_ combination of the terms, ie those with `windows` or `10` or\n`patch`) which would confuse the user just as they are trying to narrow\ndown their search results!! (ie 🤔 why am I getting `Linux patch` rules\nwhen I'm asking for `windows 10 patch`??).\n\n## How to test:\n\nTo test this PR please checkout the relevant branch and run an instance\nof kibana/elasticsearch locally.\n\n1. `Security App` > `Rules` > `Detection Rules (SIEM)` \n2. A table of installed rules should appear. Remove all installed\nElastic rules.\n3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules`\n> `Bulk actions` > `Delete` > `Delete`\n4. Go to `Add Elastic Rules`\n5. `Search rules by name` > `win` > `Enter`\n6. You should see 5 pages of results (~84 rules)\n7. `Install All`\n8. All Elastic rules have been installed > `Go back to installed Elastic\nrules`\n9. Search by rule name > `win` > `Enter`\n10. You should see 5 pages of results (~84 rules - same as in Step 6.)\n\nPlease feel free to try some additional search queries. \n\n<details>\n<summary>\n\n### 👉 Click for additional testing ideas\n\n</summary>\n\nHere are some single term searches:\n - `goog`\n - `proc`\n - `sql` (should include `Postgresql` and `MSSQL`)\n - `shell` (should include `Powershell`)\n - `inject` (should include `injection`)\n - `git` (should include `GitHub`)\n - `.exe`\n - `pub/sub` (exact matches only)\n - `user-agent` (exact matches only)\n - `/bin`\n - `CVE-2025` (should include partial matches)\n - `-` (should return matches with dash in the name like `user-agent`)\n- `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`)\n - `|` (should return no matches - no elastic rules with this)\n\nAnd behavior for multiple term searches:\n- `AWS` (check the count), then `AWS IAM` (there should be less results\nfor `AWS IAM`)\n- `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub\ntop` partial match has no matches!)\n- `root` then `root cert` then `root certificate` (the second has no\nresults, `root certificate` should only return exact matches)\n- `proc`, then `process`, then `process injection`, then `potential\nprocess injection` (each should return less results)\n\n\n</details>\n\n## Screenshots\n\n\n\n\n\n\n## Special character support\n\nNote that by switching to wildcard searches (ie `*win*`) on the\n`.keyword` index and fully escaping special characters in KQL we'll\n**_ALSO_** be allowing special character searches on single search\nterms.\n\nFor example, searching by `user-agent` will return results that only\nmatch `User-Agent` but not `user` or `agent` individually.\n\nSome other useful example of these approach are searches for: `Pub/Sub`,\n`CVE-2025`, `/bin`, `_`, `.exe`.\n\nThis support, in addition for escaping the backslash `\\` character will\nallow us to close the next TWO ISSUES in this epic 🥳 wohoo!!\n\n👉 #97094 (special chars in rule name) and,\n👉 #194066 (special chars in tags)\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c\"\n/>\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7\"\n/>\n\n\n\n### Testing special character support.\n\nIn addition to the steps above (installing all elastic rules), we can:\n\n1. Create a rule with special characters: ie `Rule with special chars\n(&, *, #, $, ?, >, @, \\, /, \", ‘, {, [, ;)`\n2. Try some single term matches:\n - `@`\n - `&` (additional elastic rules with this term should appear).\n - `*` \n - `\"`\n - `:`\n - `>`\n - `{`\n - `\\` (backslash, this used to break the search under #97094)\n\n## Risks\n\nThese are some of the risks that could be identified by using this\napproach.\n\n### 1. Dependencies that reuse query logic\n\nNote that the `searchTerm -> KQL` conversion for searching for rules is\nused a few places.\n\n- Installed Rules\n- Rule Monitoring\n- Bulk actions\n\n**Note**: All these paths are tested manually, and form part of the\nautomated tests.\n\n### 2. Performance (ie `allowLeadingWildcards`)\n\nWe're using wildcard searches before and after the search term (ie\n`*win*`) in order to replicate the behavior across both 'prebuilt' and\n'installed' rules tables. The wildcard AFTER the term (`win*` =>\nmatching terms like `Windows`) is no problem but the one BEFORE (`*win`\n=> matching terms like `Darwin`) could create an issue.\n\nOur [KQL\ndocumentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards)\nwarns that Kibana UI Advanced Settings have\n`query:allowLeadingWildcards` turned off by default. This is for\nperformance reasons as a leading wildcard can have a large impact when\nsearching indexes that have millions of terms associated with them.\n\n<img width=\"2784\" height=\"2120\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde\"\n/>\n\n> By default, leading wildcards are not allowed for performance reasons.\nYou can modify this with the\n[query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards)\nadvanced setting.\n\nPlease note that the [Query DSL docs also\nwarn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)\nabout avoiding this approach.\n\nThere is also more info and additional warnings under [Lucene API\ndocs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html).\n\nThe risk here is two pronged:\n\n1. Users with a lot (millions?) of detection rules will likely have a\ndegraded search experience as queries will take longer to execute.\n\n2. Leading wildcards appear to be working by default (in contrast to\nwhat is stated in the documentation). But the mere _existence_ of\nvarious settings to avoid them\n([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards),\n[`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard),\n[`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard))\nis a risk, because some users may have a special setup we've not been\nable to anticipate in our testing, leading to potential issues like this\none: https://github.com/elastic/kibana/issues/57828\n\n### Mitigating factors:\n\n1. Please note that in the case of security detection rules, the\nprebuilt rules package is _only ~1500 rules_! Users do not manage\nmillions of rules, they generally manage low thousands or even hundreds.\nWe've tested 100K rules successfully and there was [_no perceivable\ndifference in terms of search\nperformance_](https://github.com/elastic/kibana/pull/237496#issuecomment-3389956730).\nAnd this seems to be a realistic test when checking actual usage stats\n(our 10 largest users in Sep'25 had between 60-160K installed). Also,\ndue to current limits on pagination and bulk action logic we would\nlikely hit different kinds of problems here _before_ search performance\nbecomes an issue.\n\n2. We've tested all the documented ways to disable leading wildcards,\nincluding disabling them manually (under `Server Management > Advanced\nSettings`) and explicitly in the `kibana.yml`. None of them seemed to\naffect searches carried out on Saved Objects. This is because our\nsolution uses the\n[`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting)\nplugin under the hood which [converts the filter to KQL without\nreferring to any UI Advanced\nSettings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23).\nAnd since there is no explicit setting for `allowLeadingWildcards` the\ndefault setting of `true` gets applied instead [inside the\n`grammar.peggy`\nfile](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12).\n\n### Risks that were found acceptable\n\nIn the search for any possible settings that might affect the rollout of\nchanges under this PR [we did find a\nsetting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7)\ninside `elasticsearch.yml` that causes problems with the proposed\nsolution.\n\n**TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44\"\n/>\n\nHowever we also found bigger problems, _this setting also prevents the\ninstallation of prebuilt rules_ (see below), which are a cornerstone of\nour security solution.\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1\"\n/>\n\nIn other words, the setting `search.allow_expensive_queries=true` has\nbecome _a de-facto requirement of Detection Rules_, that has not yet\nbeen documented. Hence we including it to the\n[documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)\nas part of this PR.\n\nNote also that the errors here are not limited to detection rules.\n\nWe found that the setting _also compromised or outright broke a lot of\nfunctionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved\nObject search, Tags, Server Monitoring etc). More about this is\n[documented in this internal\ndocument](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0).\nAnd in this [internal slack\nthread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469).\nSo we're assuming that most if not all of our users will have it set to\n`true`.\n\n## Checklist\n\nCheck the PR satisfies following conditions. \n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Follow the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n- [x] Changes have been socialized with the PM and rest of the team.\n- [x] All identified Risks have been properly documented and\ninvestigated. ([internal\ninvestigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0))\n- [x] New requirements added to the technical docs (PR\n[#3543](elastic/docs-content#3543), see\n[here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))","sha":"433902b9b40c9fb3f4db5346008a61efaed3df31"}},"sourceBranch":"main","suggestedTargetBranches":["9.1"],"targetPullRequestStates":[{"branch":"main","label":"v9.3.0","branchLabelMappingKey":"^v9.3.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/237496","number":237496,"mergeCommit":{"message":"[Security Solution] Allow partial matches on rule name when searching installed rules. (#237496)\n\n**Fixes: #237278**\n**Fixes: #97094**\n**Fixes: #194066**\n\n## Summary\n\nWhen a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes\nto find all rules matching a partial string on the rule name (like `win`\nto get all rules about `Windows`) they receive 0 matches. This is in\ncontrast to the behavior they experience when searching available\nprebuilt \"Elastic Rules\", where partial name searches are available.\n\nThis PR fixes this UX inconsistency by modifying the KQL output\ngenerated by the search query.\n\nWhereas previously a search for `\"win\"` would have generated the\nfollowing KQL:\n\n```sql\n(alert.attributes.name: \"win\" OR\n alert.attributes.params.index: \"win\" OR\n alert.attributes.params.threat.tactic.id: \"win\" OR\n ...\n```\n\nIt now treats the `alert.attributes.name` differently, allowing it to\nmatch on partial terms (ie `*win*` instead of `\"win\"`) and using\n`.keyword` index for better special character support.\n\n```sql\n(alert.attributes.name.keyword: *win* OR # <-- here\n alert.attributes.params.index: \"win\" OR \n alert.attributes.params.threat.tactic.id: \"win\" OR \n ...\n```\n\n### 🕵️ BUT! .. We only do this for single term searches!\n\nPlease note that this approach only applies to single term searches. For\nmultiple term searches we maintain the \"old\" way of searching with\nquotations `\"windows 10 patch\"` instead of `*windows 10 patch*` or even\n`*windows* *10* *patch*`.\n\nThe reasoning here is that since search results are NOT sorted by score,\nwe want to avoid returning too many matches (wildcard searches match on\n_any_ combination of the terms, ie those with `windows` or `10` or\n`patch`) which would confuse the user just as they are trying to narrow\ndown their search results!! (ie 🤔 why am I getting `Linux patch` rules\nwhen I'm asking for `windows 10 patch`??).\n\n## How to test:\n\nTo test this PR please checkout the relevant branch and run an instance\nof kibana/elasticsearch locally.\n\n1. `Security App` > `Rules` > `Detection Rules (SIEM)` \n2. A table of installed rules should appear. Remove all installed\nElastic rules.\n3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules`\n> `Bulk actions` > `Delete` > `Delete`\n4. Go to `Add Elastic Rules`\n5. `Search rules by name` > `win` > `Enter`\n6. You should see 5 pages of results (~84 rules)\n7. `Install All`\n8. All Elastic rules have been installed > `Go back to installed Elastic\nrules`\n9. Search by rule name > `win` > `Enter`\n10. You should see 5 pages of results (~84 rules - same as in Step 6.)\n\nPlease feel free to try some additional search queries. \n\n<details>\n<summary>\n\n### 👉 Click for additional testing ideas\n\n</summary>\n\nHere are some single term searches:\n - `goog`\n - `proc`\n - `sql` (should include `Postgresql` and `MSSQL`)\n - `shell` (should include `Powershell`)\n - `inject` (should include `injection`)\n - `git` (should include `GitHub`)\n - `.exe`\n - `pub/sub` (exact matches only)\n - `user-agent` (exact matches only)\n - `/bin`\n - `CVE-2025` (should include partial matches)\n - `-` (should return matches with dash in the name like `user-agent`)\n- `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`)\n - `|` (should return no matches - no elastic rules with this)\n\nAnd behavior for multiple term searches:\n- `AWS` (check the count), then `AWS IAM` (there should be less results\nfor `AWS IAM`)\n- `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub\ntop` partial match has no matches!)\n- `root` then `root cert` then `root certificate` (the second has no\nresults, `root certificate` should only return exact matches)\n- `proc`, then `process`, then `process injection`, then `potential\nprocess injection` (each should return less results)\n\n\n</details>\n\n## Screenshots\n\n\n\n\n\n\n## Special character support\n\nNote that by switching to wildcard searches (ie `*win*`) on the\n`.keyword` index and fully escaping special characters in KQL we'll\n**_ALSO_** be allowing special character searches on single search\nterms.\n\nFor example, searching by `user-agent` will return results that only\nmatch `User-Agent` but not `user` or `agent` individually.\n\nSome other useful example of these approach are searches for: `Pub/Sub`,\n`CVE-2025`, `/bin`, `_`, `.exe`.\n\nThis support, in addition for escaping the backslash `\\` character will\nallow us to close the next TWO ISSUES in this epic 🥳 wohoo!!\n\n👉 #97094 (special chars in rule name) and,\n👉 #194066 (special chars in tags)\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c\"\n/>\n\n<img width=\"800\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7\"\n/>\n\n\n\n### Testing special character support.\n\nIn addition to the steps above (installing all elastic rules), we can:\n\n1. Create a rule with special characters: ie `Rule with special chars\n(&, *, #, $, ?, >, @, \\, /, \", ‘, {, [, ;)`\n2. Try some single term matches:\n - `@`\n - `&` (additional elastic rules with this term should appear).\n - `*` \n - `\"`\n - `:`\n - `>`\n - `{`\n - `\\` (backslash, this used to break the search under #97094)\n\n## Risks\n\nThese are some of the risks that could be identified by using this\napproach.\n\n### 1. Dependencies that reuse query logic\n\nNote that the `searchTerm -> KQL` conversion for searching for rules is\nused a few places.\n\n- Installed Rules\n- Rule Monitoring\n- Bulk actions\n\n**Note**: All these paths are tested manually, and form part of the\nautomated tests.\n\n### 2. Performance (ie `allowLeadingWildcards`)\n\nWe're using wildcard searches before and after the search term (ie\n`*win*`) in order to replicate the behavior across both 'prebuilt' and\n'installed' rules tables. The wildcard AFTER the term (`win*` =>\nmatching terms like `Windows`) is no problem but the one BEFORE (`*win`\n=> matching terms like `Darwin`) could create an issue.\n\nOur [KQL\ndocumentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards)\nwarns that Kibana UI Advanced Settings have\n`query:allowLeadingWildcards` turned off by default. This is for\nperformance reasons as a leading wildcard can have a large impact when\nsearching indexes that have millions of terms associated with them.\n\n<img width=\"2784\" height=\"2120\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde\"\n/>\n\n> By default, leading wildcards are not allowed for performance reasons.\nYou can modify this with the\n[query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards)\nadvanced setting.\n\nPlease note that the [Query DSL docs also\nwarn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)\nabout avoiding this approach.\n\nThere is also more info and additional warnings under [Lucene API\ndocs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html).\n\nThe risk here is two pronged:\n\n1. Users with a lot (millions?) of detection rules will likely have a\ndegraded search experience as queries will take longer to execute.\n\n2. Leading wildcards appear to be working by default (in contrast to\nwhat is stated in the documentation). But the mere _existence_ of\nvarious settings to avoid them\n([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards),\n[`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard),\n[`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard))\nis a risk, because some users may have a special setup we've not been\nable to anticipate in our testing, leading to potential issues like this\none: https://github.com/elastic/kibana/issues/57828\n\n### Mitigating factors:\n\n1. Please note that in the case of security detection rules, the\nprebuilt rules package is _only ~1500 rules_! Users do not manage\nmillions of rules, they generally manage low thousands or even hundreds.\nWe've tested 100K rules successfully and there was [_no perceivable\ndifference in terms of search\nperformance_](https://github.com/elastic/kibana/pull/237496#issuecomment-3389956730).\nAnd this seems to be a realistic test when checking actual usage stats\n(our 10 largest users in Sep'25 had between 60-160K installed). Also,\ndue to current limits on pagination and bulk action logic we would\nlikely hit different kinds of problems here _before_ search performance\nbecomes an issue.\n\n2. We've tested all the documented ways to disable leading wildcards,\nincluding disabling them manually (under `Server Management > Advanced\nSettings`) and explicitly in the `kibana.yml`. None of them seemed to\naffect searches carried out on Saved Objects. This is because our\nsolution uses the\n[`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting)\nplugin under the hood which [converts the filter to KQL without\nreferring to any UI Advanced\nSettings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23).\nAnd since there is no explicit setting for `allowLeadingWildcards` the\ndefault setting of `true` gets applied instead [inside the\n`grammar.peggy`\nfile](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12).\n\n### Risks that were found acceptable\n\nIn the search for any possible settings that might affect the rollout of\nchanges under this PR [we did find a\nsetting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7)\ninside `elasticsearch.yml` that causes problems with the proposed\nsolution.\n\n**TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44\"\n/>\n\nHowever we also found bigger problems, _this setting also prevents the\ninstallation of prebuilt rules_ (see below), which are a cornerstone of\nour security solution.\n\n<img width=\"600\" alt=\"image\"\nsrc=\"https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1\"\n/>\n\nIn other words, the setting `search.allow_expensive_queries=true` has\nbecome _a de-facto requirement of Detection Rules_, that has not yet\nbeen documented. Hence we including it to the\n[documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)\nas part of this PR.\n\nNote also that the errors here are not limited to detection rules.\n\nWe found that the setting _also compromised or outright broke a lot of\nfunctionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved\nObject search, Tags, Server Monitoring etc). More about this is\n[documented in this internal\ndocument](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0).\nAnd in this [internal slack\nthread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469).\nSo we're assuming that most if not all of our users will have it set to\n`true`.\n\n## Checklist\n\nCheck the PR satisfies following conditions. \n\n- [x] [Unit or functional\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\nwere updated or added to match the most common scenarios\n- [x] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [x] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [x] Follow the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.\n- [x] Changes have been socialized with the PM and rest of the team.\n- [x] All identified Risks have been properly documented and\ninvestigated. ([internal\ninvestigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0))\n- [x] New requirements added to the technical docs (PR\n[#3543](elastic/docs-content#3543), see\n[here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))","sha":"433902b9b40c9fb3f4db5346008a61efaed3df31"}},{"branch":"8.19","label":"v8.19.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/240940","number":240940,"state":"OPEN"},{"branch":"9.1","label":"v9.1.7","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.2","label":"v9.2.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/240942","number":240942,"state":"MERGED","mergeCommit":{"sha":"77646b96df933fa7f97cf140578c6de3ef852dc6","message":"[9.2] [Security Solution] Allow partial matches on rule name when searching installed rules. (#237496) (#240942)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.2`:\n- [[Security Solution] Allow partial matches on rule name when searching\ninstalled rules.\n(#237496)](https://github.com/elastic/kibana/pull/237496)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\nCo-authored-by: Steven de Salas <[email protected]>"}}]}] BACKPORT-->
jeramysoucy
added a commit
to jeramysoucy/kibana
that referenced
this pull request
Oct 29, 2025
commit 94b265c7c6efd62afdeed4fc371c1b64661c45fa Author: “jeramysoucy” <[email protected]> Date: Wed Oct 29 11:32:12 2025 +0100 Adds missing interface comments commit 86f149f7a7266917264f63940a1a600d20dfea03 Merge: 55da18aa9d19 b0f4c5294b36 Author: Jeramy Soucy <[email protected]> Date: Wed Oct 29 08:07:17 2025 +0100 Merge branch 'security/read-only-dashboards' into read-only-dashboards-granular-bulk-ops commit b0f4c5294b36a7bd1df2076021316c0329024ed3 Merge: e70f103173e3 73ec8eb4d7bc Author: Elastic Machine <[email protected]> Date: Tue Oct 28 14:32:33 2025 +0100 Merge branch 'main' into security/read-only-dashboards commit 73ec8eb4d7bc407cf984f7b14264dfc831a3b9b7 Author: Gerard Soldevila <[email protected]> Date: Tue Oct 28 14:16:24 2025 +0100 Use KIBANA_SOLUTIONS const (#239563) ## Summary We cannot directly import `@kbn/projects-solutions-groups` from `@kbn/repo-packages` (as it is a plain Javascript component and we're not compiling any typescript at this stage), but we can enforce types through annotations, giving us a certain degree of coupling between the definitions. commit 98df3956ba157b508add2cb6263f31a05576ab09 Author: Stratou <[email protected]> Date: Tue Oct 28 13:32:13 2025 +0100 [ES|QL] Create dataviews without asking for fields wherever is not required (#240197) ## Summary Fetching fields can be expensive. I have upgraded the esql adhoc dataview helper to have a skip fields flag. I have also looked into the codebase and I apply it wherever is not required. This will give a performance boost to the consumers especially for sources with many fields. Note: I have tested the applications and I don't see any regressions but please test carefully too! commit 71ad14a43c6ad8a00993d217d74c56f607851d42 Author: Thom Heymann <[email protected]> Date: Tue Oct 28 12:31:01 2025 +0000 [Streams] Wrap "create partition" buttons on small screens (#240957) Tiny PR ensuring create partition buttons get wrapped on small screens ## Before <img width="1406" height="1103" alt="Screenshot 2025-10-28 at 10 27 31" src="https://github.com/user-attachments/assets/0a3e9a99-f396-42fc-98a2-5e7ffdb116e3" /> ## After <img width="1409" height="1105" alt="Screenshot 2025-10-28 at 10 27 50" src="https://github.com/user-attachments/assets/72cf281e-e27f-44b9-88e3-b3e5f436ab52" /> commit 3a830e0265c628bd623fb348a8025bee301406f9 Author: Nicolas Chaulet <[email protected]> Date: Tue Oct 28 08:29:14 2025 -0400 [Fleet] Concurrently create elastic_agent alerting rules (#240474) commit 330f8c63564877b7eff7bf15e334bf326c468e20 Author: Jean-Louis Leysens <[email protected]> Date: Tue Oct 28 13:04:22 2025 +0100 [OAS] Update OAS spec definition to require `description` in parameters too (#238604) ## Changes to validation * Make our validation script check for the presence of: * `description` in parameter and property definitions * `tags`, `operationId`, `summary` and `description` for each operation * `examples` in request and response definitions ## Changes to code * Moved the JSONSchema definition out of a node_modules dependency so that we have full control over the schema definition * Captured full output before and after moving code out of the dependency and ensured output was exactly the same using `diff` CLI (also brought 3rd party test code where applicable) * Reset the error baseline snapshots: there are quite a few more errors due to this additional requirement on our OAS * Tried to improve self-servicability through better logging: guiding devs to the JSONSchema we use to validate our OAS * Added 2 dev time deps, but overall third party code shrank bc we now use a subset of `@seriousme/openapi-schema-validator`, see the original security concerns addressed in https://github.com/elastic/kibana/pull/228402 --------- Co-authored-by: kibanamachine <[email protected]> commit b0345ee16302774c7f5fa8ce1323b5a5f00505d9 Author: Bharat Pasupula <[email protected]> Date: Tue Oct 28 12:09:23 2025 +0100 [Automatic_Import_V2] Update Integrations API model (#240934) ## Summary This PR updates the API model for the integrations flow. Since we will move to having one screen for creating integrations , we do not need so many different APIs. Possible flows: #### Create New Integration - User clicks `New Integration` to get a form with integration details like Title , Description , Logo and an empty table to add data streams - User clicks `Add data stream` and gets a flyout to add data stream form with Title , Description , Input types , raw samples [ either upload or from an index]. - User clicks on `Analyze Logs` and this is the first request to backend. Here we create Integration SO , Data stream SO and start the BG task for running data stream creation. So `PUT /automatic_import_v2/integrations` should do the job here. To keep this Idempotent, PUT instead of POST. #### List All Integrations `GET /automatic_import_v2/integrations` should all auto import integrations to show up. #### Pick an integration to edit `GET /automatic_import_v2/integrations/{integrations_id}` should provide specific integration info. #### Delete an integration `DELETE /automatic_import_v2/integrations/{integrations_id}` should delete an integration. #### Edit an integration - Possibly change description / logo - Possibly add data stream(s). `PATCH /automatic_import_v2/integrations/{integrations_id}` should do this. More flows to follow: - Validate the pipeline changes. - Stop the integration flow and kill the BG task. - Fetch the status on a running task commit 76fccd2d2368b3823839df5eccafcb165822adb5 Author: Marco Antonio Ghiani <[email protected]> Date: Tue Oct 28 11:58:10 2025 +0100 [Streams] Support convert processor for stream data processing (#240023) ## 📓 Summary Closes https://github.com/elastic/streams-program/issues/498 This work supports the `convert` processor in Streamlang and adds UI support to create convert processors. To guarantee feature parity with the equivalent ES|QL transpiled query, there is additional validation to enforce the user setting the target field on the convert configuration when the processor should run behind a given condition. There is a minor issue in the recording when the processor description is shown. This is fixed with [feat(streams): update description](https://github.com/elastic/kibana/commit/f896a4336747acbafe2f59662f7f42046d8c5dab) https://github.com/user-attachments/assets/f6fd2948-0923-46db-9d4f-1f614f8de086 --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Joe Reuter <[email protected]> commit c500ff1af1e9ecf23c7c572f371d5ce903f6e405 Author: Valerio <[email protected]> Date: Tue Oct 28 11:50:58 2025 +0100 [ES|QL] Lookup join ON expressions client side validation (#240930) ## Summary Closes https://github.com/elastic/kibana/issues/236939 - JOIN validation requires async operations because it must fetch the lookup index schema to properly validate expressions in the ON clause. (context.joinSources only contains index metadata, not field schema ) - Existing validators remain synchronousv-> no problems - We update the Fork validation too - Client-side semantic validation: no mixing binary expressions with comma-separated fields Notes: There are some errors that do not have real-time validation but are present when you run the query, but they seem to me ES validation errors some examples: - JOIN with right field [agent] of type [TEXT] is not supported - JOIN ON clause only supports fields or AND of Binary Expressions at the moment - At least one condition relating the left and right side of the Lookup Join is still required --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Stratou <[email protected]> commit e70f103173e3cdfe3608c5b58a56703cd428f69b Merge: 65e673c8bcf6 f4a25208a91f Author: Sid <[email protected]> Date: Tue Oct 28 11:50:27 2025 +0100 Merge branch 'main' into security/read-only-dashboards commit f4a25208a91f69a163e78c332fc603158421c489 Author: elastic-vault-github-plugin-prod[bot] <150874479+elastic-vault-github-plugin-prod[bot]@users.noreply.github.com> Date: Tue Oct 28 10:45:28 2025 +0000 [main] Sync bundled packages with Package Storage (#240868) Automated by https://buildkite.com/elastic/package-storage-infra-kibana-discover-release-branches/builds/3636 Co-authored-by: elastic-vault-github-plugin-prod <[email protected]> Co-authored-by: Julia Bardi <[email protected]> commit b2a9a2a16943937648c032c17c503c5874402f7f Author: Alex Szabo <[email protected]> Date: Tue Oct 28 11:43:12 2025 +0100 Improve quick checks (#237905) ## Summary Multiple changes to quick-checks, to make them better. - bootstrap `.buildkite` during `yarn kbn bootstrap`, some scripts, e.g. quick-checks rely on `.buildkite` having it's `node_modules` installed. - add objects wrapping the quickcheck script names to mark if it will change files. We can add more metadata to quickchecks (e.g.: timeouts, labels, dependencies) if needed - force all checks that write files to a sequential execution to prevent mutliple parallel file edits (this increases runtime from ~8m for the quick checks to ~11m (~14m for the whole step). Fixes: https://github.com/elastic/kibana-operations/issues/310 --------- Co-authored-by: kibanamachine <[email protected]> commit c82a0241482e65d8092daeeb6c54a92d7715c946 Author: Ángeles Martínez Barrio <[email protected]> Date: Tue Oct 28 11:37:43 2025 +0100 [SharedUX][A11y] Fix maps source layer missing error text (#240818) Closes https://github.com/elastic/kibana/issues/215674 ## Summary - Added explicit error message for users to better understand why the field is in an error state: error only appears after the user interacts with the field - Added the `touchedLayerName` state initialized to false to avoid showing the error on first render - Updated snapshots since now component does not initialize as `isInvalid={true}` ### Testing **Windows + NVDA** Before: <img width="1359" height="700" alt="Screenshot 2025-10-27 at 14 10 57" src="https://github.com/user-attachments/assets/2e26cdd5-62d3-455b-9ace-8562b4a23994" /> After: <img width="1357" height="665" alt="Screenshot 2025-10-27 at 14 12 16" src="https://github.com/user-attachments/assets/8f18f0d0-e8e5-4251-b888-1838a0be1a04" /> **MacOS + VO** Before: <img width="904" height="630" alt="Screenshot 2025-10-27 at 13 37 15" src="https://github.com/user-attachments/assets/33f4a829-ed3c-47bd-ae0c-57a0b156cde9" /> After: <img width="869" height="529" alt="Screenshot 2025-10-27 at 13 36 03" src="https://github.com/user-attachments/assets/a87a6d2c-4f39-4912-9830-2cbd8136baff" /> --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Elastic Machine <[email protected]> commit 91ba49d720aec7df87bb56db9a5b4e1369ba66c9 Author: Chris <[email protected]> Date: Tue Oct 28 10:31:33 2025 +0000 [Agent Builder] Refactor Conversation to be context aware (#240311) commit a6a7847102bbe1a3b86010542770ce122faaa4d1 Author: Stratou <[email protected]> Date: Tue Oct 28 11:13:35 2025 +0100 [ES|QL] Supports multi values variables in MV_CONTAINS (#239266) ## Summary Closes https://github.com/elastic/kibana/issues/237228 Supports multivalue support when the user uses the variable in mv_contains both in Discover and Dashboards.  As the issue indicates: - Included a multi-select option only when creating a control for multivalues either static values or values from a query - This is not possible for fields (??field) - The multi-select option is OFF by default unless the user creates the variable control coming from the `MV_CONTAINS` function - Supported for both Discover and Dashboard - Based on the existing controls functionality for multi-selections <img width="550" height="228" alt="image" src="https://github.com/user-attachments/assets/58c6c17b-56a7-41c4-b6b6-d1fc00c314dd" /> <img width="542" height="683" alt="image" src="https://github.com/user-attachments/assets/c18cea19-f7b1-46e2-a001-79de3d9396c5" /> ### Checklist - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed commit d44c4de221dcd7d6b7cc1953845f16edbcd6f6cb Author: Maryam Saeidi <[email protected]> Date: Tue Oct 28 11:11:57 2025 +0100 Disable empty FTR configs (#240433) ## Summary Disable empty FTR configs. commit 6989832bfe32584e6b0d32f04c71981a440a541d Author: Sebastian Delle Donne <[email protected]> Date: Tue Oct 28 11:07:04 2025 +0100 [ES|QL] Index editor- Improve accessibility (#240462) Part of https://github.com/elastic/kibana/issues/235730 ## Summary Provides a more consistent keyboard accessibility experience for the index editor. * `Tab` moves the focus to the next element in the page (The Close X button). * `Arrows` allow to move between cells and headers. * `Enter` on cells trigger the edit mode. * `Enter` on headers opens the header actions (Edit or Delete). * `Enter` on the input edit mode, saves the changes and focus the cell/header. * `Esc` goes out of the edit mode without saving, keeps focus on the cell/header. ### Before * Focus was lost from the flyout after saving a column name or hitting `Esc`. * 2 enters were required to edit a column name, after the first enter, this was shown: <img width="321" height="56" alt="image" src="https://github.com/user-attachments/assets/c3056fcb-bc13-443d-ac94-893c4d05b503" />  ### After * Focus remains set to column header cell after exiting the name input, allowing to continue to do editions without using the mouse. * Editing the column name is now a header action. <img width="317" height="187" alt="image" src="https://github.com/user-attachments/assets/1f7ba575-a91d-49fd-84cf-a72297a09ba3" /> * You can still access the edition mode by clicking the name with the mouse. * For editing using the keyboard, you need to hit enter, and then select the edit action. **Why is this needed?** The datagrid wont let us handle correctly the focus if there are other actions available in the header, in our case we have `Delete columns and values`, this has a good accessibility reason as this action will become inaccessible if we open the edition mode straight away. * Hitting tab while editing the next column, now moves to the first column of the next row.  commit 47d1bd7d68b399faac7190b9952b4eb7c115c450 Author: Alexey Antonov <[email protected]> Date: Tue Oct 28 11:48:41 2025 +0200 fix: Failing test: APM Cypress.x-pack/solutions/observability/plugins/apm/ftr_e2e/cypress/e2e/dependencies/dependencies·cy·ts - Dependencies dependency overview page has no detectable a11y violations on load has no detectable a11y violations on load (#240832) Closes: https://github.com/elastic/kibana/issues/240813 ## Summary I’m not exactly sure why, but sometimes `Cypress` renders an empty table and other times it displays data. When the table contained data, we encountered some accessibility (a11y) violations, which I’ve now fixed in the relevant `AgentIcon` component. Tested locally 10 times commit 543cce60adde2da5d8aef38b9bd3a92b6cb6ed47 Author: Alex Szabo <[email protected]> Date: Tue Oct 28 10:27:56 2025 +0100 Revert "[CI] Trigger backports when `v*` labels are added afterwards (#234452)" (#240946) ## Summary This reverts commit 040e409caa72ee45013ea09e2cb6747678411456 in #234452 commit 040e409caa72ee45013ea09e2cb6747678411456 Author: Alex Szabo <[email protected]> Date: Tue Oct 28 10:22:41 2025 +0100 [CI] Trigger backports when `v*` labels are added afterwards (#234452) ## Summary backport should now trigger if `vX.Y.Z` labels are added last - introduce a concurrency group to avoid multiple runs This was already done in https://github.com/elastic/kibana/pull/225511 then reverted in #226840 This time, we're also adding a concurrency group to avoid the multiple runs (reason for the revert), and adding some wait time before the backport is started in (https://github.com/elastic/kibana-github-actions/pull/62) Also, as per @mistic's suggestion, we're calculating the do/don't in a javascript block, so that we're not struggling with inline condition language. commit 55da18aa9d199537d1478242934676d20f2c6d31 Merge: 1ae4f1f99f06 65e673c8bcf6 Author: Jeramy Soucy <[email protected]> Date: Tue Oct 28 09:46:13 2025 +0100 Merge branch 'security/read-only-dashboards' into read-only-dashboards-granular-bulk-ops commit 65e673c8bcf6c0d90001c4e138f4dac652a8f4f6 Author: kibanamachine <[email protected]> Date: Tue Oct 28 08:44:43 2025 +0000 [CI] Auto-commit changed files from 'node scripts/generate codeowners' commit 1ae4f1f99f065c274d8dccb943b726caaa8fa748 Merge: 5871c9233dfc 7d0d88bb7137 Author: Jeramy Soucy <[email protected]> Date: Tue Oct 28 09:35:16 2025 +0100 Merge branch 'security/read-only-dashboards' into read-only-dashboards-granular-bulk-ops commit 7d0d88bb713730d731189faa08983d1a88ad0b20 Merge: 4d914e01812f 8ae3b01dfa0b Author: Sid <[email protected]> Date: Tue Oct 28 09:31:16 2025 +0100 Merge branch 'main' into security/read-only-dashboards commit 8ae3b01dfa0b861afd6bcb3f68773f42611423de Author: Marco Vettorello <[email protected]> Date: Tue Oct 28 09:26:44 2025 +0100 [Vega] Update Vega packages to latest (#240831) ## Summary Update Vega packages: - vega-utils - vega-interpreter - vega-expression commit 433902b9b40c9fb3f4db5346008a61efaed3df31 Author: Steven de Salas <[email protected]> Date: Tue Oct 28 09:25:18 2025 +0100 [Security Solution] Allow partial matches on rule name when searching installed rules. (#237496) **Fixes: #237278** **Fixes: #97094** **Fixes: #194066** ## Summary When a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes to find all rules matching a partial string on the rule name (like `win` to get all rules about `Windows`) they receive 0 matches. This is in contrast to the behavior they experience when searching available prebuilt "Elastic Rules", where partial name searches are available. This PR fixes this UX inconsistency by modifying the KQL output generated by the search query. Whereas previously a search for `"win"` would have generated the following KQL: ```sql (alert.attributes.name: "win" OR alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` It now treats the `alert.attributes.name` differently, allowing it to match on partial terms (ie `*win*` instead of `"win"`) and using `.keyword` index for better special character support. ```sql (alert.attributes.name.keyword: *win* OR # <-- here alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` ### 🕵️ BUT! .. We only do this for single term searches! Please note that this approach only applies to single term searches. For multiple term searches we maintain the "old" way of searching with quotations `"windows 10 patch"` instead of `*windows 10 patch*` or even `*windows* *10* *patch*`. The reasoning here is that since search results are NOT sorted by score, we want to avoid returning too many matches (wildcard searches match on _any_ combination of the terms, ie those with `windows` or `10` or `patch`) which would confuse the user just as they are trying to narrow down their search results!! (ie 🤔 why am I getting `Linux patch` rules when I'm asking for `windows 10 patch`??). ## How to test: To test this PR please checkout the relevant branch and run an instance of kibana/elasticsearch locally. 1. `Security App` > `Rules` > `Detection Rules (SIEM)` 2. A table of installed rules should appear. Remove all installed Elastic rules. 3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules` > `Bulk actions` > `Delete` > `Delete` 4. Go to `Add Elastic Rules` 5. `Search rules by name` > `win` > `Enter` 6. You should see 5 pages of results (~84 rules) 7. `Install All` 8. All Elastic rules have been installed > `Go back to installed Elastic rules` 9. Search by rule name > `win` > `Enter` 10. You should see 5 pages of results (~84 rules - same as in Step 6.) Please feel free to try some additional search queries. <details> <summary> ### 👉 Click for additional testing ideas </summary> Here are some single term searches: - `goog` - `proc` - `sql` (should include `Postgresql` and `MSSQL`) - `shell` (should include `Powershell`) - `inject` (should include `injection`) - `git` (should include `GitHub`) - `.exe` - `pub/sub` (exact matches only) - `user-agent` (exact matches only) - `/bin` - `CVE-2025` (should include partial matches) - `-` (should return matches with dash in the name like `user-agent`) - `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`) - `|` (should return no matches - no elastic rules with this) And behavior for multiple term searches: - `AWS` (check the count), then `AWS IAM` (there should be less results for `AWS IAM`) - `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub top` partial match has no matches!) - `root` then `root cert` then `root certificate` (the second has no results, `root certificate` should only return exact matches) - `proc`, then `process`, then `process injection`, then `potential process injection` (each should return less results) </details> ## Screenshots  ## Special character support Note that by switching to wildcard searches (ie `*win*`) on the `.keyword` index and fully escaping special characters in KQL we'll **_ALSO_** be allowing special character searches on single search terms. For example, searching by `user-agent` will return results that only match `User-Agent` but not `user` or `agent` individually. Some other useful example of these approach are searches for: `Pub/Sub`, `CVE-2025`, `/bin`, `_`, `.exe`. This support, in addition for escaping the backslash `\` character will allow us to close the next TWO ISSUES in this epic 🥳 wohoo!! 👉 #97094 (special chars in rule name) and, 👉 #194066 (special chars in tags) <img width="800" alt="image" src="https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7" /> ### Testing special character support. In addition to the steps above (installing all elastic rules), we can: 1. Create a rule with special characters: ie `Rule with special chars (&, *, #, $, ?, >, @, \, /, ", ‘, {, [, ;)` 2. Try some single term matches: - `@` - `&` (additional elastic rules with this term should appear). - `*` - `"` - `:` - `>` - `{` - `\` (backslash, this used to break the search under #97094) ## Risks These are some of the risks that could be identified by using this approach. ### 1. Dependencies that reuse query logic Note that the `searchTerm -> KQL` conversion for searching for rules is used a few places. - Installed Rules - Rule Monitoring - Bulk actions **Note**: All these paths are tested manually, and form part of the automated tests. ### 2. Performance (ie `allowLeadingWildcards`) We're using wildcard searches before and after the search term (ie `*win*`) in order to replicate the behavior across both 'prebuilt' and 'installed' rules tables. The wildcard AFTER the term (`win*` => matching terms like `Windows`) is no problem but the one BEFORE (`*win` => matching terms like `Darwin`) could create an issue. Our [KQL documentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards) warns that Kibana UI Advanced Settings have `query:allowLeadingWildcards` turned off by default. This is for performance reasons as a leading wildcard can have a large impact when searching indexes that have millions of terms associated with them. <img width="2784" height="2120" alt="image" src="https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde" /> > By default, leading wildcards are not allowed for performance reasons. You can modify this with the [query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards) advanced setting. Please note that the [Query DSL docs also warn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard) about avoiding this approach. There is also more info and additional warnings under [Lucene API docs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html). The risk here is two pronged: 1. Users with a lot (millions?) of detection rules will likely have a degraded search experience as queries will take longer to execute. 2. Leading wildcards appear to be working by default (in contrast to what is stated in the documentation). But the mere _existence_ of various settings to avoid them ([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards), [`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard), [`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)) is a risk, because some users may have a special setup we've not been able to anticipate in our testing, leading to potential issues like this one: https://github.com/elastic/kibana/issues/57828 ### Mitigating factors: 1. Please note that in the case of security detection rules, the prebuilt rules package is _only ~1500 rules_! Users do not manage millions of rules, they generally manage low thousands or even hundreds. We've tested 100K rules successfully and there was [_no perceivable difference in terms of search performance_](https://github.com/elastic/kibana/pull/237496#issuecomment-3389956730). And this seems to be a realistic test when checking actual usage stats (our 10 largest users in Sep'25 had between 60-160K installed). Also, due to current limits on pagination and bulk action logic we would likely hit different kinds of problems here _before_ search performance becomes an issue. 2. We've tested all the documented ways to disable leading wildcards, including disabling them manually (under `Server Management > Advanced Settings`) and explicitly in the `kibana.yml`. None of them seemed to affect searches carried out on Saved Objects. This is because our solution uses the [`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting) plugin under the hood which [converts the filter to KQL without referring to any UI Advanced Settings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23). And since there is no explicit setting for `allowLeadingWildcards` the default setting of `true` gets applied instead [inside the `grammar.peggy` file](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12). ### Risks that were found acceptable In the search for any possible settings that might affect the rollout of changes under this PR [we did find a setting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7) inside `elasticsearch.yml` that causes problems with the proposed solution. **TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯 <img width="600" alt="image" src="https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44" /> However we also found bigger problems, _this setting also prevents the installation of prebuilt rules_ (see below), which are a cornerstone of our security solution. <img width="600" alt="image" src="https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1" /> In other words, the setting `search.allow_expensive_queries=true` has become _a de-facto requirement of Detection Rules_, that has not yet been documented. Hence we including it to the [documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) as part of this PR. Note also that the errors here are not limited to detection rules. We found that the setting _also compromised or outright broke a lot of functionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved Object search, Tags, Server Monitoring etc). More about this is [documented in this internal document](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0). And in this [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469). So we're assuming that most if not all of our users will have it set to `true`. ## Checklist Check the PR satisfies following conditions. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Follow the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. - [x] Changes have been socialized with the PM and rest of the team. - [x] All identified Risks have been properly documented and investigated. ([internal investigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0)) - [x] New requirements added to the technical docs (PR [#3543](https://github.com/elastic/docs-content/pull/3543), see [here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements)) commit d82256937c50fafdbb82c1698702b0bace4a9e06 Author: Jiawei Wu <[email protected]> Date: Tue Oct 28 01:15:17 2025 -0700 [ResponseOps] Clear flapping when rules are enabled or bulk enabled (#235024) ## Summary Resolves: https://github.com/elastic/kibana/issues/149950 Clears flapping on alerts generated by rules when the user enables or bulk enables the rule when it is disabled. Should only clear flapping on active or recovered alerts, meaning untracked alerts are untouched. ### To test: 1. Ensure flapping is enabled 2. Create a rule(s) and generate a few alerts 3. Disable the rule(s) 4. Assert the alerts have a flapping history 5. Enable the rule(s) 6. Assert the previously generated active or recovered alerts now have flapping: false and flapping history is cleared 7. Assert that this only works on lifecycle rules (not siem rules) ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Elastic Machine <[email protected]> commit acb3a279b7766634b3b0ada24953d55c1bbf8da2 Author: Carlos Crespo <[email protected]> Date: Tue Oct 28 09:08:41 2025 +0100 [Metrics][Discover] Integrate metrics experience state with discover state (#239732) closes [#236395](https://github.com/elastic/kibana/issues/236395) ## Summary This PR integrates the Metrics Experience state with Discover state, enabling the integration with Discover tabs. The internal Metrics Experience redux state was removed in favor o Discover's. **Isolated tabs state**  **Duplicate tab action**  ## How to test - Clone https://github.com/simianhacker/simian-forge/tree/main - Run ` ./forge --dataset hosts --count 25 --interval 30s` - For beats metrics, the system integration package MUST be installed (after installing it, make sure to delete all existing metrics data streams) - Set the following config to `kibana.dev.yml` ```yml feature_flags.overrides: metricsExperienceEnabled: true ``` - Navigate to Discover and Switch to ESQL mode - Create new tabs and modify their state - Duplicate tabs --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Elastic Machine <[email protected]> commit c8754cfa364607b9196ea61af83c69508aed6530 Author: Angela Chuang <[email protected]> Date: Tue Oct 28 07:51:56 2025 +0000 [Dashboards as code] Convert anomaly single metric view Embeddable types to API schemas (#239161) ## Summary Convert SingleMetricViewerEmbeddableState to API schemas --------- Co-authored-by: kibanamachine <[email protected]> commit 40d53ff1d99615edb34eec012b05924a20b8551b Author: Julia Bardi <[email protected]> Date: Tue Oct 28 08:40:27 2025 +0100 [Fleet] enable skipped test (#240759) ## Summary Closes https://github.com/elastic/kibana/issues/238960 Closes https://github.com/elastic/kibana/issues/214862 commit f9300bdb54c49ce3ca2ffc14f3c7cf373f6c48e0 Author: Vladimir Filonov <[email protected]> Date: Tue Oct 28 10:57:18 2025 +0400 [One Workflow] Remove obsolete 'enabled' property from workflow trigger schemas (#240844) ## Summary closes: https://github.com/elastic/security-team/issues/14382 <img width="991" height="483" alt="image" src="https://github.com/user-attachments/assets/36b90e23-3e03-42c3-89f8-5a7a1a393572" /> commit 62cb13017a1c687612b27ad48ccfc587941b523f Author: Philippe Oberti <[email protected]> Date: Tue Oct 28 00:32:56 2025 -0500 [Security Solution] remove riskScoringRoutesEnabled feature flag (#240539) ## Summary This PR removes the `riskScoringRoutesEnabled` feature flag. This feature flag was enabled back in 2023 . **_Feel free to close this PR if our intention is to keep the feature flag around!_** > [!IMPORTANT] > As the feature flag has been eanbled for a while,, no UI or behavior changes should be introduced by this PR! ### Checklist - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. commit aa9f5fe0b4bc4b76a6af80332ff25b68d22ca933 Author: Quynh Nguyen (Quinn) <[email protected]> Date: Mon Oct 27 23:43:37 2025 -0500 [AI Infra] Fix index names causing incompatible cluster error when product docs are installed with for multiple Inference IDs (#240506) ## Summary This PR fixes a problem where product doc search is not performed correctly when BOTH E5 and ELSER product docs are installed due to incompatible cluster issue. Steps to reproduce on 9.2: - Install product docs for both E5 and ELSER - Go to Observability AI assistant and ask 'Hey can you look up documentation to how to use agent builder' - See that the assistant doesn't return meaningful answer on the new agent builder After: <img width="1381" height="756" alt="Screenshot 2025-10-24 at 13 54 35" src="https://github.com/user-attachments/assets/5db4dce5-3005-4f04-9fe9-9d5d3b0e9d85" /> ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... --------- Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Robert Jaszczurek <[email protected]> commit 8ce6cc0b134e9ca25217c626e267d33c5b405bf6 Author: Davis McPhee <[email protected]> Date: Tue Oct 28 01:30:14 2025 -0300 [Discover] Share profile provider services across Discover instances (#240527) ## Summary This PR updates the Discover context awareness framework to share profile provider services across profiles manager instances. Previously `createProfileProviderServices` was called every time a `ProfilesManager` was instantiated, which happens for each Discover panel in a dashboard. We allow services to perform async work on init, such as hitting an API endpoint to fetch configs, so this could unnecessarily slow down each Discover panel. With these changes we only take the hit once. It also parallelizes some async calls on init to avoid waiting on them sequentially. Before: https://github.com/user-attachments/assets/c1f54c06-8ac5-4a59-99b7-7b91678adaa6 After: https://github.com/user-attachments/assets/3d08eaee-9767-41d4-ac93-3cba52fde471 ### Checklist - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: kibanamachine <[email protected]> commit 0a95dc1c42c6cc842c6469b6a9e03239fe8faf66 Author: Brad White <[email protected]> Date: Tue Oct 28 13:06:48 2025 +1000 [FIPS] Add 9.1 branch to daily pipeline (#240551) ## Summary - Adds `9.1` branch to daily FIPS pipeline since it is used for FRH environment commit 99340a1bdf0b4e093a2ffbff21aab809e84b503c Author: Philippe Oberti <[email protected]> Date: Mon Oct 27 20:33:45 2025 -0500 [Security Solution] remove riskScoringPersistence feature flag (#240538) ## Summary This PR removes the `riskScoringPersistence` feature flag. This feature flag was enabled back in 2023 . **_Feel free to close this PR if our intention is to keep the feature flag around!_** > [!IMPORTANT] > As the feature flag has been eanbled for a while,, no UI or behavior changes should be introduced by this PR! ### Checklist - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. commit f257a6503c2295a8180e4e4c90941de4c32a5455 Author: Davis McPhee <[email protected]> Date: Mon Oct 27 22:01:57 2025 -0300 [Discover] Optimize Discover plugin init by running async calls in parallel (#240536) ## Summary This PR parallelizes async calls during plugin init where possible to avoid waiting on them sequentially. In practice the improvements are probably marginal (although a couple spots let us parallelize dynamic bundle imports), but it encourages a good pattern for future additions. ### Checklist - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: kibanamachine <[email protected]> commit f9bde2f0959cc00689f427761ba71c2b2ba071ae Author: Davis McPhee <[email protected]> Date: Mon Oct 27 22:00:17 2025 -0300 [Discover] [Metrics Experience] Move metrics profile folder into `common` (#240534) ## Summary A small PR that moves the metrics data source profile to the `common` folder to align with the guidance in the [dev docs](https://github.com/elastic/kibana/blob/main/src/platform/plugins/shared/discover/public/context_awareness/README.md#registering-a-profile). Also updates the naming of a couple other profile provider folders to be consistent with the naming conventions. Lastly updates `CODEOWNERS` to make sure folder paths and ownership are correct. ### Checklist - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: kibanamachine <[email protected]> commit 9c71fcc548bad108d99479ecdb3be82682360c84 Author: Davis McPhee <[email protected]> Date: Mon Oct 27 21:58:40 2025 -0300 [Discover] Logs profile providers cleanup (#240545) ## Summary This PR simplifies the Observability logs sub profile providers by combining their definitions into an array in a single file and iterating over them to generate the providers. This will mean fewer changes and files to touch when modifying these profiles. It also extracts the resolution match mocks in test files into a common const for similar reasons. Resolves #219338. ### Checklist - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: kibanamachine <[email protected]> commit 1ceb71c6c621641062ff97778e359ac81bcad4a7 Author: Davis McPhee <[email protected]> Date: Mon Oct 27 21:57:33 2025 -0300 [Discover] Remove `SolutionId` casting (#240547) ## Summary Just a tiny PR to remove the `SolutionId` casting that was added a while back when `getActiveSolutionNavId$` was updated to be strongly typed. Also aligns our `SolutionType.Search` value to the actual value, since these changes surfaced a type mismatch there. ### Checklist - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. --------- Co-authored-by: kibanamachine <[email protected]> commit 745fda44fbfa5e40c50c149f6ca7cb5023dd1c7b Author: Kibana Machine <[email protected]> Date: Mon Oct 27 23:52:13 2025 +0100 skip failing test suite (#240900) commit 854b9cb37768d77305a246a257cce3aaf5daf304 Author: Sergi Massaneda <[email protected]> Date: Mon Oct 27 23:37:10 2025 +0100 [One Workflow] Actions to redux thunks (#238296) ## Summary Closes: https://github.com/elastic/security-team/issues/14105 Closes: https://github.com/elastic/security-team/issues/14001 Closes: https://github.com/elastic/security-team/issues/14305 This PR migrates workflow management actions from direct API calls to Redux thunks, centralizing state management and improving the overall architecture of the workflows management plugin. ### Key Changes #### 🔄 **Redux Store Architecture Refactor** - **Renamed store structure**: `WorkflowEditorState` → `WorkflowDetailState` for better clarity - **Updated store namespace**: `workflow` → `detail` in the Redux state tree - **Enhanced state management**: Added workflow metadata, test modal state, and change tracking - **Improved type safety**: Better TypeScript types with `WorkflowsStore` and `WorkflowDetailState` #### 🆕 **New Redux Thunks & Hooks** Created comprehensive thunk-based actions for all workflow operations: **New Thunks:** - `loadWorkflowThunk` - Load workflow data from API - `saveYamlThunk` - Save workflow YAML changes - `testWorkflowThunk` - Test workflow execution - `updateWorkflowThunk` - Update workflow properties (enable/disable) **New Custom Hooks:** - `useLoadWorkflow` - Load workflow with loading states - `useSaveYaml` - Save YAML with error handling - `useTestWorkflow` - Test workflow execution - `useToggleWorkflow` - Enable/disable workflow #### 🧹 **Cleanup of Unused Actions & Hooks** **Removed unused actions from `useWorkflowActions`:** - `createWorkflow` - Replaced by Redux thunk-based workflow creation in `saveYamlThunk` - `testWorkflow` - Replaced by Redux thunk-based testing in `testWorkflowThunk` **Removed unused hooks:** - `useWorkflowDetail` - Replaced by `loadWorkflowThunk` and Redux state management **Remaining actions in `useWorkflowActions`:** - `updateWorkflow` - Used in workflow list and bulk actions - `deleteWorkflows` - Used in workflow list and bulk actions - `runWorkflow` - Used in workflow list - `runIndividualStep` - Used in workflow detail editor - `cloneWorkflow` - Used in workflow list #### 🎨 **UI Component Restructuring** - **Enhanced `WorkflowDetailEditor`**: Integrated Redux state management and new action hooks (renamed from `WorkflowEditor`) - **Enhanced `WorkflowDetailHeader`**: Integrated Redux state management and new action hooks - **Simplified `WorkflowDetailPage`**: Reduced complexity by leveraging Redux store #### 🔧 **State Management Improvements** - **Centralized workflow data**: All workflow information now managed through Redux - **Better change tracking**: `selectHasChanges` selector for unsaved changes detection - **Enhanced selectors**: New selectors for workflow metadata, enabled state, and test modal - **Improved error handling**: Consistent error handling across all thunk actions #### 🌐 **API Integration** - **Consistent API calls**: All workflow operations now go through Redux thunks - **Better error handling**: Standardized error messages and toast notifications - **Query invalidation**: Automatic cache invalidation after mutations - **Loading states**: Proper loading indicators for all async operations #### 📁 **File Structure Changes** **Added:** - 6 new custom hooks in `lib/store/hooks/` - 4 new thunk files in `lib/store/thunks/` - `workflow_default_yml.ts` for default YAML content ### Technical Benefits 1. **Centralized State**: All workflow data managed in one place 2. **Better Error Handling**: Consistent error management across components 3. **Improved Performance**: Reduced prop drilling and unnecessary re-renders 4. **Type Safety**: Enhanced TypeScript support with proper typing 5. **Maintainability**: Cleaner separation of concerns between UI and data logic 6. **Code Cleanup**: Removed unused actions, hooks, and imports, reducing bundle size 7. **Excellent DevTools support**: Time-travel debugging, action replay This refactor significantly improves the maintainability and scalability of the workflows management feature while providing a more consistent user experience and a cleaner codebase. ## Screenshots <img width="1364" height="879" alt="Captura de pantalla 2025-10-24 a les 18 54 12" src="https://github.com/user-attachments/assets/92f83ba4-cfa5-4766-8555-708c216be69c" /> --------- Co-authored-by: kibanamachine <[email protected]> commit f7f0f9f65cfb76d03b9f99d8c45afc5ffc5d4a64 Author: Kibana Machine <[email protected]> Date: Mon Oct 27 22:55:33 2025 +0100 skip failing test suite (#240903) commit f7f55d2c9cadf607da98e733c2ba503a35e57071 Author: Alexi Doak <[email protected]> Date: Mon Oct 27 14:39:43 2025 -0700 [ResponseOps] Update CSV warning log for ES|QL reports (#240865) ## Summary This PR updates the max size reached log to help distinguish between ES|QL csv reports and regular ES search csv reports. ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios commit ed483fd5335c4d5a32d39fb6d07d024fe20cb945 Author: Saikat Sarkar <[email protected]> Date: Mon Oct 27 15:34:53 2025 -0600 Fix the responsiveness of Getting Started page (#240873) ## Summary Fix the responsiveness issue for the "create index" onboarding page. ### Before Fix <img width="970" height="669" alt="Screenshot 2025-10-27 at 11 41 29 AM" src="https://github.com/user-attachments/assets/776561c2-a979-4730-9403-83ca74b45f89" /> ### After Fix <img width="986" height="661" alt="Screenshot 2025-10-27 at 11 41 37 AM" src="https://github.com/user-attachments/assets/434a25a9-868c-4a4f-b4e7-a8fce733801e" /> ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... commit 717783b9b19af6a783b6171712a8a8173a2bc496 Author: Leszek Kubik <[email protected]> Date: Mon Oct 27 22:18:07 2025 +0100 Free disk space threshold actions error response (#240796) ## Summary Adds actions error response explaining user that action could not be completed because free disk space falls below threshold set in advanced Endpoint policy. ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.ht…
qn895
pushed a commit
to qn895/kibana
that referenced
this pull request
Oct 30, 2025
… installed rules. (elastic#237496) **Fixes: elastic#237278** **Fixes: elastic#97094** **Fixes: elastic#194066** ## Summary When a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes to find all rules matching a partial string on the rule name (like `win` to get all rules about `Windows`) they receive 0 matches. This is in contrast to the behavior they experience when searching available prebuilt "Elastic Rules", where partial name searches are available. This PR fixes this UX inconsistency by modifying the KQL output generated by the search query. Whereas previously a search for `"win"` would have generated the following KQL: ```sql (alert.attributes.name: "win" OR alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` It now treats the `alert.attributes.name` differently, allowing it to match on partial terms (ie `*win*` instead of `"win"`) and using `.keyword` index for better special character support. ```sql (alert.attributes.name.keyword: *win* OR # <-- here alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` ### 🕵️ BUT! .. We only do this for single term searches! Please note that this approach only applies to single term searches. For multiple term searches we maintain the "old" way of searching with quotations `"windows 10 patch"` instead of `*windows 10 patch*` or even `*windows* *10* *patch*`. The reasoning here is that since search results are NOT sorted by score, we want to avoid returning too many matches (wildcard searches match on _any_ combination of the terms, ie those with `windows` or `10` or `patch`) which would confuse the user just as they are trying to narrow down their search results!! (ie 🤔 why am I getting `Linux patch` rules when I'm asking for `windows 10 patch`??). ## How to test: To test this PR please checkout the relevant branch and run an instance of kibana/elasticsearch locally. 1. `Security App` > `Rules` > `Detection Rules (SIEM)` 2. A table of installed rules should appear. Remove all installed Elastic rules. 3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules` > `Bulk actions` > `Delete` > `Delete` 4. Go to `Add Elastic Rules` 5. `Search rules by name` > `win` > `Enter` 6. You should see 5 pages of results (~84 rules) 7. `Install All` 8. All Elastic rules have been installed > `Go back to installed Elastic rules` 9. Search by rule name > `win` > `Enter` 10. You should see 5 pages of results (~84 rules - same as in Step 6.) Please feel free to try some additional search queries. <details> <summary> ### 👉 Click for additional testing ideas </summary> Here are some single term searches: - `goog` - `proc` - `sql` (should include `Postgresql` and `MSSQL`) - `shell` (should include `Powershell`) - `inject` (should include `injection`) - `git` (should include `GitHub`) - `.exe` - `pub/sub` (exact matches only) - `user-agent` (exact matches only) - `/bin` - `CVE-2025` (should include partial matches) - `-` (should return matches with dash in the name like `user-agent`) - `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`) - `|` (should return no matches - no elastic rules with this) And behavior for multiple term searches: - `AWS` (check the count), then `AWS IAM` (there should be less results for `AWS IAM`) - `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub top` partial match has no matches!) - `root` then `root cert` then `root certificate` (the second has no results, `root certificate` should only return exact matches) - `proc`, then `process`, then `process injection`, then `potential process injection` (each should return less results) </details> ## Screenshots  ## Special character support Note that by switching to wildcard searches (ie `*win*`) on the `.keyword` index and fully escaping special characters in KQL we'll **_ALSO_** be allowing special character searches on single search terms. For example, searching by `user-agent` will return results that only match `User-Agent` but not `user` or `agent` individually. Some other useful example of these approach are searches for: `Pub/Sub`, `CVE-2025`, `/bin`, `_`, `.exe`. This support, in addition for escaping the backslash `\` character will allow us to close the next TWO ISSUES in this epic 🥳 wohoo!! 👉 elastic#97094 (special chars in rule name) and, 👉 elastic#194066 (special chars in tags) <img width="800" alt="image" src="https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7" /> ### Testing special character support. In addition to the steps above (installing all elastic rules), we can: 1. Create a rule with special characters: ie `Rule with special chars (&, *, #, $, ?, >, @, \, /, ", ‘, {, [, ;)` 2. Try some single term matches: - `@` - `&` (additional elastic rules with this term should appear). - `*` - `"` - `:` - `>` - `{` - `\` (backslash, this used to break the search under elastic#97094) ## Risks These are some of the risks that could be identified by using this approach. ### 1. Dependencies that reuse query logic Note that the `searchTerm -> KQL` conversion for searching for rules is used a few places. - Installed Rules - Rule Monitoring - Bulk actions **Note**: All these paths are tested manually, and form part of the automated tests. ### 2. Performance (ie `allowLeadingWildcards`) We're using wildcard searches before and after the search term (ie `*win*`) in order to replicate the behavior across both 'prebuilt' and 'installed' rules tables. The wildcard AFTER the term (`win*` => matching terms like `Windows`) is no problem but the one BEFORE (`*win` => matching terms like `Darwin`) could create an issue. Our [KQL documentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards) warns that Kibana UI Advanced Settings have `query:allowLeadingWildcards` turned off by default. This is for performance reasons as a leading wildcard can have a large impact when searching indexes that have millions of terms associated with them. <img width="2784" height="2120" alt="image" src="https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde" /> > By default, leading wildcards are not allowed for performance reasons. You can modify this with the [query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards) advanced setting. Please note that the [Query DSL docs also warn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard) about avoiding this approach. There is also more info and additional warnings under [Lucene API docs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html). The risk here is two pronged: 1. Users with a lot (millions?) of detection rules will likely have a degraded search experience as queries will take longer to execute. 2. Leading wildcards appear to be working by default (in contrast to what is stated in the documentation). But the mere _existence_ of various settings to avoid them ([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards), [`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard), [`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)) is a risk, because some users may have a special setup we've not been able to anticipate in our testing, leading to potential issues like this one: elastic#57828 ### Mitigating factors: 1. Please note that in the case of security detection rules, the prebuilt rules package is _only ~1500 rules_! Users do not manage millions of rules, they generally manage low thousands or even hundreds. We've tested 100K rules successfully and there was [_no perceivable difference in terms of search performance_](elastic#237496 (comment)). And this seems to be a realistic test when checking actual usage stats (our 10 largest users in Sep'25 had between 60-160K installed). Also, due to current limits on pagination and bulk action logic we would likely hit different kinds of problems here _before_ search performance becomes an issue. 2. We've tested all the documented ways to disable leading wildcards, including disabling them manually (under `Server Management > Advanced Settings`) and explicitly in the `kibana.yml`. None of them seemed to affect searches carried out on Saved Objects. This is because our solution uses the [`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting) plugin under the hood which [converts the filter to KQL without referring to any UI Advanced Settings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23). And since there is no explicit setting for `allowLeadingWildcards` the default setting of `true` gets applied instead [inside the `grammar.peggy` file](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12). ### Risks that were found acceptable In the search for any possible settings that might affect the rollout of changes under this PR [we did find a setting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7) inside `elasticsearch.yml` that causes problems with the proposed solution. **TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯 <img width="600" alt="image" src="https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44" /> However we also found bigger problems, _this setting also prevents the installation of prebuilt rules_ (see below), which are a cornerstone of our security solution. <img width="600" alt="image" src="https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1" /> In other words, the setting `search.allow_expensive_queries=true` has become _a de-facto requirement of Detection Rules_, that has not yet been documented. Hence we including it to the [documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) as part of this PR. Note also that the errors here are not limited to detection rules. We found that the setting _also compromised or outright broke a lot of functionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved Object search, Tags, Server Monitoring etc). More about this is [documented in this internal document](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0). And in this [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469). So we're assuming that most if not all of our users will have it set to `true`. ## Checklist Check the PR satisfies following conditions. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Follow the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. - [x] Changes have been socialized with the PM and rest of the team. - [x] All identified Risks have been properly documented and investigated. ([internal investigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0)) - [x] New requirements added to the technical docs (PR [elastic#3543](elastic/docs-content#3543), see [here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))
ana-davydova
pushed a commit
to ana-davydova/kibana
that referenced
this pull request
Nov 3, 2025
… installed rules. (elastic#237496) **Fixes: elastic#237278** **Fixes: elastic#97094** **Fixes: elastic#194066** ## Summary When a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes to find all rules matching a partial string on the rule name (like `win` to get all rules about `Windows`) they receive 0 matches. This is in contrast to the behavior they experience when searching available prebuilt "Elastic Rules", where partial name searches are available. This PR fixes this UX inconsistency by modifying the KQL output generated by the search query. Whereas previously a search for `"win"` would have generated the following KQL: ```sql (alert.attributes.name: "win" OR alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` It now treats the `alert.attributes.name` differently, allowing it to match on partial terms (ie `*win*` instead of `"win"`) and using `.keyword` index for better special character support. ```sql (alert.attributes.name.keyword: *win* OR # <-- here alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` ### 🕵️ BUT! .. We only do this for single term searches! Please note that this approach only applies to single term searches. For multiple term searches we maintain the "old" way of searching with quotations `"windows 10 patch"` instead of `*windows 10 patch*` or even `*windows* *10* *patch*`. The reasoning here is that since search results are NOT sorted by score, we want to avoid returning too many matches (wildcard searches match on _any_ combination of the terms, ie those with `windows` or `10` or `patch`) which would confuse the user just as they are trying to narrow down their search results!! (ie 🤔 why am I getting `Linux patch` rules when I'm asking for `windows 10 patch`??). ## How to test: To test this PR please checkout the relevant branch and run an instance of kibana/elasticsearch locally. 1. `Security App` > `Rules` > `Detection Rules (SIEM)` 2. A table of installed rules should appear. Remove all installed Elastic rules. 3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules` > `Bulk actions` > `Delete` > `Delete` 4. Go to `Add Elastic Rules` 5. `Search rules by name` > `win` > `Enter` 6. You should see 5 pages of results (~84 rules) 7. `Install All` 8. All Elastic rules have been installed > `Go back to installed Elastic rules` 9. Search by rule name > `win` > `Enter` 10. You should see 5 pages of results (~84 rules - same as in Step 6.) Please feel free to try some additional search queries. <details> <summary> ### 👉 Click for additional testing ideas </summary> Here are some single term searches: - `goog` - `proc` - `sql` (should include `Postgresql` and `MSSQL`) - `shell` (should include `Powershell`) - `inject` (should include `injection`) - `git` (should include `GitHub`) - `.exe` - `pub/sub` (exact matches only) - `user-agent` (exact matches only) - `/bin` - `CVE-2025` (should include partial matches) - `-` (should return matches with dash in the name like `user-agent`) - `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`) - `|` (should return no matches - no elastic rules with this) And behavior for multiple term searches: - `AWS` (check the count), then `AWS IAM` (there should be less results for `AWS IAM`) - `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub top` partial match has no matches!) - `root` then `root cert` then `root certificate` (the second has no results, `root certificate` should only return exact matches) - `proc`, then `process`, then `process injection`, then `potential process injection` (each should return less results) </details> ## Screenshots  ## Special character support Note that by switching to wildcard searches (ie `*win*`) on the `.keyword` index and fully escaping special characters in KQL we'll **_ALSO_** be allowing special character searches on single search terms. For example, searching by `user-agent` will return results that only match `User-Agent` but not `user` or `agent` individually. Some other useful example of these approach are searches for: `Pub/Sub`, `CVE-2025`, `/bin`, `_`, `.exe`. This support, in addition for escaping the backslash `\` character will allow us to close the next TWO ISSUES in this epic 🥳 wohoo!! 👉 elastic#97094 (special chars in rule name) and, 👉 elastic#194066 (special chars in tags) <img width="800" alt="image" src="https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7" /> ### Testing special character support. In addition to the steps above (installing all elastic rules), we can: 1. Create a rule with special characters: ie `Rule with special chars (&, *, #, $, ?, >, @, \, /, ", ‘, {, [, ;)` 2. Try some single term matches: - `@` - `&` (additional elastic rules with this term should appear). - `*` - `"` - `:` - `>` - `{` - `\` (backslash, this used to break the search under elastic#97094) ## Risks These are some of the risks that could be identified by using this approach. ### 1. Dependencies that reuse query logic Note that the `searchTerm -> KQL` conversion for searching for rules is used a few places. - Installed Rules - Rule Monitoring - Bulk actions **Note**: All these paths are tested manually, and form part of the automated tests. ### 2. Performance (ie `allowLeadingWildcards`) We're using wildcard searches before and after the search term (ie `*win*`) in order to replicate the behavior across both 'prebuilt' and 'installed' rules tables. The wildcard AFTER the term (`win*` => matching terms like `Windows`) is no problem but the one BEFORE (`*win` => matching terms like `Darwin`) could create an issue. Our [KQL documentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards) warns that Kibana UI Advanced Settings have `query:allowLeadingWildcards` turned off by default. This is for performance reasons as a leading wildcard can have a large impact when searching indexes that have millions of terms associated with them. <img width="2784" height="2120" alt="image" src="https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde" /> > By default, leading wildcards are not allowed for performance reasons. You can modify this with the [query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards) advanced setting. Please note that the [Query DSL docs also warn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard) about avoiding this approach. There is also more info and additional warnings under [Lucene API docs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html). The risk here is two pronged: 1. Users with a lot (millions?) of detection rules will likely have a degraded search experience as queries will take longer to execute. 2. Leading wildcards appear to be working by default (in contrast to what is stated in the documentation). But the mere _existence_ of various settings to avoid them ([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards), [`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard), [`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)) is a risk, because some users may have a special setup we've not been able to anticipate in our testing, leading to potential issues like this one: elastic#57828 ### Mitigating factors: 1. Please note that in the case of security detection rules, the prebuilt rules package is _only ~1500 rules_! Users do not manage millions of rules, they generally manage low thousands or even hundreds. We've tested 100K rules successfully and there was [_no perceivable difference in terms of search performance_](elastic#237496 (comment)). And this seems to be a realistic test when checking actual usage stats (our 10 largest users in Sep'25 had between 60-160K installed). Also, due to current limits on pagination and bulk action logic we would likely hit different kinds of problems here _before_ search performance becomes an issue. 2. We've tested all the documented ways to disable leading wildcards, including disabling them manually (under `Server Management > Advanced Settings`) and explicitly in the `kibana.yml`. None of them seemed to affect searches carried out on Saved Objects. This is because our solution uses the [`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting) plugin under the hood which [converts the filter to KQL without referring to any UI Advanced Settings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23). And since there is no explicit setting for `allowLeadingWildcards` the default setting of `true` gets applied instead [inside the `grammar.peggy` file](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12). ### Risks that were found acceptable In the search for any possible settings that might affect the rollout of changes under this PR [we did find a setting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7) inside `elasticsearch.yml` that causes problems with the proposed solution. **TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯 <img width="600" alt="image" src="https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44" /> However we also found bigger problems, _this setting also prevents the installation of prebuilt rules_ (see below), which are a cornerstone of our security solution. <img width="600" alt="image" src="https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1" /> In other words, the setting `search.allow_expensive_queries=true` has become _a de-facto requirement of Detection Rules_, that has not yet been documented. Hence we including it to the [documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) as part of this PR. Note also that the errors here are not limited to detection rules. We found that the setting _also compromised or outright broke a lot of functionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved Object search, Tags, Server Monitoring etc). More about this is [documented in this internal document](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0). And in this [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469). So we're assuming that most if not all of our users will have it set to `true`. ## Checklist Check the PR satisfies following conditions. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Follow the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. - [x] Changes have been socialized with the PM and rest of the team. - [x] All identified Risks have been properly documented and investigated. ([internal investigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0)) - [x] New requirements added to the technical docs (PR [elastic#3543](elastic/docs-content#3543), see [here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))
albertoblaz
pushed a commit
to albertoblaz/kibana
that referenced
this pull request
Nov 4, 2025
… installed rules. (elastic#237496) **Fixes: elastic#237278** **Fixes: elastic#97094** **Fixes: elastic#194066** ## Summary When a user navigates to `Rules` > `Detection Rules (SIEM)` and wishes to find all rules matching a partial string on the rule name (like `win` to get all rules about `Windows`) they receive 0 matches. This is in contrast to the behavior they experience when searching available prebuilt "Elastic Rules", where partial name searches are available. This PR fixes this UX inconsistency by modifying the KQL output generated by the search query. Whereas previously a search for `"win"` would have generated the following KQL: ```sql (alert.attributes.name: "win" OR alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` It now treats the `alert.attributes.name` differently, allowing it to match on partial terms (ie `*win*` instead of `"win"`) and using `.keyword` index for better special character support. ```sql (alert.attributes.name.keyword: *win* OR # <-- here alert.attributes.params.index: "win" OR alert.attributes.params.threat.tactic.id: "win" OR ... ``` ### 🕵️ BUT! .. We only do this for single term searches! Please note that this approach only applies to single term searches. For multiple term searches we maintain the "old" way of searching with quotations `"windows 10 patch"` instead of `*windows 10 patch*` or even `*windows* *10* *patch*`. The reasoning here is that since search results are NOT sorted by score, we want to avoid returning too many matches (wildcard searches match on _any_ combination of the terms, ie those with `windows` or `10` or `patch`) which would confuse the user just as they are trying to narrow down their search results!! (ie 🤔 why am I getting `Linux patch` rules when I'm asking for `windows 10 patch`??). ## How to test: To test this PR please checkout the relevant branch and run an instance of kibana/elasticsearch locally. 1. `Security App` > `Rules` > `Detection Rules (SIEM)` 2. A table of installed rules should appear. Remove all installed Elastic rules. 3. `Elastic Rules` (filter) > Tick to select all > `Select all X Rules` > `Bulk actions` > `Delete` > `Delete` 4. Go to `Add Elastic Rules` 5. `Search rules by name` > `win` > `Enter` 6. You should see 5 pages of results (~84 rules) 7. `Install All` 8. All Elastic rules have been installed > `Go back to installed Elastic rules` 9. Search by rule name > `win` > `Enter` 10. You should see 5 pages of results (~84 rules - same as in Step 6.) Please feel free to try some additional search queries. <details> <summary> ### 👉 Click for additional testing ideas </summary> Here are some single term searches: - `goog` - `proc` - `sql` (should include `Postgresql` and `MSSQL`) - `shell` (should include `Powershell`) - `inject` (should include `injection`) - `git` (should include `GitHub`) - `.exe` - `pub/sub` (exact matches only) - `user-agent` (exact matches only) - `/bin` - `CVE-2025` (should include partial matches) - `-` (should return matches with dash in the name like `user-agent`) - `_` (should return matches with dash in the name like `CAP_SYS_ADMIN`) - `|` (should return no matches - no elastic rules with this) And behavior for multiple term searches: - `AWS` (check the count), then `AWS IAM` (there should be less results for `AWS IAM`) - `pub/sub topic` (should be less than for `pub/sub`, note that `pub/sub top` partial match has no matches!) - `root` then `root cert` then `root certificate` (the second has no results, `root certificate` should only return exact matches) - `proc`, then `process`, then `process injection`, then `potential process injection` (each should return less results) </details> ## Screenshots  ## Special character support Note that by switching to wildcard searches (ie `*win*`) on the `.keyword` index and fully escaping special characters in KQL we'll **_ALSO_** be allowing special character searches on single search terms. For example, searching by `user-agent` will return results that only match `User-Agent` but not `user` or `agent` individually. Some other useful example of these approach are searches for: `Pub/Sub`, `CVE-2025`, `/bin`, `_`, `.exe`. This support, in addition for escaping the backslash `\` character will allow us to close the next TWO ISSUES in this epic 🥳 wohoo!! 👉 elastic#97094 (special chars in rule name) and, 👉 elastic#194066 (special chars in tags) <img width="800" alt="image" src="https://github.com/user-attachments/assets/5411d8e8-b3f7-4a3f-9863-c64cc2a7df2c" /> <img width="800" alt="image" src="https://github.com/user-attachments/assets/c3589add-9f80-4646-807a-3f0c8a2ec5a7" /> ### Testing special character support. In addition to the steps above (installing all elastic rules), we can: 1. Create a rule with special characters: ie `Rule with special chars (&, *, #, $, ?, >, @, \, /, ", ‘, {, [, ;)` 2. Try some single term matches: - `@` - `&` (additional elastic rules with this term should appear). - `*` - `"` - `:` - `>` - `{` - `\` (backslash, this used to break the search under elastic#97094) ## Risks These are some of the risks that could be identified by using this approach. ### 1. Dependencies that reuse query logic Note that the `searchTerm -> KQL` conversion for searching for rules is used a few places. - Installed Rules - Rule Monitoring - Bulk actions **Note**: All these paths are tested manually, and form part of the automated tests. ### 2. Performance (ie `allowLeadingWildcards`) We're using wildcard searches before and after the search term (ie `*win*`) in order to replicate the behavior across both 'prebuilt' and 'installed' rules tables. The wildcard AFTER the term (`win*` => matching terms like `Windows`) is no problem but the one BEFORE (`*win` => matching terms like `Darwin`) could create an issue. Our [KQL documentation](https://www.elastic.co/docs/reference/query-languages/kql#_filter_for_documents_using_wildcards) warns that Kibana UI Advanced Settings have `query:allowLeadingWildcards` turned off by default. This is for performance reasons as a leading wildcard can have a large impact when searching indexes that have millions of terms associated with them. <img width="2784" height="2120" alt="image" src="https://github.com/user-attachments/assets/9e0a754c-c30b-453e-a3ae-60a104b13fde" /> > By default, leading wildcards are not allowed for performance reasons. You can modify this with the [query:allowLeadingWildcards](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards) advanced setting. Please note that the [Query DSL docs also warn](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard) about avoiding this approach. There is also more info and additional warnings under [Lucene API docs](https://lucene.apache.org/core/9_12_3/core/org/apache/lucene/search/WildcardQuery.html). The risk here is two pronged: 1. Users with a lot (millions?) of detection rules will likely have a degraded search experience as queries will take longer to execute. 2. Leading wildcards appear to be working by default (in contrast to what is stated in the documentation). But the mere _existence_ of various settings to avoid them ([`allowLeadingWildcards`](https://www.elastic.co/docs/reference/kibana/advanced-settings#query-allowleadingwildcards), [`allow_leading_wildcard`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard), [`analyze_wildcards`](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-query-string-query#query-string-wildcard)) is a risk, because some users may have a special setup we've not been able to anticipate in our testing, leading to potential issues like this one: elastic#57828 ### Mitigating factors: 1. Please note that in the case of security detection rules, the prebuilt rules package is _only ~1500 rules_! Users do not manage millions of rules, they generally manage low thousands or even hundreds. We've tested 100K rules successfully and there was [_no perceivable difference in terms of search performance_](elastic#237496 (comment)). And this seems to be a realistic test when checking actual usage stats (our 10 largest users in Sep'25 had between 60-160K installed). Also, due to current limits on pagination and bulk action logic we would likely hit different kinds of problems here _before_ search performance becomes an issue. 2. We've tested all the documented ways to disable leading wildcards, including disabling them manually (under `Server Management > Advanced Settings`) and explicitly in the `kibana.yml`. None of them seemed to affect searches carried out on Saved Objects. This is because our solution uses the [`alerting`](http://github.com/elastic/kibana/tree/main/x-pack/platform/plugins/shared/alerting) plugin under the hood which [converts the filter to KQL without referring to any UI Advanced Settings](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/x-pack/platform/plugins/shared/alerting/server/rules_client/common/build_kuery_node_filter.ts#L23). And since there is no explicit setting for `allowLeadingWildcards` the default setting of `true` gets applied instead [inside the `grammar.peggy` file](https://github.com/elastic/kibana/blob/91cab0e1369473846dc2712aa7dfe38b8580a9a5/src/platform/packages/shared/kbn-es-query/src/kuery/grammar/grammar.peggy#L12). ### Risks that were found acceptable In the search for any possible settings that might affect the rollout of changes under this PR [we did find a setting](https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-wildcard-query#_allow_expensive_queries_7) inside `elasticsearch.yml` that causes problems with the proposed solution. **TL/DR** 👉 `search.allow_expensive_queries=false` breaks the search.💥🤯 <img width="600" alt="image" src="https://github.com/user-attachments/assets/8d523d15-f832-4488-8c78-4ed60c474d44" /> However we also found bigger problems, _this setting also prevents the installation of prebuilt rules_ (see below), which are a cornerstone of our security solution. <img width="600" alt="image" src="https://github.com/user-attachments/assets/2b8d91e8-29b9-4f06-9f76-6b1edb999fd1" /> In other words, the setting `search.allow_expensive_queries=true` has become _a de-facto requirement of Detection Rules_, that has not yet been documented. Hence we including it to the [documentation](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements) as part of this PR. Note also that the errors here are not limited to detection rules. We found that the setting _also compromised or outright broke a lot of functionality in Kibana_ 💥🤯 (including Fleet, API Keys, Timelines, Saved Object search, Tags, Server Monitoring etc). More about this is [documented in this internal document](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0). And in this [internal slack thread](https://elastic.slack.com/archives/C02HA9E8221/p1760694975799469). So we're assuming that most if not all of our users will have it set to `true`. ## Checklist Check the PR satisfies following conditions. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [x] Follow the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. - [x] Changes have been socialized with the PM and rest of the team. - [x] All identified Risks have been properly documented and investigated. ([internal investigation](https://docs.google.com/document/d/1HLOXQZFcm1-KBj9DHTqwcF3wDdLOE6CcgUzqzZA2CAg/edit?tab=t.0)) - [x] New requirements added to the technical docs (PR [elastic#3543](elastic/docs-content#3543), see [here](https://www.elastic.co/docs/solutions/security/detect-and-alert/detections-requirements))
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
De-facto requirement of Security Detections, (and much of Kibana) that has not been documented properly.
Errors go as far back as
v8.18(but likely much further), and not only affect Security app, but also manyServer Managementfeatures such as Fleet, Saved Objects, Tags, etc.For more info: elastic/kibana#237496 -> Section
Risks that were found acceptableWe're documenting it under the correct location to make sure that users know about it. There is an internal slack thread for discussion.
Note also that this is documented as a requirement of the 'alerting' plugin that we're using under the hood.