Skip to content

Commit 482953d

Browse files
nikitaindiknikitaindik
authored
[Security Solution] Fix "too many clauses" error on prebuilt rules installation page (#223240)
**Resolves: #223399 ## Summary This PR fixes an error on the "Add Elastic rules" page. The error is shown when running a local dev environment from `main` branch and going to the "Add Elastic rules" page. <img width="1741" alt="Screenshot 2025-06-10 at 11 28 19" src="https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6" /> ## Changes PR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES into smaller chunks to avoid the error. ## Cause Kibana makes a search request to ES with a filter that has too many clauses, so ES rejects with an error. More specifically, `/prebuilt_rules/installation/_review` route handler calls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all installable rules. To do this, we construct a request with thousands of clauses in a filter. ES counts the number of clauses in a filter and rejects because it's bigger than `maxClauseCount`. `maxClauseCount` value is computed dynamically by ES and its size depends on hardware and available resources ([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html), [code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)). The minimum value for `maxClauseCount` is 1024. ## Why it didn't fail before Two reasons: 1. ES changed how `maxClauseCount` is computed. They've recently merged a [PR](elastic/elasticsearch#128293) that made queries against numeric types count three times towards the `maxClauseCount` limit. They plan to revert the change in [this PR](elastic/elasticsearch#129206). 2. Prebuilt rule packages are growing bigger with each version, resulting in a bigger number of clauses. I've tested behaviour with ES change in place on different package versions: - 8.17.1 (contains 1262 rule versions) - no "too many clauses" error - 8.18.1 (contains 1356 rule versions) - causes "too many clauses" error - 9.0.1 (also contains 1356 rule versions) - causes "too many clauses" error The precise number of versions that start to cause errors is 1293 on my laptop. So even if ES team rolls back their change, we still need to make sure we don't go over the limit with ever-growing prebuilt rule package sizes.
1 parent e6605dc commit 482953d

File tree

1 file changed

+120
-52
lines changed

1 file changed

+120
-52
lines changed

‎x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client.ts‎

Lines changed: 120 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,8 @@
55
* 2.0.
66
*/
77

8-
import { uniqBy } from 'lodash';
8+
import { chunk, uniqBy } from 'lodash';
9+
import pMap from 'p-map';
910
import type {
1011
AggregationsMultiBucketAggregateBase,
1112
AggregationsTopHitsAggregate,
@@ -18,7 +19,10 @@ import { validatePrebuiltRuleAssets } from './prebuilt_rule_assets_validation';
1819
import { PREBUILT_RULE_ASSETS_SO_TYPE } from './prebuilt_rule_assets_type';
1920
import type { RuleVersionSpecifier } from '../rule_versions/rule_version_specifier';
2021

22+
const RULE_ASSET_ATTRIBUTES = `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes`;
2123
const MAX_PREBUILT_RULES_COUNT = 10_000;
24+
const ES_MAX_CLAUSE_COUNT = 1024;
25+
const ES_MAX_CONCURRENT_REQUESTS = 2;
2226

2327
export interface IPrebuiltRuleAssetsClient {
2428
fetchLatestAssets: () => Promise<PrebuiltRuleAsset[]>;
@@ -81,50 +85,60 @@ export const createPrebuiltRuleAssetsClient = (
8185
return [];
8286
}
8387

84-
const filter = ruleIds
85-
?.map((ruleId) => `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes.rule_id: ${ruleId}`)
86-
.join(' OR ');
87-
88-
const findResult = await savedObjectsClient.find<
89-
PrebuiltRuleAsset,
90-
{
91-
rules: AggregationsMultiBucketAggregateBase<{
92-
latest_version: AggregationsTopHitsAggregate;
93-
}>;
94-
}
95-
>({
96-
type: PREBUILT_RULE_ASSETS_SO_TYPE,
97-
filter,
98-
aggs: {
99-
rules: {
100-
terms: {
101-
field: `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes.rule_id`,
102-
size: MAX_PREBUILT_RULES_COUNT,
103-
},
104-
aggs: {
105-
latest_version: {
106-
top_hits: {
107-
size: 1,
108-
sort: [
109-
{
110-
[`${PREBUILT_RULE_ASSETS_SO_TYPE}.version`]: 'desc',
111-
},
112-
],
113-
_source: [
114-
`${PREBUILT_RULE_ASSETS_SO_TYPE}.rule_id`,
115-
`${PREBUILT_RULE_ASSETS_SO_TYPE}.version`,
116-
],
88+
const fetchLatestVersionInfo = async (filter?: string) => {
89+
const findResult = await savedObjectsClient.find<
90+
PrebuiltRuleAsset,
91+
{
92+
rules: AggregationsMultiBucketAggregateBase<{
93+
latest_version: AggregationsTopHitsAggregate;
94+
}>;
95+
}
96+
>({
97+
type: PREBUILT_RULE_ASSETS_SO_TYPE,
98+
filter,
99+
aggs: {
100+
rules: {
101+
terms: {
102+
field: `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes.rule_id`,
103+
size: MAX_PREBUILT_RULES_COUNT,
104+
},
105+
aggs: {
106+
latest_version: {
107+
top_hits: {
108+
size: 1,
109+
sort: [
110+
{
111+
[`${PREBUILT_RULE_ASSETS_SO_TYPE}.version`]: 'desc',
112+
},
113+
],
114+
_source: [
115+
`${PREBUILT_RULE_ASSETS_SO_TYPE}.rule_id`,
116+
`${PREBUILT_RULE_ASSETS_SO_TYPE}.version`,
117+
],
118+
},
117119
},
118120
},
119121
},
120122
},
121-
},
122-
});
123+
});
123124

124-
const buckets = findResult.aggregations?.rules?.buckets ?? [];
125-
invariant(Array.isArray(buckets), 'Expected buckets to be an array');
125+
const aggregatedBuckets = findResult.aggregations?.rules?.buckets ?? [];
126+
invariant(Array.isArray(aggregatedBuckets), 'Expected buckets to be an array');
127+
128+
return aggregatedBuckets;
129+
};
130+
131+
const filters = ruleIds
132+
? createChunkedFilters({
133+
items: ruleIds,
134+
mapperFn: (ruleId) => `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes.rule_id: ${ruleId}`,
135+
clausesPerItem: 2,
136+
})
137+
: undefined;
126138

127-
return buckets.map((bucket) => {
139+
const buckets = await chunkedFetch(fetchLatestVersionInfo, filters);
140+
141+
const latestVersions = buckets.map((bucket) => {
128142
const hit = bucket.latest_version.hits.hits[0];
129143
const soAttributes = hit._source[PREBUILT_RULE_ASSETS_SO_TYPE];
130144
const versionInfo: RuleVersionSpecifier = {
@@ -133,6 +147,8 @@ export const createPrebuiltRuleAssetsClient = (
133147
};
134148
return versionInfo;
135149
});
150+
151+
return latestVersions;
136152
});
137153
},
138154

@@ -143,21 +159,26 @@ export const createPrebuiltRuleAssetsClient = (
143159
return [];
144160
}
145161

146-
const attr = `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes`;
147-
const filter = versions
148-
.map((v) => `(${attr}.rule_id: ${v.rule_id} AND ${attr}.version: ${v.version})`)
149-
.join(' OR ');
150-
151-
// Usage of savedObjectsClient.bulkGet() is ~25% more performant and
152-
// simplifies deduplication but too many tests get broken.
153-
// See https://github.com/elastic/kibana/issues/218198
154-
const findResult = await savedObjectsClient.find<PrebuiltRuleAsset>({
155-
type: PREBUILT_RULE_ASSETS_SO_TYPE,
156-
filter,
157-
perPage: MAX_PREBUILT_RULES_COUNT,
162+
const filters = createChunkedFilters({
163+
items: versions,
164+
mapperFn: (versionSpecifier) =>
165+
`(${RULE_ASSET_ATTRIBUTES}.rule_id: ${versionSpecifier.rule_id} AND ${RULE_ASSET_ATTRIBUTES}.version: ${versionSpecifier.version})`,
166+
clausesPerItem: 4,
158167
});
159168

160-
const ruleAssets = findResult.saved_objects.map((so) => so.attributes);
169+
const ruleAssets = await chunkedFetch(async (filter) => {
170+
// Usage of savedObjectsClient.bulkGet() is ~25% more performant and
171+
// simplifies deduplication but too many tests get broken.
172+
// See https://github.com/elastic/kibana/issues/218198
173+
const findResult = await savedObjectsClient.find<PrebuiltRuleAsset>({
174+
type: PREBUILT_RULE_ASSETS_SO_TYPE,
175+
filter,
176+
perPage: MAX_PREBUILT_RULES_COUNT,
177+
});
178+
179+
return findResult.saved_objects.map((so) => so.attributes);
180+
}, filters);
181+
161182
// Rule assets may have duplicates we have to get rid of.
162183
// In particular prebuilt rule assets package v8.17.1 has duplicates.
163184
const uniqueRuleAssets = uniqBy(ruleAssets, 'rule_id');
@@ -167,3 +188,50 @@ export const createPrebuiltRuleAssetsClient = (
167188
},
168189
};
169190
};
191+
192+
/**
193+
* Creates an array of KQL filter strings for a collection of items.
194+
* Uses chunking to ensure that the number of filter clauses does not exceed the ES "too_many_clauses" limit.
195+
* See: https://github.com/elastic/kibana/pull/223240
196+
*
197+
* @param {object} options
198+
* @param {T[]} options.items - Array of items to create filters for.
199+
* @param {(item: T) => string} options.mapperFn - A function that maps an item to a filter string.
200+
* @param {number} options.clausesPerItem - Number of Elasticsearch clauses generated per item. Determined empirically by converting a KQL filter into a Query DSL query.
201+
* More complex filters will result in more clauses. Info about clauses in docs: https://www.elastic.co/docs/explore-analyze/query-filter/languages/querydsl#query-dsl
202+
* @returns {string[]} An array of filter strings
203+
*/
204+
function createChunkedFilters<T>({
205+
items,
206+
mapperFn,
207+
clausesPerItem,
208+
}: {
209+
items: T[];
210+
mapperFn: (item: T) => string;
211+
clausesPerItem: number;
212+
}): string[] {
213+
return chunk(items, ES_MAX_CLAUSE_COUNT / clausesPerItem).map((singleChunk) =>
214+
singleChunk.map(mapperFn).join(' OR ')
215+
);
216+
}
217+
218+
/**
219+
* Fetches objects using a provided function.
220+
* If filters are provided fetches concurrently in chunks.
221+
*
222+
* @param {(filter?: string) => Promise<T[]>} chunkFetchFn - Function that fetches a chunk.
223+
* @param {string[]} [filters] - An optional array of filter strings. If provided, `chunkFetchFn` will be called for each filter concurrently.
224+
* @returns {Promise<T[]>} A promise that resolves to an array of fetched objects.
225+
*/
226+
function chunkedFetch<T>(
227+
chunkFetchFn: (filter?: string) => Promise<T[]>,
228+
filters?: string[]
229+
): Promise<T[]> {
230+
if (filters?.length) {
231+
return pMap(filters, chunkFetchFn, {
232+
concurrency: ES_MAX_CONCURRENT_REQUESTS,
233+
}).then((results) => results.flat());
234+
}
235+
236+
return chunkFetchFn();
237+
}

0 commit comments

Comments
 (0)