Skip to content

Commit 35ce7e0

Browse files
nikitaindiknikitaindik
committed
[Security Solution] Fix "too many clauses" error on prebuilt rules installation page (elastic#223240)
**Resolves: elastic#223399 ## Summary This PR fixes an error on the "Add Elastic rules" page. The error is shown when running a local dev environment from `main` branch and going to the "Add Elastic rules" page. <img width="1741" alt="Screenshot 2025-06-10 at 11 28 19" src="https://github.com/user-attachments/assets/f8f81f88-3749-491f-bcdb-cd51f465bda6" /> ## Changes PR updates methods of `PrebuiltRuleAssetsClient` to split requests to ES into smaller chunks to avoid the error. ## Cause Kibana makes a search request to ES with a filter that has too many clauses, so ES rejects with an error. More specifically, `/prebuilt_rules/installation/_review` route handler calls `PrebuiltRuleAssetsClient.fetchAssetsByVersion` to fetch all installable rules. To do this, we construct a request with thousands of clauses in a filter. ES counts the number of clauses in a filter and rejects because it's bigger than `maxClauseCount`. `maxClauseCount` value is computed dynamically by ES and its size depends on hardware and available resources ([docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.18/search-settings.html), [code](https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/search/SearchUtils.java)). The minimum value for `maxClauseCount` is 1024. ## Why it didn't fail before Two reasons: 1. ES changed how `maxClauseCount` is computed. They've recently merged a [PR](elastic/elasticsearch#128293) that made queries against numeric types count three times towards the `maxClauseCount` limit. They plan to revert the change in [this PR](elastic/elasticsearch#129206). 2. Prebuilt rule packages are growing bigger with each version, resulting in a bigger number of clauses. I've tested behaviour with ES change in place on different package versions: - 8.17.1 (contains 1262 rule versions) - no "too many clauses" error - 8.18.1 (contains 1356 rule versions) - causes "too many clauses" error - 9.0.1 (also contains 1356 rule versions) - causes "too many clauses" error The precise number of versions that start to cause errors is 1293 on my laptop. So even if ES team rolls back their change, we still need to make sure we don't go over the limit with ever-growing prebuilt rule package sizes. (cherry picked from commit 482953d) # Conflicts: # x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client.ts
1 parent c9f6c3d commit 35ce7e0

File tree

1 file changed

+123
-51
lines changed

1 file changed

+123
-51
lines changed

‎x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client.ts‎

Lines changed: 123 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,8 @@
55
* 2.0.
66
*/
77

8-
import { uniqBy } from 'lodash';
8+
import { chunk, uniqBy } from 'lodash';
9+
import pMap from 'p-map';
910
import type {
1011
AggregationsMultiBucketAggregateBase,
1112
AggregationsTopHitsAggregate,
@@ -18,7 +19,10 @@ import { validatePrebuiltRuleAssets } from './prebuilt_rule_assets_validation';
1819
import { PREBUILT_RULE_ASSETS_SO_TYPE } from './prebuilt_rule_assets_type';
1920
import type { RuleVersionSpecifier } from '../rule_versions/rule_version_specifier';
2021

22+
const RULE_ASSET_ATTRIBUTES = `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes`;
2123
const MAX_PREBUILT_RULES_COUNT = 10_000;
24+
const ES_MAX_CLAUSE_COUNT = 1024;
25+
const ES_MAX_CONCURRENT_REQUESTS = 2;
2226

2327
export interface IPrebuiltRuleAssetsClient {
2428
fetchLatestAssets: () => Promise<PrebuiltRuleAsset[]>;
@@ -77,50 +81,64 @@ export const createPrebuiltRuleAssetsClient = (
7781

7882
fetchLatestVersions: (ruleIds: string[] = []): Promise<RuleVersionSpecifier[]> => {
7983
return withSecuritySpan('IPrebuiltRuleAssetsClient.fetchLatestVersions', async () => {
80-
const filter = ruleIds
81-
.map((ruleId) => `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes.rule_id: ${ruleId}`)
82-
.join(' OR ');
84+
if (ruleIds && ruleIds.length === 0) {
85+
return [];
86+
}
8387

84-
const findResult = await savedObjectsClient.find<
85-
PrebuiltRuleAsset,
86-
{
87-
rules: AggregationsMultiBucketAggregateBase<{
88-
latest_version: AggregationsTopHitsAggregate;
89-
}>;
90-
}
91-
>({
92-
type: PREBUILT_RULE_ASSETS_SO_TYPE,
93-
filter,
94-
aggs: {
95-
rules: {
96-
terms: {
97-
field: `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes.rule_id`,
98-
size: MAX_PREBUILT_RULES_COUNT,
99-
},
100-
aggs: {
101-
latest_version: {
102-
top_hits: {
103-
size: 1,
104-
sort: [
105-
{
106-
[`${PREBUILT_RULE_ASSETS_SO_TYPE}.version`]: 'desc',
107-
},
108-
],
109-
_source: [
110-
`${PREBUILT_RULE_ASSETS_SO_TYPE}.rule_id`,
111-
`${PREBUILT_RULE_ASSETS_SO_TYPE}.version`,
112-
],
88+
const fetchLatestVersionInfo = async (filter?: string) => {
89+
const findResult = await savedObjectsClient.find<
90+
PrebuiltRuleAsset,
91+
{
92+
rules: AggregationsMultiBucketAggregateBase<{
93+
latest_version: AggregationsTopHitsAggregate;
94+
}>;
95+
}
96+
>({
97+
type: PREBUILT_RULE_ASSETS_SO_TYPE,
98+
filter,
99+
aggs: {
100+
rules: {
101+
terms: {
102+
field: `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes.rule_id`,
103+
size: MAX_PREBUILT_RULES_COUNT,
104+
},
105+
aggs: {
106+
latest_version: {
107+
top_hits: {
108+
size: 1,
109+
sort: [
110+
{
111+
[`${PREBUILT_RULE_ASSETS_SO_TYPE}.version`]: 'desc',
112+
},
113+
],
114+
_source: [
115+
`${PREBUILT_RULE_ASSETS_SO_TYPE}.rule_id`,
116+
`${PREBUILT_RULE_ASSETS_SO_TYPE}.version`,
117+
],
118+
},
113119
},
114120
},
115121
},
116122
},
117-
},
118-
});
123+
});
119124

120-
const buckets = findResult.aggregations?.rules?.buckets ?? [];
121-
invariant(Array.isArray(buckets), 'Expected buckets to be an array');
125+
const aggregatedBuckets = findResult.aggregations?.rules?.buckets ?? [];
126+
invariant(Array.isArray(aggregatedBuckets), 'Expected buckets to be an array');
127+
128+
return aggregatedBuckets;
129+
};
130+
131+
const filters = ruleIds
132+
? createChunkedFilters({
133+
items: ruleIds,
134+
mapperFn: (ruleId) => `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes.rule_id: ${ruleId}`,
135+
clausesPerItem: 2,
136+
})
137+
: undefined;
122138

123-
return buckets.map((bucket) => {
139+
const buckets = await chunkedFetch(fetchLatestVersionInfo, filters);
140+
141+
const latestVersions = buckets.map((bucket) => {
124142
const hit = bucket.latest_version.hits.hits[0];
125143
const soAttributes = hit._source[PREBUILT_RULE_ASSETS_SO_TYPE];
126144
const versionInfo: RuleVersionSpecifier = {
@@ -129,6 +147,8 @@ export const createPrebuiltRuleAssetsClient = (
129147
};
130148
return versionInfo;
131149
});
150+
151+
return latestVersions;
132152
});
133153
},
134154

@@ -139,21 +159,26 @@ export const createPrebuiltRuleAssetsClient = (
139159
return [];
140160
}
141161

142-
const attr = `${PREBUILT_RULE_ASSETS_SO_TYPE}.attributes`;
143-
const filter = versions
144-
.map((v) => `(${attr}.rule_id: ${v.rule_id} AND ${attr}.version: ${v.version})`)
145-
.join(' OR ');
146-
147-
// Usage of savedObjectsClient.bulkGet() is ~25% more performant and
148-
// simplifies deduplication but too many tests get broken.
149-
// See https://github.com/elastic/kibana/issues/218198
150-
const findResult = await savedObjectsClient.find<PrebuiltRuleAsset>({
151-
type: PREBUILT_RULE_ASSETS_SO_TYPE,
152-
filter,
153-
perPage: MAX_PREBUILT_RULES_COUNT,
162+
const filters = createChunkedFilters({
163+
items: versions,
164+
mapperFn: (versionSpecifier) =>
165+
`(${RULE_ASSET_ATTRIBUTES}.rule_id: ${versionSpecifier.rule_id} AND ${RULE_ASSET_ATTRIBUTES}.version: ${versionSpecifier.version})`,
166+
clausesPerItem: 4,
154167
});
155168

156-
const ruleAssets = findResult.saved_objects.map((so) => so.attributes);
169+
const ruleAssets = await chunkedFetch(async (filter) => {
170+
// Usage of savedObjectsClient.bulkGet() is ~25% more performant and
171+
// simplifies deduplication but too many tests get broken.
172+
// See https://github.com/elastic/kibana/issues/218198
173+
const findResult = await savedObjectsClient.find<PrebuiltRuleAsset>({
174+
type: PREBUILT_RULE_ASSETS_SO_TYPE,
175+
filter,
176+
perPage: MAX_PREBUILT_RULES_COUNT,
177+
});
178+
179+
return findResult.saved_objects.map((so) => so.attributes);
180+
}, filters);
181+
157182
// Rule assets may have duplicates we have to get rid of.
158183
// In particular prebuilt rule assets package v8.17.1 has duplicates.
159184
const uniqueRuleAssets = uniqBy(ruleAssets, 'rule_id');
@@ -163,3 +188,50 @@ export const createPrebuiltRuleAssetsClient = (
163188
},
164189
};
165190
};
191+
192+
/**
193+
* Creates an array of KQL filter strings for a collection of items.
194+
* Uses chunking to ensure that the number of filter clauses does not exceed the ES "too_many_clauses" limit.
195+
* See: https://github.com/elastic/kibana/pull/223240
196+
*
197+
* @param {object} options
198+
* @param {T[]} options.items - Array of items to create filters for.
199+
* @param {(item: T) => string} options.mapperFn - A function that maps an item to a filter string.
200+
* @param {number} options.clausesPerItem - Number of Elasticsearch clauses generated per item. Determined empirically by converting a KQL filter into a Query DSL query.
201+
* More complex filters will result in more clauses. Info about clauses in docs: https://www.elastic.co/docs/explore-analyze/query-filter/languages/querydsl#query-dsl
202+
* @returns {string[]} An array of filter strings
203+
*/
204+
function createChunkedFilters<T>({
205+
items,
206+
mapperFn,
207+
clausesPerItem,
208+
}: {
209+
items: T[];
210+
mapperFn: (item: T) => string;
211+
clausesPerItem: number;
212+
}): string[] {
213+
return chunk(items, ES_MAX_CLAUSE_COUNT / clausesPerItem).map((singleChunk) =>
214+
singleChunk.map(mapperFn).join(' OR ')
215+
);
216+
}
217+
218+
/**
219+
* Fetches objects using a provided function.
220+
* If filters are provided fetches concurrently in chunks.
221+
*
222+
* @param {(filter?: string) => Promise<T[]>} chunkFetchFn - Function that fetches a chunk.
223+
* @param {string[]} [filters] - An optional array of filter strings. If provided, `chunkFetchFn` will be called for each filter concurrently.
224+
* @returns {Promise<T[]>} A promise that resolves to an array of fetched objects.
225+
*/
226+
function chunkedFetch<T>(
227+
chunkFetchFn: (filter?: string) => Promise<T[]>,
228+
filters?: string[]
229+
): Promise<T[]> {
230+
if (filters?.length) {
231+
return pMap(filters, chunkFetchFn, {
232+
concurrency: ES_MAX_CONCURRENT_REQUESTS,
233+
}).then((results) => results.flat());
234+
}
235+
236+
return chunkFetchFn();
237+
}

0 commit comments

Comments
 (0)