Skip to content

Commit fa8ef65

Browse files
adcoelhokibanamachine
adcoelho
and
kibanamachine
authored
[ResponseOps] Cases analytics index creation and backfill (#219211)
Closes [#221234](#221234) **Merging into a feature branch** ## Summary This PR adds: - The analytics index creation when the cases plugin starts. - Backfill task registration. - Backfill task scheduling. - Mapping version 1 for: - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - All corresponding painless scripts. ## How to review 1. The main functionality is in `analytics_index.ts`. The class in this file implements the index creation algorithm of the RFC. 2. Three folders instantiate this class to create the indexes we need: - `cases_analytics/cases_index/` - `cases_analytics/comments_index/` - `cases_analytics/attachments_index/` 3. The backfill logic can be found in `cases_analytics/tasks/backfill_task`. 4. I moved the `retry_service` previously used by the cases connector to a `common` folder and started using it in the `AnalyticsIndex` class. 5. The call to create the indexes is done in `x-pack/platform/plugins/shared/cases/server/plugin.ts`. ## How to test A few things are needed. 1. Create a role that has index creation permission. Below is what works for me; it might be overkill. <img width="1030" alt="Screenshot 2025-04-25 at 11 30 45" src="https://github.com/user-attachments/assets/cbb7b384-e438-43ec-bbca-a18e62243523" /> 2. Create a user with the role created in step 1. Let's call the user `index_creator`. **The password should be** `changeme`. <img width="842" alt="Screenshot 2025-04-25 at 11 31 27" src="https://github.com/user-attachments/assets/a9547c96-086e-43b4-a442-a18f558dcb96" /> 6. Start Kibana with this user as the Elasticsearch user - `yarn start` with the option `--elasticsearch.username=index_creator`. 7. Kibana startup logs should show relevant messages: ``` [INFO ][plugins.cases] [.internal.cases] Checking if index exists. [INFO ][plugins.cases] [.internal.cases-attachments] Checking if index exists. [INFO ][plugins.cases] [.internal.cases-comments] Checking if index exists. ``` 8. You can check the content of these indexes in the Dev Tools. ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search ``` 9. You can check the mapping of each of these indexes in the Dev Tools. ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments ``` 10. You can check the painless scripts in the Dev Tools. ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 ``` --------- Co-authored-by: kibanamachine <[email protected]>
1 parent 491c669 commit fa8ef65

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+2300
-65
lines changed

‎x-pack/platform/plugins/shared/cases/server/cases_analytics/analytics_index.test.ts‎

Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import { elasticsearchServiceMock, loggingSystemMock } from '@kbn/core/server/mocks';
9+
import { taskManagerMock } from '@kbn/task-manager-plugin/server/mocks';
10+
import { errors as esErrors } from '@elastic/elasticsearch';
11+
12+
import { AnalyticsIndex } from './analytics_index';
13+
import type {
14+
IndicesCreateResponse,
15+
IndicesPutMappingResponse,
16+
MappingTypeMapping,
17+
QueryDslQueryContainer,
18+
StoredScript,
19+
} from '@elastic/elasticsearch/lib/api/types';
20+
import { fullJitterBackoffFactory } from '../common/retry_service/full_jitter_backoff';
21+
import { scheduleCAIBackfillTask } from './tasks/backfill_task';
22+
23+
jest.mock('../common/retry_service/full_jitter_backoff');
24+
jest.mock('./tasks/backfill_task');
25+
26+
const fullJitterBackoffFactoryMock = fullJitterBackoffFactory as jest.Mock;
27+
const scheduleCAIBackfillTaskMock = scheduleCAIBackfillTask as jest.Mock;
28+
29+
describe('AnalyticsIndex', () => {
30+
const logger = loggingSystemMock.createLogger();
31+
const esClient = elasticsearchServiceMock.createElasticsearchClient();
32+
const taskManager = taskManagerMock.createStart();
33+
const isServerless = false;
34+
const indexName = '.test-index-name';
35+
const indexVersion = 1;
36+
const painlessScriptId = 'painless_script_id';
37+
const taskId = 'foobar_task_id';
38+
const sourceIndex = '.source-index';
39+
40+
const painlessScript: StoredScript = {
41+
lang: 'painless',
42+
source: 'ctx._source.remove("foobar");',
43+
};
44+
const mappings: MappingTypeMapping = {
45+
dynamic: false,
46+
properties: {
47+
title: {
48+
type: 'keyword',
49+
},
50+
},
51+
};
52+
const mappingsMeta = {
53+
mapping_version: indexVersion,
54+
painless_script_id: painlessScriptId,
55+
};
56+
const sourceQuery: QueryDslQueryContainer = {
57+
term: {
58+
type: 'cases',
59+
},
60+
};
61+
62+
let index: AnalyticsIndex;
63+
64+
// 1ms delay before retrying
65+
const nextBackOff = jest.fn().mockReturnValue(1);
66+
67+
const backOffFactory = {
68+
create: () => ({ nextBackOff }),
69+
};
70+
71+
beforeEach(() => {
72+
jest.clearAllMocks();
73+
74+
fullJitterBackoffFactoryMock.mockReturnValue(backOffFactory);
75+
76+
index = new AnalyticsIndex({
77+
esClient,
78+
logger,
79+
indexName,
80+
indexVersion,
81+
isServerless,
82+
mappings,
83+
painlessScript,
84+
painlessScriptId,
85+
sourceIndex,
86+
sourceQuery,
87+
taskId,
88+
taskManager,
89+
});
90+
});
91+
92+
it('checks if the index exists', async () => {
93+
await index.upsertIndex();
94+
95+
expect(esClient.indices.exists).toBeCalledWith({ index: indexName });
96+
});
97+
98+
it('creates index if it does not exist', async () => {
99+
esClient.indices.exists.mockResolvedValueOnce(false);
100+
101+
await index.upsertIndex();
102+
103+
expect(esClient.indices.exists).toBeCalledWith({ index: indexName });
104+
expect(esClient.putScript).toBeCalledWith({ id: painlessScriptId, script: painlessScript });
105+
expect(esClient.indices.create).toBeCalledWith({
106+
index: indexName,
107+
timeout: '300s',
108+
mappings: {
109+
...mappings,
110+
_meta: mappingsMeta,
111+
},
112+
settings: {
113+
index: {
114+
auto_expand_replicas: '0-1',
115+
mode: 'lookup',
116+
number_of_shards: 1,
117+
refresh_interval: '15s',
118+
},
119+
},
120+
});
121+
expect(scheduleCAIBackfillTaskMock).toHaveBeenCalledWith({
122+
taskId,
123+
sourceIndex,
124+
sourceQuery,
125+
destIndex: indexName,
126+
taskManager,
127+
logger,
128+
});
129+
});
130+
131+
it('updates index if it exists and the mapping has a lower version number', async () => {
132+
esClient.indices.exists.mockResolvedValueOnce(true);
133+
esClient.indices.getMapping.mockResolvedValueOnce({
134+
[indexName]: {
135+
mappings: {
136+
_meta: {
137+
mapping_version: 0, // lower version number
138+
painless_script_id: painlessScriptId,
139+
},
140+
dynamic: mappings.dynamic,
141+
properties: mappings.properties,
142+
},
143+
},
144+
});
145+
146+
await index.upsertIndex();
147+
148+
expect(esClient.indices.exists).toBeCalledWith({ index: indexName });
149+
expect(esClient.indices.getMapping).toBeCalledWith({ index: indexName });
150+
expect(esClient.putScript).toBeCalledWith({ id: painlessScriptId, script: painlessScript });
151+
expect(esClient.indices.putMapping).toBeCalledWith({
152+
index: indexName,
153+
...mappings,
154+
_meta: mappingsMeta,
155+
});
156+
expect(scheduleCAIBackfillTaskMock).toBeCalledWith({
157+
taskId,
158+
sourceIndex,
159+
sourceQuery,
160+
destIndex: indexName,
161+
taskManager,
162+
logger,
163+
});
164+
});
165+
166+
it('does not update index if it exists and the mapping has a higher version number', async () => {
167+
esClient.indices.exists.mockResolvedValueOnce(true);
168+
esClient.indices.getMapping.mockResolvedValueOnce({
169+
[indexName]: {
170+
mappings: {
171+
_meta: {
172+
mapping_version: 10, // higher version number
173+
painless_script_id: painlessScriptId,
174+
},
175+
dynamic: mappings.dynamic,
176+
properties: mappings.properties,
177+
},
178+
},
179+
});
180+
181+
await index.upsertIndex();
182+
183+
expect(esClient.indices.exists).toBeCalledWith({ index: indexName });
184+
expect(esClient.indices.getMapping).toBeCalledWith({ index: indexName });
185+
expect(esClient.putScript).toBeCalledTimes(0);
186+
expect(esClient.indices.putMapping).toBeCalledTimes(0);
187+
expect(scheduleCAIBackfillTaskMock).toBeCalledTimes(0);
188+
189+
expect(logger.debug).toBeCalledWith(
190+
`[${indexName}] Mapping version is up to date. Skipping update.`
191+
);
192+
});
193+
194+
it('does not update index if it exists and the mapping has the same version number', async () => {
195+
esClient.indices.exists.mockResolvedValueOnce(true);
196+
esClient.indices.getMapping.mockResolvedValueOnce({ [indexName]: { mappings } });
197+
198+
await index.upsertIndex();
199+
200+
expect(esClient.indices.exists).toBeCalledWith({ index: indexName });
201+
expect(esClient.indices.getMapping).toBeCalledWith({ index: indexName });
202+
expect(esClient.putScript).toBeCalledTimes(0);
203+
expect(esClient.indices.putMapping).toBeCalledTimes(0);
204+
expect(scheduleCAIBackfillTaskMock).toBeCalledTimes(0);
205+
206+
expect(logger.debug).toBeCalledWith(
207+
`[${indexName}] Mapping version is up to date. Skipping update.`
208+
);
209+
});
210+
211+
describe('Error handling', () => {
212+
it('retries if the esClient throws a retryable error', async () => {
213+
esClient.indices.exists
214+
.mockRejectedValueOnce(new esErrors.ConnectionError('My retryable error A'))
215+
.mockRejectedValueOnce(new esErrors.TimeoutError('My retryable error B'))
216+
.mockResolvedValue(true);
217+
await index.upsertIndex();
218+
219+
expect(nextBackOff).toBeCalledTimes(2);
220+
expect(esClient.indices.exists).toBeCalledTimes(3);
221+
expect(esClient.indices.exists).toBeCalledWith({ index: indexName });
222+
});
223+
224+
it('retries if the esClient throws a retryable error when creating an index', async () => {
225+
esClient.indices.exists.mockResolvedValue(false);
226+
esClient.indices.create
227+
.mockRejectedValueOnce(new esErrors.ConnectionError('My retryable error A'))
228+
.mockResolvedValue({} as IndicesCreateResponse);
229+
230+
await index.upsertIndex();
231+
232+
expect(nextBackOff).toBeCalledTimes(1);
233+
expect(esClient.indices.exists).toBeCalledWith({ index: indexName });
234+
expect(esClient.putScript).toBeCalledWith({ id: painlessScriptId, script: painlessScript });
235+
expect(esClient.indices.create).toBeCalledTimes(2);
236+
expect(scheduleCAIBackfillTaskMock).toHaveBeenCalledWith({
237+
taskId,
238+
sourceIndex,
239+
sourceQuery,
240+
destIndex: indexName,
241+
taskManager,
242+
logger,
243+
});
244+
});
245+
246+
it('retries if the esClient throws a retryable error when updating an index', async () => {
247+
esClient.indices.exists.mockResolvedValue(true);
248+
esClient.indices.getMapping.mockResolvedValue({
249+
[indexName]: {
250+
mappings: {
251+
_meta: {
252+
mapping_version: 0, // lower version number
253+
painless_script_id: painlessScriptId,
254+
},
255+
dynamic: mappings.dynamic,
256+
properties: mappings.properties,
257+
},
258+
},
259+
});
260+
261+
esClient.indices.putMapping
262+
.mockRejectedValueOnce(new esErrors.ConnectionError('My retryable error A'))
263+
.mockResolvedValue({} as IndicesPutMappingResponse);
264+
265+
await index.upsertIndex();
266+
267+
expect(nextBackOff).toBeCalledTimes(1);
268+
expect(esClient.indices.exists).toBeCalledWith({ index: indexName });
269+
expect(esClient.indices.getMapping).toBeCalledWith({ index: indexName });
270+
expect(esClient.putScript).toBeCalledWith({ id: painlessScriptId, script: painlessScript });
271+
272+
expect(esClient.indices.putMapping).toBeCalledTimes(2);
273+
expect(esClient.indices.putMapping).toBeCalledWith({
274+
index: indexName,
275+
...mappings,
276+
_meta: mappingsMeta,
277+
});
278+
279+
expect(scheduleCAIBackfillTaskMock).toBeCalledWith({
280+
taskId,
281+
sourceIndex,
282+
sourceQuery,
283+
destIndex: indexName,
284+
taskManager,
285+
logger,
286+
});
287+
});
288+
289+
it('does not retry if the eexecution throws a non-retryable error', async () => {
290+
esClient.indices.exists.mockRejectedValue(new Error('My terrible error'));
291+
292+
await expect(index.upsertIndex()).resolves.not.toThrow();
293+
294+
expect(nextBackOff).toBeCalledTimes(0);
295+
// Paths in the algorithm after the error are not called.
296+
expect(esClient.indices.getMapping).not.toHaveBeenCalled();
297+
});
298+
});
299+
});

0 commit comments

Comments
 (0)