Skip to content

Commit e566fec

Browse files
adcoelhokibanamachineelasticmachinecnasikas
authored
[ResponseOps] Cases analytics index (#223405)
This PR is for a feature branch that is being merged into main. The relevant PRs are: - #219211 - #222820 - #223241 - #224388 - #224682 ## Summary This PR adds 4 new indexes with case analytics data, which are created when the cases plugin starts. - `.internal.cases` - `.internal.cases-comments` - `.internal.cases-attachments` - `.internal.cases-activity` After the indexes are created, a backfill task for each of them is scheduled to run 1 minute after creation. This task populates the indexes with relevant data from `.kibana_alerting_cases`. A second type of task is registered, the index synchronization task. Four of these tasks, one for each index, are scheduled to run every 5 minutes. The synchronization tasks populated the indexes with data from `.kibana_alerting_cases` that was created or updated in the last five minutes. ## How to test You might want to start Kibana with `--verbose` to see relevant index messages in the console. Alternatively(what I normally do), is go to `analytics_index.ts`, `backfill_task_runner.ts`, and `synchronization_task_runner.ts`, and change the `logDebug` function to call `this.logger.info` instead. This way, you will have less spam in the console. Every log message starts with the index name between square brackets, so you can look for `[.internal.cases-` and follow what is happening. 1. You should have some existing case data, so before anything else, please create some activity, attachments, etc. 2. Add `xpack.cases.analytics.index.enabled: true` to `kibana.dev.yml` and restart Kibana. 3. Check out [this branch](elastic/elasticsearch#129414) from the ES project. 4. Start Elastic Search with `yarn es source`. This will use the above version of Elasticsearch. 5. Wait a bit for the indexes to be created and populated(backfilled). 6. Using the dev tools: - Confirm the indexes exist. - Check the index mapping. Does it match the one in the code? Is the `_meta` field correct? - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/mappings.ts` - Check that the painless scripts match the code. - `x-pack/platform/plugins/shared/cases/server/cases_analytics/******_index/painless_scripts.ts` - Confirm your existing case data is in the indexes. (See **Queries** section below.) 7. Play around with cases. Some examples: - Create a case - Change status/severity - Attach alerts - Add files - Change category/tags - Add comments - etc 8. Go to the dev tools again and confirm all this shows up in the relevant indexes. (See **Queries** section below.) ## Queries In addition to the ones, below I have a few more. Things like reindexing with specific scripts or fetching relevant data from `.kibana_alerting_cases`. Ping me if you want those queries. ### Checking index content ``` GET /.internal.cases/_search GET /.internal.cases-comments/_search GET /.internal.cases-attachments/_search GET /.internal.cases-activity/_search ``` ### Checking index mappings ``` GET /.internal.cases GET /.internal.cases-comments GET /.internal.cases-attachments GET /.internal.cases-activity ``` ### Fetching the painless scripts ``` GET /_scripts/cai_cases_script_1 GET /_scripts/cai_attachments_script_1 GET /_scripts/cai_comments_script_1 GET /_scripts/cai_activity_script_1 ``` ### Emptying the indexes It is sometimes useful for testing. ``` POST /.internal.cases/_delete_by_query POST /.internal.cases-comments/_delete_by_query POST /.internal.cases-attachments/_delete_by_query POST /.internal.cases-activity/_delete_by_query ``` ### Deleting the indexes It is sometimes useful for testing. ``` DELETE /.internal.cases DELETE /.internal.cases-comments DELETE /.internal.cases-attachments DELETE /.internal.cases-activity ``` ## Release notes Four dedicated case analytics indexes were created, allowing users to build dashboards and metrics over case data. These indexes are created on Kibana startup and updated periodically with cases, comments, attachments, and activity data. --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Christos Nasikas <[email protected]>
1 parent c989634 commit e566fec

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+3809
-66
lines changed

‎x-pack/platform/plugins/shared/cases/server/cases_analytics/activity_index/constants.ts‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import type { QueryDslQueryContainer } from '@elastic/elasticsearch/lib/api/types';
9+
10+
import { ALERTING_CASES_SAVED_OBJECT_INDEX } from '@kbn/core-saved-objects-server';
11+
12+
export const CAI_ACTIVITY_INDEX_NAME = '.internal.cases-activity';
13+
14+
export const CAI_ACTIVITY_INDEX_ALIAS = '.cases-activity';
15+
16+
export const CAI_ACTIVITY_INDEX_VERSION = 1;
17+
18+
export const CAI_ACTIVITY_SOURCE_QUERY: QueryDslQueryContainer = {
19+
bool: {
20+
must: [
21+
{
22+
term: {
23+
type: 'cases-user-actions',
24+
},
25+
},
26+
{
27+
bool: {
28+
should: [
29+
{
30+
term: {
31+
'cases-user-actions.type': 'severity',
32+
},
33+
},
34+
{
35+
term: {
36+
'cases-user-actions.type': 'delete_case',
37+
},
38+
},
39+
{
40+
term: {
41+
'cases-user-actions.type': 'category',
42+
},
43+
},
44+
{
45+
term: {
46+
'cases-user-actions.type': 'status',
47+
},
48+
},
49+
{
50+
term: {
51+
'cases-user-actions.type': 'tags',
52+
},
53+
},
54+
],
55+
minimum_should_match: 1,
56+
},
57+
},
58+
],
59+
},
60+
};
61+
62+
export const CAI_ACTIVITY_SOURCE_INDEX = ALERTING_CASES_SAVED_OBJECT_INDEX;
63+
64+
export const CAI_ACTIVITY_BACKFILL_TASK_ID = 'cai_activity_backfill_task';
65+
66+
export const CAI_ACTIVITY_SYNCHRONIZATION_TASK_ID = 'cai_cases_activity_synchronization_task';
67+
68+
export const getActivitySynchronizationSourceQuery = (
69+
lastSyncAt: Date
70+
): QueryDslQueryContainer => ({
71+
bool: {
72+
must: [
73+
{
74+
term: {
75+
type: 'cases-user-actions',
76+
},
77+
},
78+
{
79+
range: {
80+
'cases-user-actions.created_at': {
81+
gte: lastSyncAt.toISOString(),
82+
},
83+
},
84+
},
85+
{
86+
bool: {
87+
should: [
88+
{
89+
term: {
90+
'cases-user-actions.type': 'severity',
91+
},
92+
},
93+
{
94+
term: {
95+
'cases-user-actions.type': 'delete_case',
96+
},
97+
},
98+
{
99+
term: {
100+
'cases-user-actions.type': 'category',
101+
},
102+
},
103+
{
104+
term: {
105+
'cases-user-actions.type': 'status',
106+
},
107+
},
108+
{
109+
term: {
110+
'cases-user-actions.type': 'tags',
111+
},
112+
},
113+
],
114+
minimum_should_match: 1,
115+
},
116+
},
117+
],
118+
},
119+
});

‎x-pack/platform/plugins/shared/cases/server/cases_analytics/activity_index/index.ts‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import type { ElasticsearchClient, Logger } from '@kbn/core/server';
9+
import type { TaskManagerStartContract } from '@kbn/task-manager-plugin/server';
10+
import { AnalyticsIndex } from '../analytics_index';
11+
import {
12+
CAI_ACTIVITY_INDEX_NAME,
13+
CAI_ACTIVITY_INDEX_ALIAS,
14+
CAI_ACTIVITY_INDEX_VERSION,
15+
CAI_ACTIVITY_SOURCE_INDEX,
16+
CAI_ACTIVITY_SOURCE_QUERY,
17+
CAI_ACTIVITY_BACKFILL_TASK_ID,
18+
CAI_ACTIVITY_SYNCHRONIZATION_TASK_ID,
19+
} from './constants';
20+
import { CAI_ACTIVITY_INDEX_MAPPINGS } from './mappings';
21+
import { CAI_ACTIVITY_INDEX_SCRIPT, CAI_ACTIVITY_INDEX_SCRIPT_ID } from './painless_scripts';
22+
import { scheduleCAISynchronizationTask } from '../tasks/synchronization_task';
23+
24+
export const createActivityAnalyticsIndex = ({
25+
esClient,
26+
logger,
27+
isServerless,
28+
taskManager,
29+
}: {
30+
esClient: ElasticsearchClient;
31+
logger: Logger;
32+
isServerless: boolean;
33+
taskManager: TaskManagerStartContract;
34+
}): AnalyticsIndex =>
35+
new AnalyticsIndex({
36+
logger,
37+
esClient,
38+
isServerless,
39+
taskManager,
40+
indexName: CAI_ACTIVITY_INDEX_NAME,
41+
indexAlias: CAI_ACTIVITY_INDEX_ALIAS,
42+
indexVersion: CAI_ACTIVITY_INDEX_VERSION,
43+
mappings: CAI_ACTIVITY_INDEX_MAPPINGS,
44+
painlessScriptId: CAI_ACTIVITY_INDEX_SCRIPT_ID,
45+
painlessScript: CAI_ACTIVITY_INDEX_SCRIPT,
46+
taskId: CAI_ACTIVITY_BACKFILL_TASK_ID,
47+
sourceIndex: CAI_ACTIVITY_SOURCE_INDEX,
48+
sourceQuery: CAI_ACTIVITY_SOURCE_QUERY,
49+
});
50+
51+
export const scheduleActivityAnalyticsSyncTask = ({
52+
taskManager,
53+
logger,
54+
}: {
55+
taskManager: TaskManagerStartContract;
56+
logger: Logger;
57+
}) => {
58+
scheduleCAISynchronizationTask({
59+
taskId: CAI_ACTIVITY_SYNCHRONIZATION_TASK_ID,
60+
sourceIndex: CAI_ACTIVITY_SOURCE_INDEX,
61+
destIndex: CAI_ACTIVITY_INDEX_NAME,
62+
taskManager,
63+
logger,
64+
}).catch((e) => {
65+
logger.error(
66+
`Error scheduling ${CAI_ACTIVITY_SYNCHRONIZATION_TASK_ID} task, received ${e.message}`
67+
);
68+
});
69+
};

‎x-pack/platform/plugins/shared/cases/server/cases_analytics/activity_index/mappings.ts‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import type { MappingTypeMapping } from '@elastic/elasticsearch/lib/api/types';
9+
10+
export const CAI_ACTIVITY_INDEX_MAPPINGS: MappingTypeMapping = {
11+
dynamic: false,
12+
properties: {
13+
'@timestamp': {
14+
type: 'date',
15+
},
16+
case_id: {
17+
type: 'keyword',
18+
},
19+
action: {
20+
type: 'keyword',
21+
},
22+
type: {
23+
type: 'keyword',
24+
},
25+
payload: {
26+
properties: {
27+
status: {
28+
type: 'keyword',
29+
},
30+
tags: {
31+
type: 'keyword',
32+
},
33+
category: {
34+
type: 'keyword',
35+
},
36+
severity: {
37+
type: 'keyword',
38+
},
39+
},
40+
},
41+
created_at: {
42+
type: 'date',
43+
},
44+
created_at_ms: {
45+
type: 'long',
46+
},
47+
created_by: {
48+
properties: {
49+
username: {
50+
type: 'keyword',
51+
},
52+
profile_uid: {
53+
type: 'keyword',
54+
},
55+
full_name: {
56+
type: 'keyword',
57+
},
58+
email: {
59+
type: 'keyword',
60+
},
61+
},
62+
},
63+
owner: {
64+
type: 'keyword',
65+
},
66+
space_ids: {
67+
type: 'keyword',
68+
},
69+
},
70+
};

‎x-pack/platform/plugins/shared/cases/server/cases_analytics/activity_index/painless_scripts.ts‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License
4+
* 2.0; you may not use this file except in compliance with the Elastic License
5+
* 2.0.
6+
*/
7+
8+
import type { StoredScript } from '@elastic/elasticsearch/lib/api/types';
9+
import { CAI_ACTIVITY_INDEX_VERSION } from './constants';
10+
11+
export const CAI_ACTIVITY_INDEX_SCRIPT_ID = `cai_activity_script_${CAI_ACTIVITY_INDEX_VERSION}`;
12+
export const CAI_ACTIVITY_INDEX_SCRIPT: StoredScript = {
13+
lang: 'painless',
14+
source: `
15+
def source = [:];
16+
source.putAll(ctx._source);
17+
ctx._source.clear();
18+
19+
ctx._source.action = source["cases-user-actions"].action;
20+
ctx._source.type = source["cases-user-actions"].type;
21+
22+
long milliSinceEpoch = new Date().getTime();
23+
Instant instant = Instant.ofEpochMilli(milliSinceEpoch);
24+
ctx._source['@timestamp'] = ZonedDateTime.ofInstant(instant, ZoneId.of('Z'));
25+
26+
ZonedDateTime zdt_created =
27+
ZonedDateTime.parse(source["cases-user-actions"].created_at);
28+
ctx._source.created_at_ms = zdt_created.toInstant().toEpochMilli();
29+
ctx._source.created_at = source["cases-user-actions"].created_at;
30+
31+
if (source["cases-user-actions"].created_by != null) {
32+
ctx._source.created_by = new HashMap();
33+
ctx._source.created_by.full_name = source["cases-user-actions"].created_by.full_name;
34+
ctx._source.created_by.username = source["cases-user-actions"].created_by.username;
35+
ctx._source.created_by.profile_uid = source["cases-user-actions"].created_by.profile_uid;
36+
ctx._source.created_by.email = source["cases-user-actions"].created_by.email;
37+
}
38+
39+
if (source["cases-user-actions"].payload != null) {
40+
ctx._source.payload = new HashMap();
41+
42+
if (source["cases-user-actions"].type == "severity" && source["cases-user-actions"].payload.severity != null) {
43+
ctx._source.payload.severity = source["cases-user-actions"].payload.severity;
44+
}
45+
46+
if (source["cases-user-actions"].type == "category" && source["cases-user-actions"].payload.category != null) {
47+
ctx._source.payload.category = source["cases-user-actions"].payload.category;
48+
}
49+
50+
if (source["cases-user-actions"].type == "status" && source["cases-user-actions"].payload.status != null) {
51+
ctx._source.payload.status = source["cases-user-actions"].payload.status;
52+
}
53+
54+
if (source["cases-user-actions"].type == "tags" && source["cases-user-actions"].payload.tags != null) {
55+
ctx._source.payload.tags = source["cases-user-actions"].payload.tags;
56+
}
57+
}
58+
59+
if (source.references != null) {
60+
for (item in source.references) {
61+
if (item.type == "cases") {
62+
ctx._source.case_id = item.id;
63+
}
64+
}
65+
}
66+
67+
ctx._source.owner = source["cases-user-actions"].owner;
68+
ctx._source.space_ids = source.namespaces;
69+
`,
70+
};

0 commit comments

Comments
 (0)