Skip to content

Commit 748e3a9

Browse files
Supplementingkibanamachineelasticmachine
authored and
shahargl
committed
[Fleet] Knowledge base integration support (elastic#230107)
## Summary Closes elastic/ingest-dev#5678 - Adds support for writing the knowledge_base docs in a package to a system index `.integration_knowledge` with per-document indexing. - Also checks that the license is appropriate (`enterprise`), otherwise skips this step as usage of inference models is needed due to the `semantic_text` field. - Automatically removes knowledge base files from the index when the package is removed - Updates the docs when the package updates - Also adds an internal endpoint for getting indexed knowledge base docs for a specific package `GET /internal/fleet/epm/packages/{pkgName}/knowledge_base` - In order to improve UX, we also added a generic doc during fleet setup with some basic fleet knowledge that will be indexed asynchronously (if the user has the correct license of `enterprise`) so that later on when the docs need indexed, we dont need to wait for inference model deployment. See [this comment](elastic#230107 (comment)) for more context. _**As part of these changes, I also realized there was an issue with the custom integrations upload feature where it would error due to a missing integration name as it relied on the pipeline, so I added a step to use the name from the `_meta` field (with fallback to the pipeline) in order to stop issues during manual testing. Can remove before finalized if needed cc: @elastic/security-scalability. If kept, will close elastic#231712 . https://github.com/user-attachments/assets/0a7559af-b312-4356-bdb1-e05531721f89 **NOTE**: In the video, the indexing fails the first time. Turns out, it was due to a timeout as the index was being created by ES and the indexing would fail as creation took too long. It is now wrapped in a retry so this has been resolved. ## Testing instructions Cloud Deployment for testing without pulling everything down to local: https://supplementing-pr-230107-knowledge-base-integration-support.kbndev.co/ Manual Testing: 1. If elastic/elasticsearch#132506 has been merged, run `yarn es snapshot` in Kibana, otherwise, checkout that branch in your local ES and then in kibana run `yarn es source` in order to use that version of ES which contains the index management, mappings, etc. 2. Install a package with any number of knowledge base docs in the `docs/knowledge_base` folder. You can use [this sample package](https://github.com/user-attachments/files/21867395/masonstestpackage-0.0.1.zip), or create your own following the guide below: - Using `elastic-package`, create a new package using `elastic-package create integration` - Once created add `knowledge_base` as a folder inside of the generated `docs` folder of the integration - Add an arbitrary amount of `.md` files to the knowledge_base folder - Run `elastic-package build` to build the package - There are a lot of different options for installing the package in a local kibana instance. I prefer to just take the generated .zip folder from `/build` in `elastic-package` and upload it to kibana using the custom integrations feature. You can also expose the package registry, or whatever you see fit. 3. Watch the Kibana logs for errors/debug messages etc 4. Use the new endpoint or just directly check the index using `GET /.integration_knowledge/_search` to verify that the documents are ingested into the system index of `.integration_knowledge` 5. Update the package and verify that the KB documents are updated by checking the response again, they should have the updated pkgVersion on the associated docs. 6. Remove the package and then verify (using the endpoint) that the docs are removed from the index ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. ### Identify risks Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss. Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging. - [ ] [See some risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) - [ ] ... # Release Note Adds support for indexing package knowledge base docs into the .integration_knowledge system index, with per-document updates and automatic removal when a package is deleted. To be utilized by package developers allowing the AI assistants to have greater context relevant to particular packages. --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>
1 parent 309eae0 commit 748e3a9

File tree

48 files changed

+2779
-25
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+2779
-25
lines changed

‎oas_docs/bundle.json‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -21262,7 +21262,8 @@
2126221262
"ilm_policy",
2126321263
"data_stream_ilm_policy",
2126421264
"transform",
21265-
"ml_model"
21265+
"ml_model",
21266+
"knowledge_base"
2126621267
],
2126721268
"type": "string"
2126821269
},
@@ -21828,7 +21829,8 @@
2182821829
"ilm_policy",
2182921830
"data_stream_ilm_policy",
2183021831
"transform",
21831-
"ml_model"
21832+
"ml_model",
21833+
"knowledge_base"
2183221834
],
2183321835
"type": "string"
2183421836
},
@@ -22276,7 +22278,8 @@
2227622278
"ilm_policy",
2227722279
"data_stream_ilm_policy",
2227822280
"transform",
22279-
"ml_model"
22281+
"ml_model",
22282+
"knowledge_base"
2228022283
],
2228122284
"type": "string"
2228222285
},
@@ -22494,7 +22497,8 @@
2249422497
"ilm_policy",
2249522498
"data_stream_ilm_policy",
2249622499
"transform",
22497-
"ml_model"
22500+
"ml_model",
22501+
"knowledge_base"
2249822502
],
2249922503
"type": "string"
2250022504
},
@@ -23562,7 +23566,8 @@
2356223566
"ilm_policy",
2356323567
"data_stream_ilm_policy",
2356423568
"transform",
23565-
"ml_model"
23569+
"ml_model",
23570+
"knowledge_base"
2356623571
],
2356723572
"type": "string"
2356823573
},
@@ -23981,7 +23986,8 @@
2398123986
"ilm_policy",
2398223987
"data_stream_ilm_policy",
2398323988
"transform",
23984-
"ml_model"
23989+
"ml_model",
23990+
"knowledge_base"
2398523991
],
2398623992
"type": "string"
2398723993
},
@@ -24513,7 +24519,8 @@
2451324519
"ilm_policy",
2451424520
"data_stream_ilm_policy",
2451524521
"transform",
24516-
"ml_model"
24522+
"ml_model",
24523+
"knowledge_base"
2451724524
],
2451824525
"type": "string"
2451924526
},
@@ -24929,7 +24936,8 @@
2492924936
"ilm_policy",
2493024937
"data_stream_ilm_policy",
2493124938
"transform",
24932-
"ml_model"
24939+
"ml_model",
24940+
"knowledge_base"
2493324941
],
2493424942
"type": "string"
2493524943
},

‎oas_docs/bundle.serverless.json‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -21262,7 +21262,8 @@
2126221262
"ilm_policy",
2126321263
"data_stream_ilm_policy",
2126421264
"transform",
21265-
"ml_model"
21265+
"ml_model",
21266+
"knowledge_base"
2126621267
],
2126721268
"type": "string"
2126821269
},
@@ -21828,7 +21829,8 @@
2182821829
"ilm_policy",
2182921830
"data_stream_ilm_policy",
2183021831
"transform",
21831-
"ml_model"
21832+
"ml_model",
21833+
"knowledge_base"
2183221834
],
2183321835
"type": "string"
2183421836
},
@@ -22276,7 +22278,8 @@
2227622278
"ilm_policy",
2227722279
"data_stream_ilm_policy",
2227822280
"transform",
22279-
"ml_model"
22281+
"ml_model",
22282+
"knowledge_base"
2228022283
],
2228122284
"type": "string"
2228222285
},
@@ -22494,7 +22497,8 @@
2249422497
"ilm_policy",
2249522498
"data_stream_ilm_policy",
2249622499
"transform",
22497-
"ml_model"
22500+
"ml_model",
22501+
"knowledge_base"
2249822502
],
2249922503
"type": "string"
2250022504
},
@@ -23562,7 +23566,8 @@
2356223566
"ilm_policy",
2356323567
"data_stream_ilm_policy",
2356423568
"transform",
23565-
"ml_model"
23569+
"ml_model",
23570+
"knowledge_base"
2356623571
],
2356723572
"type": "string"
2356823573
},
@@ -23981,7 +23986,8 @@
2398123986
"ilm_policy",
2398223987
"data_stream_ilm_policy",
2398323988
"transform",
23984-
"ml_model"
23989+
"ml_model",
23990+
"knowledge_base"
2398523991
],
2398623992
"type": "string"
2398723993
},
@@ -24513,7 +24519,8 @@
2451324519
"ilm_policy",
2451424520
"data_stream_ilm_policy",
2451524521
"transform",
24516-
"ml_model"
24522+
"ml_model",
24523+
"knowledge_base"
2451724524
],
2451824525
"type": "string"
2451924526
},
@@ -24929,7 +24936,8 @@
2492924936
"ilm_policy",
2493024937
"data_stream_ilm_policy",
2493124938
"transform",
24932-
"ml_model"
24939+
"ml_model",
24940+
"knowledge_base"
2493324941
],
2493424942
"type": "string"
2493524943
},

‎oas_docs/output/kibana.serverless.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24480,6 +24480,7 @@ paths:
2448024480
- data_stream_ilm_policy
2448124481
- transform
2448224482
- ml_model
24483+
- knowledge_base
2448324484
type: string
2448424485
version:
2448524486
type: string
@@ -24872,6 +24873,7 @@ paths:
2487224873
- data_stream_ilm_policy
2487324874
- transform
2487424875
- ml_model
24876+
- knowledge_base
2487524877
type: string
2487624878
version:
2487724879
type: string
@@ -25175,6 +25177,7 @@ paths:
2517525177
- data_stream_ilm_policy
2517625178
- transform
2517725179
- ml_model
25180+
- knowledge_base
2517825181
type: string
2517925182
version:
2518025183
type: string
@@ -25325,6 +25328,7 @@ paths:
2532525328
- data_stream_ilm_policy
2532625329
- transform
2532725330
- ml_model
25331+
- knowledge_base
2532825332
type: string
2532925333
version:
2533025334
type: string
@@ -25787,6 +25791,7 @@ paths:
2578725791
- data_stream_ilm_policy
2578825792
- transform
2578925793
- ml_model
25794+
- knowledge_base
2579025795
type: string
2579125796
version:
2579225797
type: string
@@ -26076,6 +26081,7 @@ paths:
2607626081
- data_stream_ilm_policy
2607726082
- transform
2607826083
- ml_model
26084+
- knowledge_base
2607926085
type: string
2608026086
version:
2608126087
type: string
@@ -26435,6 +26441,7 @@ paths:
2643526441
- data_stream_ilm_policy
2643626442
- transform
2643726443
- ml_model
26444+
- knowledge_base
2643826445
type: string
2643926446
version:
2644026447
type: string
@@ -26723,6 +26730,7 @@ paths:
2672326730
- data_stream_ilm_policy
2672426731
- transform
2672526732
- ml_model
26733+
- knowledge_base
2672626734
type: string
2672726735
version:
2672826736
type: string

‎oas_docs/output/kibana.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28262,6 +28262,7 @@ paths:
2826228262
- data_stream_ilm_policy
2826328263
- transform
2826428264
- ml_model
28265+
- knowledge_base
2826528266
type: string
2826628267
version:
2826728268
type: string
@@ -28675,6 +28676,7 @@ paths:
2867528676
- data_stream_ilm_policy
2867628677
- transform
2867728678
- ml_model
28679+
- knowledge_base
2867828680
type: string
2867928681
version:
2868028682
type: string
@@ -28985,6 +28987,7 @@ paths:
2898528987
- data_stream_ilm_policy
2898628988
- transform
2898728989
- ml_model
28990+
- knowledge_base
2898828991
type: string
2898928992
version:
2899028993
type: string
@@ -29142,6 +29145,7 @@ paths:
2914229145
- data_stream_ilm_policy
2914329146
- transform
2914429147
- ml_model
29148+
- knowledge_base
2914529149
type: string
2914629150
version:
2914729151
type: string
@@ -29639,6 +29643,7 @@ paths:
2963929643
- data_stream_ilm_policy
2964029644
- transform
2964129645
- ml_model
29646+
- knowledge_base
2964229647
type: string
2964329648
version:
2964429649
type: string
@@ -29928,6 +29933,7 @@ paths:
2992829933
- data_stream_ilm_policy
2992929934
- transform
2993029935
- ml_model
29936+
- knowledge_base
2993129937
type: string
2993229938
version:
2993329939
type: string
@@ -30300,6 +30306,7 @@ paths:
3030030306
- data_stream_ilm_policy
3030130307
- transform
3030230308
- ml_model
30309+
- knowledge_base
3030330310
type: string
3030430311
version:
3030530312
type: string
@@ -30595,6 +30602,7 @@ paths:
3059530602
- data_stream_ilm_policy
3059630603
- transform
3059730604
- ml_model
30605+
- knowledge_base
3059830606
type: string
3059930607
version:
3060030608
type: string

‎x-pack/platform/plugins/shared/automatic_import/public/common/lib/api.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ import {
3939

4040
export interface EpmPackageResponse {
4141
items: [{ id: string; type: string }];
42+
_meta?: {
43+
install_source: string;
44+
name: string;
45+
};
4246
}
4347

4448
const defaultHeaders = {

‎x-pack/platform/plugins/shared/automatic_import/public/common/lib/api_parsers.test.ts‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,4 +61,37 @@ describe('getIntegrationNameFromResponse', () => {
6161
} as EpmPackageResponse;
6262
expect(getIntegrationNameFromResponse(response)).toEqual('');
6363
});
64+
65+
it('should return the integration name from _meta.name when available', () => {
66+
const response = {
67+
items: [{ type: 'ingest_pipeline', id: 'audit-security.data-stream-1.0.0' }],
68+
_meta: {
69+
install_source: 'upload',
70+
name: 'my-custom-integration',
71+
},
72+
} as EpmPackageResponse;
73+
expect(getIntegrationNameFromResponse(response)).toEqual('my-custom-integration');
74+
});
75+
76+
it('should fall back to parsing ingest pipeline when _meta.name is not available', () => {
77+
const response = {
78+
items: [{ type: 'ingest_pipeline', id: 'audit-security.data-stream-1.0.0' }],
79+
_meta: {
80+
install_source: 'upload',
81+
name: '',
82+
},
83+
} as EpmPackageResponse;
84+
expect(getIntegrationNameFromResponse(response)).toEqual('security-1.0.0');
85+
});
86+
87+
it('should prefer _meta.name over ingest pipeline parsing', () => {
88+
const response = {
89+
items: [{ type: 'ingest_pipeline', id: 'audit-old-name.data-stream-1.0.0' }],
90+
_meta: {
91+
install_source: 'upload',
92+
name: 'new-integration-name',
93+
},
94+
} as EpmPackageResponse;
95+
expect(getIntegrationNameFromResponse(response)).toEqual('new-integration-name');
96+
});
6497
});

‎x-pack/platform/plugins/shared/automatic_import/public/common/lib/api_parsers.ts‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,11 +8,18 @@
88
import type { EpmPackageResponse } from './api';
99

1010
/**
11-
* This is a hacky way to get the integration name from the response.
12-
* Since the integration name is not returned in the response we have to parse it from the ingest pipeline name.
11+
* Gets the integration name from the response.
12+
* First tries to get it from the _meta.name field, then falls back to parsing it from ingest pipeline names.
13+
* Since the integration name is not always returned in the response we have to parse it from the ingest pipeline name.
1314
* TODO: Return the package name from the fleet API: https://github.com/elastic/kibana/issues/185932
1415
*/
1516
export const getIntegrationNameFromResponse = (response: EpmPackageResponse) => {
17+
// First try to get the name from the _meta field
18+
if (response?._meta?.name) {
19+
return response._meta.name;
20+
}
21+
22+
// Fall back to parsing from ingest pipeline name (legacy behavior)
1623
return (
1724
response?.items
1825
?.find((item) => item.type === 'ingest_pipeline')

‎x-pack/platform/plugins/shared/fleet/.storybook/context/fixtures/integration.nginx.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -301,6 +301,7 @@ export const item: GetInfoResponse['item'] = {
301301
index_template: [],
302302
transform: [],
303303
ml_model: [],
304+
knowledge_base: [],
304305
},
305306
},
306307
policy_templates: [

‎x-pack/platform/plugins/shared/fleet/.storybook/context/fixtures/integration.okta.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,7 @@ export const item: GetInfoResponse['item'] = {
130130
index_template: [],
131131
transform: [],
132132
ml_model: [],
133+
knowledge_base: [],
133134
},
134135
},
135136
policy_templates: [

‎x-pack/platform/plugins/shared/fleet/common/constants/epm.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,9 @@ export const installationStatuses = {
114114
// This array also controls the order in which the asset types are displayed
115115
export const displayedAssetTypes: DisplayedAssetTypes = [
116116
...Object.values(KibanaSavedObjectType),
117-
...Object.values(ElasticsearchAssetType),
117+
...(Object.values(ElasticsearchAssetType).filter(
118+
(assetType) => assetType !== ElasticsearchAssetType.knowledgeBase
119+
) as ElasticsearchAssetType[]), // Filter out knowledgeBase assets, we dont want to expose to the user
118120
];
119121

120122
export const displayedAssetTypesLookup = new Set<string>(displayedAssetTypes);

0 commit comments

Comments
 (0)