[Obs AI Assistant] aware of new .integration_knowledge* system index (elastic#237085)

Closes elastic/obs-ai-team#357
## Summary

This PR adds the awareness of `.integration_knowledge*` index as another
index for recalling. The Obs AI Assistant will retrieve integration
knowledge from the index

Value added to the Obs AI Assistant
(elastic/obs-ai-team#357 (comment)):
> The assistant will become aware of LLM-facing documentation for any
installed integrations in the user's cluster. For example, the Logstash
integration might ship documentation that explains how to understand the
health reporting metrics collected by the integration and the assistant
could answer prompts like "why was my logstash server down yesterday?"
using the user's real data.

Manual Testing: 
1 -> Follow the instructions from
elastic#230107 (comment):

> 1. If elastic/elasticsearch#132506 has been
merged, run `yarn es snapshot` in Kibana, otherwise, checkout that
branch in your local ES and then in kibana run `yarn es source` in order
to use that version of ES which contains the index management, mappings,
etc.
>2. Install a package with any number of knowledge base docs in the
`docs/knowledge_base` folder. You can use [this sample
package](https://github.com/user-attachments/files/21867395/masonstestpackage-0.0.1.zip),
or create your own following the guide below:
>
> - Using `elastic-package`, create a new package using `elastic-package
create integration`
- Once created add `knowledge_base` as a folder inside of the generated
`docs` folder of the integration
- Add an arbitrary amount of `.md` files to the knowledge_base folder
      - Run `elastic-package build` to build the package
- There are a lot of different options for installing the package in a
local kibana instance. I prefer to just take the generated .zip folder
from `/build` in `elastic-package` and upload it to kibana using the
custom integrations feature. You can also expose the package registry,
or whatever you see fit.
>3. Watch the Kibana logs for errors/debug messages etc
>4. Use the new endpoint or just directly check the index using `GET
/.integration_knowledge/_search` to verify that the documents are
ingested into the system index of `.integration_knowledge`
>5. Update the package and verify that the KB documents are updated by
checking the response again, they should have the updated pkgVersion on
the associated docs.
>6. Remove the package and then verify (using the endpoint) that the
docs are removed from the index

2 - Ask the AI Assistant about information contained in the integration
documents

3- check that the documents are listed on the response of executed the
function context inside learnings

### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Sandra G <[email protected]>

Author: 3 people

x-pack/platform/plugins/shared/observability_ai_assistant/common/types.ts

@@ -205,3 +205,12 @@ export enum ConversationAccess {
   SHARED = 'shared',
   PRIVATE = 'private',
 }
+
+export interface IntegrationKnowledgeBaseEntry {
+  content: string;
+  package_name: string;
+  filename: string;
+  version: string;
+  path: string;
+  installed_at: string;
+}

x-pack/platform/plugins/shared/observability_ai_assistant/server/service/knowledge_base_service/index.ts

@@ -13,7 +13,11 @@ import { encode } from 'gpt-tokenizer';
 import { isLockAcquisitionError } from '@kbn/lock-manager';
 import type { DocumentationManagerAPI } from '@kbn/product-doc-base-plugin/server/services/doc_manager';
 import { resourceNames } from '..';
-import type { Instruction, KnowledgeBaseEntry } from '../../../common/types';
+import type {
+  Instruction,
+  IntegrationKnowledgeBaseEntry,
+  KnowledgeBaseEntry,
+} from '../../../common/types';
 import { KnowledgeBaseEntryRole, KnowledgeBaseType } from '../../../common/types';
 import { getAccessQuery, getUserAccessFilters } from '../util/get_access_query';
 import { getCategoryQuery } from '../util/get_category_query';
@@ -36,6 +40,8 @@ import { getInferenceIdFromWriteIndex } from './get_inference_id_from_write_inde
 import { createOrUpdateKnowledgeBaseIndexAssets } from '../index_assets/create_or_update_knowledge_base_index_assets';
 import { LEGACY_CUSTOM_INFERENCE_ID } from '../../../common/preconfigured_inference_ids';
 
+const INTEGRATION_KNOWLEDGE_INDEX = '.integration_knowledge';
+
 interface Dependencies {
   core: CoreSetup<ObservabilityAIAssistantPluginStartDependencies>;
   esClient: {
@@ -112,6 +118,45 @@ export class KnowledgeBaseService {
     }));
   }
 
+  private async recallFromIntegrationsKnowledge({
+    queries,
+    esClient,
+  }: {
+    queries: Array<{ text: string; boost?: number }>;
+    esClient: { asCurrentUser: ElasticsearchClient; asInternalUser: ElasticsearchClient };
+  }): Promise<RecalledEntry[]> {
+    try {
+      // Search the .integration_knowledge index using semantic search on the content field
+      const response = await esClient.asInternalUser.search<IntegrationKnowledgeBaseEntry>({
+        index: INTEGRATION_KNOWLEDGE_INDEX,
+        query: {
+          bool: {
+            should: queries.map(({ text, boost = 1 }) => ({
+              semantic: {
+                field: 'content',
+                query: text,
+                boost,
+              },
+            })),
+          },
+        },
+        size: 10,
+        _source: ['package_name', 'filename', 'content', 'version'],
+      });
+      return response.hits.hits.map((hit) => ({
+        text: hit._source?.content!,
+        labels: { filename: hit._source?.filename ?? '', version: hit._source?.version ?? '' },
+        title: hit._source?.package_name,
+        esScore: hit._score!,
+        id: hit._id!,
+      }));
+    } catch (error) {
+      this.dependencies.logger.error(`Error recalling from ${INTEGRATION_KNOWLEDGE_INDEX} index`);
+      this.dependencies.logger.debug(error);
+      return [];
+    }
+  }
+
   recall = async ({
     user,
     queries,
@@ -137,40 +182,55 @@ export class KnowledgeBaseService {
       () => `Recalling entries from KB for queries: "${JSON.stringify(queries)}"`
     );
 
-    const [documentsFromKb, documentsFromConnectors] = await Promise.all([
-      this.recallFromKnowledgeBase({
-        user,
-        queries,
-        categories,
-        namespace,
-      }).catch((error) => {
-        if (isInferenceEndpointMissingOrUnavailable(error)) {
-          throwKnowledgeBaseNotReady(error);
-        }
-        throw error;
-      }),
-      recallFromSearchConnectors({
-        esClient,
-        uiSettingsClient,
-        queries,
-        core: this.dependencies.core,
-        logger: this.dependencies.logger,
-      }).catch((error) => {
-        this.dependencies.logger.error('Error getting data from search indices');
-        this.dependencies.logger.debug(error);
-        return [];
-      }),
-    ]);
+    const [documentsFromKb, documentsFromConnectors, documentsFromIntegrations] = await Promise.all(
+      [
+        this.recallFromKnowledgeBase({
+          user,
+          queries,
+          categories,
+          namespace,
+        }).catch((error) => {
+          if (isInferenceEndpointMissingOrUnavailable(error)) {
+            throwKnowledgeBaseNotReady(error);
+          }
+          throw error;
+        }),
+        recallFromSearchConnectors({
+          esClient,
+          uiSettingsClient,
+          queries,
+          core: this.dependencies.core,
+          logger: this.dependencies.logger,
+        }).catch((error) => {
+          this.dependencies.logger.error('Error getting data from search indices');
+          this.dependencies.logger.debug(error);
+          return [];
+        }),
+        this.recallFromIntegrationsKnowledge({
+          esClient,
+          queries,
+        }).catch((error) => {
+          this.dependencies.logger.error(
+            `Error getting data from ${INTEGRATION_KNOWLEDGE_INDEX} index`
+          );
+          this.dependencies.logger.debug(error);
+          return [];
+        }),
+      ]
+    );
 
     this.dependencies.logger.debug(
       `documentsFromKb: ${JSON.stringify(documentsFromKb.slice(0, 5), null, 2)}`
     );
     this.dependencies.logger.debug(
       `documentsFromConnectors: ${JSON.stringify(documentsFromConnectors.slice(0, 5), null, 2)}`
     );
+    this.dependencies.logger.debug(
+      `documentsFromIntegrations: ${JSON.stringify(documentsFromIntegrations.slice(0, 5), null, 2)}`
+    );
 
     const sortedEntries = orderBy(
-      documentsFromKb.concat(documentsFromConnectors),
+      [...documentsFromKb, ...documentsFromConnectors, ...documentsFromIntegrations],
       'esScore',
       'desc'
     ).slice(0, limit.size ?? 20);