[Security Assistant] Fixes issue preventing the creation of Knowledge Base Index Entries in deployments with a large number of indices/mappings (elastic#231376)

## Summary

This PR fixes an issue with the Security Assistant KB Index Entries
interface introduced in `9.1` where a large number of indices/mappings
could result in Kibana crashing and preventing the creation of the Index
Entry.

This is technically a 'fix-hancement' as we are bypassing the underlying
issue altogether by switching to use common core API's for index/field
suggestions (just as is done in the Discover 'Create a data view'
interface), and in turn supporting all fields of type `text`, not just
`semantic_text` (elastic#230863).

### Original Issue
The underlying issue here was introduced by a change to the `field_caps`
API in `9.1` (elastic/elasticsearch#127664) that
resulted in the `/internal/elastic_assistant/knowledge_base/_indices`
route not finding any indices with a `semantic_text` field, and thus
inadvertently falling back to doing a full scan of all mappings
([source](https://github.com/elastic/kibana/blob/b128cee4ee7ccc367e8acf159dbf58a75f081867/x-pack/solutions/security/plugins/elastic_assistant/server/routes/knowledge_base/get_knowledge_base_indices.ts#L69)).
A fix to this API was initially investigated, but there was no
reasonable API available for fetching all occurrences of `semantic_text`
fields, so with `match` queries adding support for `semantic_text` in
`8.18`, it was decided to go ahead and enable support for all `text`
fields.


### Fix Details

The `Index` input field now uses the `dataViews.getIndices()` API for
suggestions (instead of the `useKnowledgeBaseIndices` hook/route), which
is backed by the [resolve indices ES
API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-resolve-index).
System indices are filtered out with the `*,-.*` filter. The initial
call will return all indices just as the Discover 'Create a data view'
interface, which is further filtered upon as the user continues to type.
Note: Discover makes subsequent calls upon further user input, though
I'm not entirely sure this is necessary here as all indices are
initially returned and available for client-side filtering within the
input. I will perform further stress testing with many indices/mappings
to confirm.

The `Field` input now uses the fields already queried for the `Output
fields` input suggestions (via `dataViews.getFieldsForWildcard()`), just
filtered to those whose `field.esTypes?.includes('text')`. This is
potentially still a hot path for client-side code with many mappings, so
I will also confirm this with further stress testing.


### Docs

@elastic/security-docs / @benironside, we will need to update the
Security Assistant KB docs
[here](https://www.elastic.co/docs/solutions/security/ai/ai-assistant-knowledge-base#knowledge-base-add-knowledge-index)
to indicate that any `text` field is now supported for retrieval.


### Testing

#### Functional Testing

To confirm proper retrieval of both `text` and `semantic_text` fields
we'll need to create an index, add a document, then create the KB Index
Entry. Then update the KB Index Entry to reference the other field type,
and test again.

<details><summary>Create Index</summary>
<p>

``` JSON
PUT project-details
{
  "mappings": {
    "properties": {
      "project_issue": {
        "type": "text"
      },
      "project_name": {
        "type": "text"
      },
      "summary": {
        "type": "semantic_text",
        "inference_id": ".elser-2-elasticsearch",
        "model_settings": {
          "service": "elasticsearch",
          "task_type": "sparse_embedding"
        }
      }
    }
  }
}
```
</p>
</details> 

<details><summary>Create Sample Doc</summary>
<p>

``` JSON
PUT project-details/_doc/doc1
{
    "project_issue": "The main issue at hand is the breaking of the space plane",
    "project_name": "Issue 5",
    "summary": "This is a summary that contains the word yellow"
}
```
</p>
</details> 

<details><summary>Create KB Index Entry (Text)</summary>
<p>

``` JSON
POST kbn:api/security_ai_assistant/knowledge_base/entries
{
  "type": "index",
  "name": "Project Details Tool",
  "index": "project-details",
  "field": "project_issue",
  "outputFields": [],
  "description": "Use this index to answer questions about any project details.",
  "queryDescription": "Key terms to search for from the user's prompt."
}
```
</p>
</details> 

Now you can open the Assistant and perform a query like:
```
Do I have any project details about the issue at hand?
```

Which should result in a [trace like
this](https://smith.langchain.com/public/208338a6-42f3-4a9d-bc4c-0ff38d06d34c/r)
which calls the generated tool that will then perform a _lexical search_
against the configured index. Ensure citations work as expected for the
returned document.

Now open the Index Entry in the KB Settings UI and change the `field` to
from `project_issue` to `summary` for testing `semantic_text`.

Open the Assistant and perform a query like:
```
Do I have any project details for Project Yellow?
```

Which should result in a [trace like
this](https://smith.langchain.com/public/3041a48b-5dfd-4c89-9f51-0bcdffd38f63/r)
which calls the generated tool that will then perform a _semantic
search_ against the configured index. Ensure citations work as expected
for the returned document.







#### Performance Testing

⚠️ In progress -- I will include a script for generating many
indices/mappings for testing, and also prepare the `ci-cloud-deploy`
instance with the same setup for confirmation.


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [X] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
  * Will coordinate with @elastic/security-docs on the docs update here.
- [X] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: kibanamachine <[email protected]>

Author: 2 people

src/platform/packages/shared/kbn-doc-links/src/get_doc_links.ts

@@ -483,7 +483,9 @@ export const getDocLinks = ({ kibanaBranch, buildFlavor }: GetDocLinkOptions): D
         mlAnomalyDetection: `${ELASTIC_DOCS}explore-analyze/machine-learning/anomaly-detection`,
       },
       detectionEngineOverview: `${ELASTIC_DOCS}solutions/security/detect-and-alert`,
+      // TODO: Follow-up PR for creating an aiAssistant category adding all relevant doc links
       aiAssistant: `${ELASTIC_DOCS}solutions/security/ai/ai-assistant`,
+      aiAssistantKnowledgeBaseIndexEntries: `${ELASTIC_DOCS}solutions/security/ai/ai-assistant-knowledge-base#knowledge-base-add-knowledge-index`,
       signalsMigrationApi: isServerless
         ? `${KIBANA_APIS}group/endpoint-security-detections-api`
         : `${KIBANA_SERVERLESS_APIS}group/endpoint-security-detections-api`,

src/platform/packages/shared/kbn-doc-links/src/types.ts

@@ -301,6 +301,7 @@ export interface DocLinks {
   };
   readonly securitySolution: {
     readonly aiAssistant: string;
+    readonly aiAssistantKnowledgeBaseIndexEntries: string;
     readonly cloudSecurityPosture: string;
     readonly installElasticDefend: string;
     readonly artifactControl: string;

x-pack/platform/packages/shared/kbn-elastic-assistant-common/constants.ts

@@ -54,8 +54,6 @@ export const ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_ENTRIES_URL_FIND =
 export const ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_ENTRIES_URL_BULK_ACTION =
   `${ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_ENTRIES_URL}/_bulk_action` as const;
 
-export const ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_INDICES_URL =
-  `${ELASTIC_AI_ASSISTANT_INTERNAL_URL}/knowledge_base/_indices` as const;
 export const ELASTIC_AI_ASSISTANT_EVALUATE_URL =
   `${ELASTIC_AI_ASSISTANT_INTERNAL_URL}/evaluate` as const;
 

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/schemas/index.ts

@@ -60,7 +60,6 @@ export * from './actions_connector/post_actions_connector_execute_route.gen';
 
 // Knowledge Base Schemas
 export * from './knowledge_base/crud_kb_route.gen';
-export * from './knowledge_base/get_knowledge_base_indices_route.gen';
 export * from './knowledge_base/entries/bulk_crud_knowledge_base_entries_route.gen';
 export * from './knowledge_base/entries/common_attributes.gen';
 export * from './knowledge_base/entries/crud_knowledge_base_entries_route.gen';

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/schemas/knowledge_base/get_knowledge_base_indices_route.gen.ts

@@ -7,7 +7,7 @@
 
 import { HttpSetup } from '@kbn/core-http-browser';
 
-import { getKnowledgeBaseIndices, getKnowledgeBaseStatus, postKnowledgeBase } from './api';
+import { getKnowledgeBaseStatus, postKnowledgeBase } from './api';
 import { API_VERSIONS } from '@kbn/spaces-plugin/common';
 
 jest.mock('@kbn/core-http-browser');
@@ -73,29 +73,4 @@ describe('API tests', () => {
       await expect(postKnowledgeBase(knowledgeBaseArgs)).rejects.toThrowError('simulated error');
     });
   });
-
-  describe('getKnowledgeBaseIndices', () => {
-    it('calls the knowledge base API when correct resource path', async () => {
-      await getKnowledgeBaseIndices({ http: mockHttp });
-
-      expect(mockHttp.fetch).toHaveBeenCalledWith(
-        '/internal/elastic_assistant/knowledge_base/_indices',
-        {
-          method: 'GET',
-          signal: undefined,
-          version: '1',
-        }
-      );
-    });
-    it('returns error when error is an error', async () => {
-      const error = 'simulated error';
-      (mockHttp.fetch as jest.Mock).mockImplementation(() => {
-        throw new Error(error);
-      });
-
-      await expect(getKnowledgeBaseIndices({ http: mockHttp })).resolves.toThrowError(
-        'simulated error'
-      );
-    });
-  });
 });

x-pack/platform/packages/shared/kbn-elastic-assistant-common/impl/schemas/knowledge_base/get_knowledge_base_indices_route.schema.yaml

@@ -9,9 +9,7 @@ import {
   API_VERSIONS,
   CreateKnowledgeBaseRequestParams,
   CreateKnowledgeBaseResponse,
-  ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_INDICES_URL,
   ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_URL,
-  GetKnowledgeBaseIndicesResponse,
   ReadKnowledgeBaseRequestParams,
   ReadKnowledgeBaseResponse,
 } from '@kbn/elastic-assistant-common';
@@ -76,32 +74,3 @@ export const postKnowledgeBase = async ({
 
   return response as CreateKnowledgeBaseResponse;
 };
-
-/**
- * API call for getting indices that have fields of `semantic_text` type.
- *
- * @param {Object} options - The options object.
- * @param {HttpSetup} options.http - HttpSetup
- * @param {AbortSignal} [options.signal] - AbortSignal
- *
- * @returns {Promise<GetKnowledgeBaseIndicesResponse | IHttpFetchError>}
- */
-export const getKnowledgeBaseIndices = async ({
-  http,
-  signal,
-}: {
-  http: HttpSetup;
-  signal?: AbortSignal | undefined;
-}): Promise<GetKnowledgeBaseIndicesResponse | IHttpFetchError> => {
-  try {
-    const response = await http.fetch(ELASTIC_AI_ASSISTANT_KNOWLEDGE_BASE_INDICES_URL, {
-      method: 'GET',
-      signal,
-      version: API_VERSIONS.internal.v1,
-    });
-
-    return response as GetKnowledgeBaseIndicesResponse;
-  } catch (error) {
-    return error as IHttpFetchError;
-  }
-};