AI Assistant Knowledge Base walkthrough (#2616)

Fixes #2529 by updating the Security AI Assistant's Knowledge Base page,
adding more detail about how to add data to KB. Also creates a new
example / tutorial page for adding info to Knowledge Base, designed to
help users get a practical idea of how to use the Knowledge Base, and
showcasing an excellent demo video.

Also fixes #2528 by updating the "Add index" to KB section to inform
users that starting with 9.1, text fields can be used instead of
semantic text fields for helping AI Assistant decide which knowledge
entries to refer to.

---------

Co-authored-by: Mike Birnstiehl <[email protected]>
Co-authored-by: Garrett Spong <[email protected]>

Author: 3 people

solutions/images/security-knowledge-base-add-on-call-rotation.png

@@ -27,7 +27,7 @@ AI Assistant’s Knowledge Base feature enables AI Assistant to recall specific
 ::::
 
 ::::{note}
-It is highly recommended to [enable autoscaling](/deploy-manage/autoscaling.md#cluster-autoscaling) to use the AI Assistant Knowledge Base.
+We strongly recommend you [enable autoscaling](/deploy-manage/autoscaling.md#cluster-autoscaling) before using Knowledge Base.
 ::::
 
 ## Role-based access control (RBAC) for Knowledge Base [knowledge-base-rbac]
@@ -53,7 +53,6 @@ You must individually enable Knowledge Base for each {{kib}} space where you wan
 ::::
 
 
-
 ### Option 1: Enable Knowledge Base from an AI Assistant conversation [_option_1_enable_knowledge_base_from_an_ai_assistant_conversation]
 
 Open a conversation with AI Assistant, select a large language model, then click **Setup Knowledge Base**. If the button doesn’t appear, Knowledge Base is already enabled.
@@ -105,9 +104,9 @@ When you enable Knowledge Base, it comes pre-populated with articles from [Elast
 
 
 
-### Add an individual document [knowledge-base-add-knowledge-document]
+### Add a markdown document [knowledge-base-add-knowledge-document]
 
-Add an individual document to Knowledge Base when you want AI Assistant to remember a specific piece of information.
+Add a markdown document to Knowledge Base when you want AI Assistant to remember a specific piece of information. 
 
 1. To open **Security AI settings**, use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md) to find "AI Assistant for Security." Select the **Knowledge Base** tab.
 2. Click **New → Document** and give it a name.
@@ -120,48 +119,67 @@ Refer to the following video for an example of adding a document to Knowledge Ba
 
 [![Add knowledge document video](https://play.vidyard.com/rQsTujEfikpx3vv1vrbfde.jpg)](https://videos.elastic.co/watch/rQsTujEfikpx3vv1vrbfde?)
 
+### Add an individual file [add-specific-file]
+
+To add an individual file to Knowledge Base, you first need to ingest it into an index and ensure that it includes a semantic text or text field. Supported file types include text, PDF, ODF, Word, Excel, PowerPoint, NDJSON, CSV, and TSV.
+
+1. Access the **Data Visualizer** interface to upload a file using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md) to find "File upload". 
+2. Review the list of currently supported file formats and sizes, then select the file you want to upload. A preview of your data appears. In the **Summary** section, click **Import**.
+3. Go to the **Advanced** tab.  In the **Index name** field , enter a name for the index that will contain the data in the uploaded file.
+4. (Optional) Review and update the mappings and ingest pipeline for your new index. 
+5. Click **Add additional field -> Add semantic text field**. 
+  - For **Field**, select the field you want to use as a semantic text field. It should contain information that AI Assistant can use to determine whether a document is relevant to a given query. Do not select a metadata field.
+  - For **Copy to field**, enter a name for your new semantic text field.
+  - For **Inference endpoint**, use the default or select another model that's enabled in your environment. 
+  - Click **Add**. The new field appears in the **Mappings** section.
+6. Click **Import**. File ingest begins and should complete within a few seconds.
+7. Once your file has been ingested to an index, add it to Knowledge Base by following the steps to [add an index](#knowledge-base-add-knowledge-index). 
+
+Refer to the following video for an example of this process (click to play video):
+
+[![Add knowledge index video](https://play.vidyard.com/Q5CjXMN4R2GYLGLUy5P177.jpg)](https://videos.elastic.co/watch/Q5CjXMN4R2GYLGLUy5P177?)
+
 
 
 ### Add an index [knowledge-base-add-knowledge-index]
 
-Add an index as a knowledge source when you want new information added to that index to automatically inform AI Assistant’s responses. Common security examples include asset inventories, network configuration information, on-call matrices, threat intelligence reports, and vulnerability scans.
+Add an index as a knowledge source when you want information in that index to inform AI Assistant’s responses. Common security examples include asset inventories, network configuration information, on-call matrices, threat intelligence reports, and vulnerability scans. When you update the index with new information, AI Assistant will gain access to the new information.
 
 ::::{important}
-Indices added to Knowledge Base must have at least one field mapped as [semantic text](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md).
-::::
+Indices added to Knowledge Base must have at least one field mapped as [semantic text](elasticsearch://reference/elasticsearch/mapping-reference/semantic-text.md). 
 
+{applies_to}`stack: ga 9.1` {applies_to}`serverless: ga` You can use a text field instead of a semantic text field. Semantic text fields offer better performance for large blobs of text and matching on semantic relevancy, while text fields perform better for retrieval based on specific document values or attributes, such as email or username. 
+::::
 
 1. To open **Security AI settings**, use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md) to find "AI Assistant for Security." Select the **Knowledge Base** tab.
 2. Click **New → Index**.
 3. Name the knowledge source.
 4. Under **Sharing**, select whether this knowledge should be **Global** or **Private**.
 5. Under **Index**, enter the name of the index you want to use as a knowledge source.
-6. Under **Field**, enter the names of one or more semantic text fields within the index.
+6. Under **Field**, enter the names of one or more semantic text ({applies_to}`stack: ga 9.1` {applies_to}`serverless: ga` or text) fields within the index.
 7. Under **Data Description**, describe when this information should be used by AI Assistant.
-8. Under **Query Instruction**, describe how AI Assistant should query this index to retrieve relevant information.
-9. Under **Output Fields**, list the fields which should be sent to AI Assistant. If none are listed, all fields will be sent.
+8. Under **Query Instruction**, describe how AI Assistant should query this index to retrieve relevant documents.
+9. Under **Output Fields**, list the fields which AI Assistant should look at when reviewing documents in this index. If none are listed, all fields are sent.
 
 :::{image} /solutions/images/security-knowledge-base-add-index-config.png
 :alt: Knowledge base's Edit index entry menu
 :::
 
-Refer to the following video for an example of adding an index to Knowledge Base (click to play video).
-
-
-[![Add knowledge index video](https://play.vidyard.com/Q5CjXMN4R2GYLGLUy5P177.jpg)](https://videos.elastic.co/watch/Q5CjXMN4R2GYLGLUy5P177?)
 
+### Add knowledge to an index using a content connector or web crawler [knowledge-base-crawler-or-connector]
 
+You can use an {{es}} connector or web crawler to create an index that contains data you want to add to Knowledge Base.
 
-### Add knowledge with a connector or web crawler [knowledge-base-crawler-or-connector]
+#### Use a content connector to ingest data from third-party applications to Knowledge Base
 
-You can use an {{es}} connector or web crawler to create an index that contains data you want to add to Knowledge Base.
+You can ingest data from third-party platforms such as Github, Jira, Teams, Google Drive, Slack, email, and [more](elasticsearch://reference/search-connectors/index.md) using [content connectors](/solutions/security/get-started/content-connectors.md). 
 
-This section provides an example of adding a threat intelligence feed to Knowledge Base using a web crawler. For more information on adding data to {{es}} using a connector, refer to [Ingest data with Elastic connectors](elasticsearch://reference/search-connectors/index.md). For more information on web crawlers, refer to [Elastic web crawler](https://www.elastic.co/guide/en/enterprise-search/current/crawler.html).
+Once you've set up a content connector, data from the selected source is ingested to an {{es}} index. To add it to Knowledge Base, follow the steps to [add an index](#knowledge-base-add-knowledge-index). 
 
 
 #### Use a web crawler to add threat intelligence to Knowledge Base [_use_a_web_crawler_to_add_threat_intelligence_to_knowledge_base]
 
-First, you’ll need to set up a web crawler to add the desired data to an index, then you’ll need to add that index to Knowledge Base.
+First, you’ll need to set up a web crawler to add the desired data to an index, then you’ll need to add that index to Knowledge Base. For more information on web crawlers, refer to [web crawlers](https://www.elastic.co/guide/en/enterprise-search/current/crawler.html).
 
 1. From the **Search** section of {{kib}}, find **Web crawlers** in the navigation menu or use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
 2. Click **New web crawler**.
@@ -190,4 +208,11 @@ Your new threat intelligence data is now included in Knowledge Base and can info
 
 Refer to the following video for an example of creating a web crawler to ingest threat intelligence data and adding it to Knowledge Base.
 
-[![Add knowledge via web crawler video](https://play.vidyard.com/eYo1e1ZRwT2mjfM7Yr9MuZ.jpg)](https://videos.elastic.co/watch/eYo1e1ZRwT2mjfM7Yr9MuZ?)
+[![Add knowledge via web crawler video](https://play.vidyard.com/eYo1e1ZRwT2mjfM7Yr9MuZ.jpg)](https://videos.elastic.co/watch/eYo1e1ZRwT2mjfM7Yr9MuZ?)
+
+
+## Additional resources
+
+- For a walkthrough of how Knowledge Base can improve the quality of AI Assistant's responses, refer to [Use AI Assistant's Knowledge Base to improve response quality](/solutions/security/ai/usecase-knowledge-base-walkthrough.md).
+- To learn more about semantic search and inference models, refer to [Elasticsearch semantic_text mapping](https://www.elastic.co/search-labs/blog/semantic-search-simplified-semantic-text).
+- For more information about how the data in Knowledge Base gets chunked, refer to [Intelligent RAG data chunking](https://www.elastic.co/search-labs/blog/advanced-chunking-fetch-surrounding-chunks).

solutions/security/ai/ai-assistant-knowledge-base.md

@@ -0,0 +1,76 @@
+---
+applies_to:
+  stack: ga 9.1
+  serverless:
+    security: all
+products:
+  - id: security
+---
+
+
+# Use AI Assistant's Knowledge Base to improve response quality
+
+You can use AI Assistant's Knowledge Base to give it information on anything from threat hunting playbooks, to on-call rotations, security research, infrastructure information, your team's internal communications from platforms like Slack or Teams, and more — constrained only by your creativity. This page guides you through ingesting data from various sources into AI Assistant's Knowledge Base, and shows how this can improve the quality of its responses in a threat response scenario. 
+
+## Prerequisites
+
+Before following this guide, review the [Knowledge Base](/solutions/security/ai/ai-assistant-knowledge-base.md) topic for general information and prerequisites, and [enable Knowledge Base](/solutions/security/ai/ai-assistant-knowledge-base.md#enable-knowledge-base).
+
+## Add relevant data from various sources to Knowledge Base
+
+AI Assistant is more useful for incident response when it can access information about your organization's specific infrastructure, threat hunting playbooks, personnel, and processes. How you can add this data to Knowledge Base depends on its format and structure. This section provides several examples of useful data and how to add it. 
+
+### Add your Slack messages to Knowledge Base
+
+You can add messages from Slack channels to Knowledge Base using the Slack content connector. For instance, if you have a Slack channel that contains information about ongoing incidents, you could include that information in Knowledge Base to give AI Assistant more context about what your security team is dealing with. 
+
+1. Use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md) to find "Content connectors". Click **+ New Connector** to open the **Create a connector** interface.
+2. Follow the steps to [create a content connector](/solutions/security/get-started/content-connectors.md). During setup, select `Slack`, then follow the steps to [configure a Slack connector](elasticsearch://reference/search-connectors/es-connectors-slack.md). This ingests your selected data into {{es}}.
+3. Follow the instructions to [add an index to Knowledge Base](/solutions/security/ai/ai-assistant-knowledge-base.md#). Select the index you created while setting up your new connector.
+
+### Add your on-call rotation to Knowledge Base
+
+If you add information about who is responsible for security incidents at different dates and times to Knowledge Base, AI Assistant can help you quickly follow the correct escalation protocol for potential threats. 
+
+If information about your on-call rotation is contained in a file, you can follow the steps to [add an individual file](/solutions/security/ai/ai-assistant-knowledge-base.md#add-specific-file) to Knowledge Base. 
+
+However, you can also copy and paste the information to directly [add it as a markdown document](/solutions/security/ai/ai-assistant-knowledge-base.md#knowledge-base-add-knowledge-document). Adding it as a markdown document is fast and easy to update when the on-call rotation changes. 
+
+:::{image} /solutions/images/security-knowledge-base-add-on-call-rotation.png
+:alt: Knowledge base's Edit document entry menu showing a snippet of an on call rotation document
+:::
+
+Whichever method you use to add the information to Knowledge Base, consider making it **Required knowledge**. This will ensure that all of AI Assistant's responses are informed by the on-call rotation, even if your prompt doesn't specify that the information is relevant. This makes it more likely that AI Assistant will suggest appropriate escalation steps when you ask it about a threat.
+
+### Add your threat hunting playbooks to Knowledge Base
+ 
+If you have threat hunting playbooks stored in a GitHub repository, you can add them to Knowledge Base using the GitHub content connector. This enables AI Assistant to tell your team about your organization's standard practices for responding to a wide range of potential threats. 
+
+1. Use the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md) to find "Content connectors". Click **+ New Connector** to open the **Create a connector** interface.
+2. Follow the steps to [create a content connector](/solutions/security/get-started/content-connectors.md). During setup, select `GitHub`, then follow the steps to [configure a GitHub connector](elasticsearch://reference/search-connectors/es-connectors-github.md). This ingests your selected data into {{es}}.
+3. Follow the instructions to [add an index to Knowledge Base](/solutions/security/ai/ai-assistant-knowledge-base.md#). Select the index you created while setting up your new connector.
+
+::::{note}
+The GitHub connector can only ingest issues, PRs and the following file types: `.markdown`, `.md`, `.rst`.
+::::
+
+## Use Knowledge Base in conversations
+
+AI Assistant will use the information you've added to Knowledge Base to inform its responses to your prompts. With the information we've added in this guide, you can ask questions like:
+
+- Is this alert related to any ongoing incidents?
+- Who should I contact to escalate this potential threat?
+- What should I do to respond to this threat?
+
+Be creative, and experiment with adding different types of information to optimize AI Assistant for your team's purposes.
+
+## Video demo: investigate an Attack Discovery using AI Assistant's Knowledge Base
+
+The following video demo starts with a potential threat identified using Attack Discovery, and shows how the information you've added to Knowledge Base greatly increases AI Assistant's ability to help guide your team's incident response (click to play video):
+
+[![Add knowledge index video](https://play.vidyard.com/SGrcygEFBCEJRURGjR8sMh.jpg)](https://videos.elastic.co/watch/SGrcygEFBCEJRURGjR8sMh?)
+
+## Additional Resources
+
+- Learn more about [Knowledge Base](https://www.elastic.co/guide/en/security/current/ai-assistant-knowledge-base.html)
+- Learn to [Ingest data with Elastic connectors](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-connectors.html)

solutions/security/ai/usecase-knowledge-base-walkthrough.md

@@ -507,6 +507,7 @@ toc:
           - file: security/ai/ai-assistant.md
             children:
               - file: security/ai/ai-assistant-knowledge-base.md
+              - file: security/ai/usecase-knowledge-base-walkthrough.md
           - file: security/ai/attack-discovery.md
           - file: security/ai/set-up-connectors-for-large-language-models-llm.md
             children: