[8.16]Adds Knowledge Base page and updates AI Assistant doc (#6040) (#6099)

* [8.16][serverless] Adds Knowledge Base page and updates AI Assistant doc

* Update knowledge-base.mdx

* ESS updates

* fix broken image refs

* minor updates

* typo fix

* updates from Charles' Feedback

* Update knowledge-base-assistant-menu-dropdown.png

* Adds Garrett's query for retrieving 8.15 knowledge

* removes query

* Delete docs/serverless/AI-for-security/ai-for-security-landing-pg.mdx

remove .mdx file

* Removes MDX files from PR

* removes mdx docnav changes

* Adds note about accessing anonymization settings

* Incorporates Janeen's review

* Incorporates Nat's review

* Incorporates Nat's review

* Update docs/AI-for-security/knowledge-base.asciidoc

* Update docs/AI-for-security/knowledge-base.asciidoc

(cherry picked from commit adbc277)

Co-authored-by: Benjamin Ironside Goldstein <[email protected]>

Author: mergify[bot]

docs/AI-for-security/ai-for-security.asciidoc

@@ -1,5 +1,5 @@
 [[ai-for-security]]
-= AI for security
+= AI for Security
 
 :frontmatter-description: Learn to use AI capabilities in {elastic-sec}.
 :frontmatter-tags-products: [security]
@@ -9,6 +9,8 @@
 You can use {elastic-sec}'s built-in AI tools to speed up your work and augment your team's capabilities. The pages in this section describe <<security-assistant, AI Assistant>>, which answers questions and enhances your workflows throughout {elastic-sec}, and <<attack-discovery, Attack discovery>>, which speeds up the triage process by finding patterns and identifying attacks spanning multiple alerts.
 
 include::ai-security-assistant.asciidoc[leveloffset=+1]
+include::knowledge-base.asciidoc[leveloffset=+2]
+
 include::attack-discovery.asciidoc[leveloffset=+1]
 
 include::connector-guides-landing-pg.asciidoc[leveloffset=+1]

docs/AI-for-security/ai-security-assistant.asciidoc

@@ -24,7 +24,7 @@ WARNING: The Elastic AI Assistant is designed to enhance your analysis with smar
 
 * To set up AI Assistant, you need the **Actions and Connectors : All** {kibana-ref}/kibana-privileges.html[privilege].
 
-* You need an account with a third-party generative AI provider, which AI Assistant uses to generate responses. Supported providers are OpenAI, Azure OpenAI Service, and Amazon Bedrock.
+* You need a <<llm-connector-guides, generative AI connector>>, which AI Assistant uses to generate responses. 
 --
 
 [discrete]
@@ -68,27 +68,25 @@ You can also chat with AI Assistant from several particular pages in {elastic-se
 * <<data-quality-dash, Data Quality dashboard>>: Select the *Incompatible fields* tab, then click *Chat*. (This is only available for fields marked red, indicating they're incompatible).
 * <<timelines-ui, Timeline>>: Select the *Security Assistant* tab.
 
-NOTE: Each user's chat history and custom quick prompts are automatically saved, so you can leave {elastic-sec} and return to pick up a conversation later.
+NOTE: Each user's chat history (up to the 99 most recent conversations) and custom Quick Prompts are automatically saved, so you can leave {elastic-sec} and return to a conversation later. Chat history appears to the left of the AI Assistant chat window, and on the **Conversations** tab of the **AI Assistant settings** menu. To access the settings menu, use the global search field to search for "AI Assistant for Security".
 
 [discrete]
 [[interact-with-assistant]]
 == Interact with AI Assistant
 
 Use these features to adjust and act on your conversations with AI Assistant:
 
-* Select a _system prompt_ at the beginning of a conversation to establish how detailed and technical you want AI Assistant's answers to be.
+* (Optional) Select a _System Prompt_ at the beginning of a conversation by using the **Select Prompt** menu. System Prompts provide context to the model, informing its response. To create a System Prompt, open the System Prompts dropdown menu and click *+ Add new System Prompt...*.
+* (Optional) Select a _Quick Prompt_ at the bottom of the chat window to get help writing a prompt for a specific purpose, such as summarizing an alert or converting a query from a legacy SIEM to {elastic-sec}.
 +
 [role="screenshot"]
-image::images/system-prompt.gif[The system prompt drop-down menu,90%]
+image::images/quick-prompts.png[Quick Prompts highlighted below a conversation,90%]
 +
-System prompts provide context to the model, informing its response. To create a custom system prompt, open the system prompts dropdown menu and click *+ Add new system prompt...*.
-
-* Select a _quick prompt_ at the bottom of the chat window to get help writing a prompt for a specific purpose, such as summarizing an alert or converting a query from a legacy SIEM to {elastic-sec}.
+* System Prompts and Quick Prompts can also be configured from the corresponding tabs on the **Security AI settings** page.
 +
-[role="screenshot"]
-image::images/quick-prompts.png[Quick prompts highlighted below a conversation,90%]
+image::images/assistant-settings-system-prompts.png[The Security AI settings menu's System Prompts tab,90%]
 +
-Quick prompt availability varies based on context — for example, the **Alert summarization** quick prompt appears when you open AI Assistant while viewing an alert. To customize existing quick prompts and create new ones, click *Add Quick prompt*.
+* Quick Prompt availability varies based on context—for example, the **Alert summarization** Quick Prompt appears when you open AI Assistant while viewing an alert. To customize existing Quick Prompts and create new ones, click *Add Quick Prompt*.
 
 * In an active conversation, you can use the inline actions that appear on messages to incorporate AI Assistant's responses into your workflows:
 
@@ -104,22 +102,16 @@ TIP: AI Assistant can remember particular information you tell it to remember. F
 [discrete]
 [[configure-ai-assistant]]
 == Configure AI Assistant
-The *Settings* menu (image:images/icon-settings.png[Settings icon,17,17]) allows you to configure default conversations, quick prompts, system prompts, and data anonymization.
-
-[role="screenshot"]
-image::images/assistant-settings-menu.png[AI Assistant's settings menu, open to the Conversations tab]
-
-The *Settings* menu has the following tabs:
+The *Security AI settings* page allows you to configure AI Assistant. To access it, use the global search field to search for "AI Assistant for Security".
 
-* **Conversations:** When you open AI Assistant from certain pages, such as Timeline or Alerts, it defaults to the relevant conversation type. Choose the default system prompt for each conversation type, the connector, and model (if applicable). The **Streaming** setting controls whether AI Assistant's responses appear word-by-word (streamed), or as a complete block of text. Streaming is currently only available for OpenAI models.
-* **Quick Prompts:** Modify existing quick prompts or create new ones. To create a new quick prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the quick prompt's text. 
-* **System Prompts:** Edit existing system prompts or create new ones. To create a new system prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the system prompt's text. Under *Contexts*, select where the system prompt should appear.
-+
-NOTE: To delete a custom prompt, open the *Name* drop-down menu, hover over the prompt you want to delete, and click the *X* that appears. You cannot delete the default prompts.
+It has the following tabs:
 
+* **Conversations:** When you open AI Assistant from certain pages, such as **Timeline** or **Alerts**, it defaults to the relevant conversation type. For each conversation type, choose the default System Prompt, the default connector, and the default model (if applicable). The **Streaming** setting controls whether AI Assistant's responses appear word-by-word (streamed), or as a complete block of text. Streaming is currently only available for OpenAI models.
+* **Connectors:** Manage all LLM connectors.
+* **System Prompts:** Edit existing System Prompts or create new ones. To create a new System Prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the System Prompt's text. Under *Contexts*, select where the System Prompt should appear.
+* **Quick Prompts:** Modify existing Quick Prompts or create new ones. To create a new Quick Prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the Quick Prompt's text. 
 * **Anonymization:** Select fields to include as plaintext, to obfuscate, and to not send when you provide events to AI Assistant as context. <<ai-assistant-anonymization, Learn more>>.
-
-* **Knowledge base:** Provide additional context to AI Assistant so it can answer questions about {esql} and alerts in your environment. <<ai-assistant-knowledge-base, Learn more>>.
+* **Knowledge base:** Provide additional context to AI Assistant. <<ai-assistant-knowledge-base, Learn more>>.
 
 [discrete]
 [[ai-assistant-anonymization]]
@@ -131,7 +123,9 @@ NOTE: To delete a custom prompt, open the *Name* drop-down menu, hover over the
 To modify Anonymization settings, you need the **Elastic AI Assistant: All** privilege, with **Customize sub-feature privileges** enabled.
 --
 
-The **Anonymization** tab of the AI Assistant settings menu allows you to define default data anonymization behavior for events you send to AI Assistant. Fields with **Allowed** toggled on are included in events provided to AI Assistant. **Allowed** fields with **Anonymized** set to **Yes** are included, but with their values obfuscated.
+The **Anonymization** tab of the Security AI settings menu allows you to define default data anonymization behavior for events you send to AI Assistant. Fields with **Allowed** toggled on are included in events provided to AI Assistant. **Allowed** fields with **Anonymized** set to **Yes** are included, but with their values obfuscated.
+
+NOTE: You can access anonymization settings directly from the **Attack Discovery** page by clicking the settings (image:images/icon-settings.png[Settings icon,17,17]) button next to the model selection dropdown menu.
 
 [role="screenshot"]
 image::images/assistant-anonymization-menu.png[AI Assistant's settings menu, open to the Anonymization tab]
@@ -143,49 +137,17 @@ The *Show anonymized* toggle controls whether you see the obfuscated or plaintex
 When you include a particular event as context, such as an alert from the Alerts page, you can adjust anonymization behavior for the specific event. Be sure the anonymization behavior meets your specifications before sending a message with the event attached.
 
 [discrete]
-[[ai-assistant-knowledge-base]]
+[[ai-assistant-page-knowledge-base]]
 === Knowledge base
-beta::[]
-
-The **Knowledge base** tab of the AI Assistant settings menu allows you to enable AI Assistant to answer questions about the Elastic Search Query Language ({esql}), and about alerts in your environment. To use knowledge base, you must <<ml-requirements, enable machine learning>>.
-
-[discrete]
-[[rag-for-esql]]
-==== Knowledge base for {esql}
 
-NOTE: {esql} is enabled by default in {kib}. It can be
-disabled using the `enableESQL` setting from the
-{kibana-ref}/advanced-options.html[Advanced Settings]. This will hide the {esql} user interface from various applications. However, users will be able to access existing {esql} artifacts like saved searches and visualizations.
-
-IMPORTANT: {esql} queries generated by AI Assistant might require additional validation. To ensure they're correct, refer to the {ref}/esql-language.html[{esql} documentation]. 
-
-When this feature is enabled, AI Assistant can help you write an {esql} query for a particular use case, or answer general questions about {esql} syntax and usage. To enable AI Assistant to answer questions about {esql}:
-
-. Turn on the knowledge base by clicking **Setup**. If the **Setup** button doesn't appear, knowledge base is already enabled.
-. Click *Save*. The knowledge base is now active. A quick prompt for {esql} queries becomes available, which provides a good starting point for your {esql} conversations and questions. 
-
-NOTE: AI Assistant's knowledge base gets additional context from {ml-docs}/ml-nlp-elser.html#download-deploy-elser[Elastic Learned Sparse EncodeR (ELSER)].
-
-[discrete]
-[[rag-for-alerts]]
-==== Knowledge base for alerts
-When this feature is enabled, AI Assistant will receive multiple alerts as context for each of your prompts. It will receive alerts from the last 24 hours that have a status of `open` or `acknowledged`, ordered first by risk score, then by recency. Building block alerts are excluded. This enables it to answer questions about multiple alerts in your environment, rather than just the individual alerts you choose to include as context. 
-
-To enable RAG for alerts:
-
-. Turn on the knowledge base by clicking **Setup**. If the **Setup** button doesn't appear, knowledge base is already enabled.
-. Use the slider to select the number of alerts to send to AI Assistant. Click **Save**.
-+
-[role="screenshot"]
-image::images/knowledge-base-settings.png["AI Assistant's settings menu open to the Knowledge Base tab",75%]
-
-NOTE: Including a large number of alerts may cause your request to exceed the maximum token length of your third-party generative AI provider. If this happens, try selecting a lower number of alerts to send.
+The **Knowledge base** tab of the **Security AI settings** page allows you to enable AI Assistant to remember specified information, and use it as context to improve response quality. To learn more, refer to <<ai-assistant-knowledge-base>>.
 
 [discrete]
 [[ai-assistant-queries]]
+[[rag-for-esql]]
 ### Get the most from your queries
 
-Elastic AI Assistant helps you take full advantage of the {elastic-sec} platform to improve your security operations. Its ability to assist you depends on the specificity and detail of your questions. The more context and detail you provide, the more tailored and useful its responses will be. 
+Elastic AI Assistant allows you to take full advantage of the {elastic-sec} platform to improve your security operations. It can help you write an {esql} query for a particular use case, or answer general questions about how to use the platform. Its ability to assist you depends on the specificity and detail of your questions. The more context and detail you provide, the more tailored and useful its responses will be. 
 
 To maximize its usefulness, consider using more detailed prompts or asking for additional information. For instance, after asking for an {esql} query example, you could ask a follow-up question like, “Could you give me some other examples?” You can also ask for clarification or further exposition, for example "Please provide comments explaining the query you just gave."
 

docs/AI-for-security/attack-discovery.asciidoc

@@ -56,7 +56,7 @@ It may take from a few seconds up to several minutes to generate discoveries, de
 
 IMPORTANT: Attack discovery is in technical preview and will only analyze opened and acknowleged alerts from the past 24 hours. By default it only analyzes up to 20 alerts within this timeframe, but you can expand this up to 100 by going to **AI Assistant → Settings (image:images/icon-settings.png[Settings icon,17,17]) → Knowledge Base** and updating the **Alerts** setting.
 
-image::images/knowledge-base-settings.png["AI Assistant's settings menu open to the Knowledge Base tab",75%]
+image::images/knowledge-base-assistant-settings-kb-tab.png["AI Assistant's settings menu open to the Knowledge Base tab",75%]
 
 IMPORTANT: Attack discovery uses the same data anonymization settings as <<security-assistant, Elastic AI Assistant>>. To configure which alert fields are sent to the LLM and which of those fields are obfuscated, use the Elastic AI Assistant settings. Consider the privacy policies of third-party LLMs before sending them sensitive data.