Merge branch 'main' into 5771-defend-data-volume

Author: natasha-moore-elastic

.backportrc.json

@@ -1,5 +1,5 @@
 {
   "upstream": "elastic/security-docs",
-  "branches": ["8.x", "8.15", "8.14", "8.13", "8.12", "8.11", "8.10", "8.9", "8.8", "8.7", "8.6", "8.5", "8.4", "8.3", "8.2", "8.1", "8.0", "7.17", "7.16", "7.15", "7.14", "7.13", "7.12", "7.11", "7.10", "7.9", "7.8"],
+  "branches": ["8.x", "8.16", "8.15", "8.14", "8.13", "8.12", "8.11", "8.10", "8.9", "8.8", "8.7", "8.6", "8.5", "8.4", "8.3", "8.2", "8.1", "8.0", "7.17", "7.16", "7.15", "7.14", "7.13", "7.12", "7.11", "7.10", "7.9", "7.8"],
   "labels": ["backport"]
 }

.mergify.yml

@@ -13,6 +13,33 @@ pull_request_rules:
             git merge upstream/{{base}}
             git push upstream {{head}}
             ```
+  - name: backport patches to main branch
+    conditions:
+      - merged
+      - label=backport-main
+    actions:
+      backport:
+        assignees:
+          - "{{ author }}"
+        labels:
+          - "backport"
+        branches:
+          - "main"
+        title: "[{{ destination_branch }}] {{ title }} (backport #{{ number }})"
+  - name: backport patches to 8.17 branch
+    conditions:
+      - merged
+      - base=main
+      - label=v8.17.0
+    actions:
+      backport:
+        assignees:
+          - "{{ author }}"
+        branches:
+          - "8.x"
+        title: "[{{ destination_branch }}] {{ title }} (backport #{{ number }})"
+        labels:
+          - backport
   - name: backport patches to 8.16 branch
     conditions:
       - merged
@@ -23,7 +50,7 @@ pull_request_rules:
         assignees:
           - "{{ author }}"
         branches:
-          - "8.x"
+          - "8.16"
         title: "[{{ destination_branch }}] {{ title }} (backport #{{ number }})"
         labels:
           - backport

README.md

@@ -9,36 +9,32 @@ Documentation Manager: Janeen Roberts (Github: `@jmikell821`)
 
 ## Contributing to Elastic Security docs
 
-You can open an issue using the appropriate [template](https://github.com/elastic/security-docs/issues/new/choose). 
+You can open an issue using the appropriate [template](https://github.com/elastic/security-docs/issues/new/choose).
 
 > [!NOTE]
-> Please report any **known issues** that need to be documented by creating an issue in our [private repo](https://github.com/elastic/security-docs-internal/issues) using the known issue template. 
+> Please report any **known issues** that need to be documented by creating an issue in our [private repo](https://github.com/elastic/security-docs-internal/issues) using the known issue template.
 
 To contribute directly to Elastic Security documentation:
 
-1. Please fork and clone the `security-docs` repo. 
-1. Check out the `main` branch and fetch the latest changes. 
-1. Check out a new branch and make your changes. 
-1. Save your changes and open a pull request. 
-1. Add all appropriate Github users as reviewers.  
-1. Add the appropriate release version label, backport version label if appropriate, and team label to the PR. 
-1. If your PR changes any [serverless docs content](https://github.com/elastic/security-docs/tree/main/docs/serverless), add the label `ci:doc-build` to generate a preview of the serverless docs on the PR.
-1. Once the docs team approves all changes, you can merge it. If a backport version label was added to a PR for stack versions 7.14.0 and newer, mergify will automatically open a backport PR. 
-1. Merge the backport PR once it passes all CI checks. 
+1. Please fork and clone the `security-docs` repo.
+1. Check out the `main` branch and fetch the latest changes.
+1. Check out a new branch and make your changes.
+1. Save your changes and open a pull request.
+1. Add all appropriate Github users as reviewers.
+1. Add the appropriate release version label, backport version label if appropriate, and team label to the PR.
+1. Once the docs team approves all changes, you can merge it. If a backport version label was added to a PR for stack versions 7.14.0 and newer, mergify will automatically open a backport PR.
+1. Merge the backport PR once it passes all CI checks.
 
 ### Preview documentation changes
 
 When you open a pull request, preview links are automatically added as a comment in the PR. Once the CI check builds successfully, the links will be live and you can click them to preview your changes.
 
-For stateful docs, you also might want to add targeted links to help reviewers find specific pages related to your PR. Preview URLs include the following pattern (replace `<YOUR_PR_NUMBER_HERE>` with the PR number):
+You also might want to add targeted links to help reviewers find specific pages related to your PR. Preview URLs include the following pattern (replace `<YOUR_PR_NUMBER_HERE>` with the PR number):
 
 ```
 https://security-docs_bk_<YOUR_PR_NUMBER_HERE>.docs-preview.app.elstc.co/guide/en/security/master/
 ```
 
-> [!NOTE]
-> Serverless docs previews don't allow targeted links, because the id in the URL changes with each rebuild.
-
 ## License
 
 Shield: [![CC BY-NC-ND 4.0][cc-by-nc-nd-shield]][cc-by-nc-nd]

docs/AI-for-security/ai-for-security.asciidoc

@@ -1,5 +1,5 @@
 [[ai-for-security]]
-= AI for security
+= AI for Security
 
 :frontmatter-description: Learn to use AI capabilities in {elastic-sec}.
 :frontmatter-tags-products: [security]
@@ -9,6 +9,8 @@
 You can use {elastic-sec}'s built-in AI tools to speed up your work and augment your team's capabilities. The pages in this section describe <<security-assistant, AI Assistant>>, which answers questions and enhances your workflows throughout {elastic-sec}, and <<attack-discovery, Attack discovery>>, which speeds up the triage process by finding patterns and identifying attacks spanning multiple alerts.
 
 include::ai-security-assistant.asciidoc[leveloffset=+1]
+include::knowledge-base.asciidoc[leveloffset=+2]
+
 include::attack-discovery.asciidoc[leveloffset=+1]
 
 include::connector-guides-landing-pg.asciidoc[leveloffset=+1]

docs/AI-for-security/ai-security-assistant.asciidoc

@@ -24,7 +24,7 @@ WARNING: The Elastic AI Assistant is designed to enhance your analysis with smar
 
 * To set up AI Assistant, you need the **Actions and Connectors : All** {kibana-ref}/kibana-privileges.html[privilege].
 
-* You need an account with a third-party generative AI provider, which AI Assistant uses to generate responses. Supported providers are OpenAI, Azure OpenAI Service, and Amazon Bedrock.
+* You need a <<llm-connector-guides, generative AI connector>>, which AI Assistant uses to generate responses. 
 --
 
 [discrete]
@@ -68,27 +68,25 @@ You can also chat with AI Assistant from several particular pages in {elastic-se
 * <<data-quality-dash, Data Quality dashboard>>: Select the *Incompatible fields* tab, then click *Chat*. (This is only available for fields marked red, indicating they're incompatible).
 * <<timelines-ui, Timeline>>: Select the *Security Assistant* tab.
 
-NOTE: Each user's chat history and custom quick prompts are automatically saved, so you can leave {elastic-sec} and return to pick up a conversation later.
+NOTE: Each user's chat history (up to the 99 most recent conversations) and custom Quick Prompts are automatically saved, so you can leave {elastic-sec} and return to a conversation later. Chat history appears to the left of the AI Assistant chat window, and on the **Conversations** tab of the **AI Assistant settings** menu. To access the settings menu, use the global search field to search for "AI Assistant for Security".
 
 [discrete]
 [[interact-with-assistant]]
 == Interact with AI Assistant
 
 Use these features to adjust and act on your conversations with AI Assistant:
 
-* Select a _system prompt_ at the beginning of a conversation to establish how detailed and technical you want AI Assistant's answers to be.
+* (Optional) Select a _System Prompt_ at the beginning of a conversation by using the **Select Prompt** menu. System Prompts provide context to the model, informing its response. To create a System Prompt, open the System Prompts dropdown menu and click *+ Add new System Prompt...*.
+* (Optional) Select a _Quick Prompt_ at the bottom of the chat window to get help writing a prompt for a specific purpose, such as summarizing an alert or converting a query from a legacy SIEM to {elastic-sec}.
 +
 [role="screenshot"]
-image::images/system-prompt.gif[The system prompt drop-down menu,90%]
+image::images/quick-prompts.png[Quick Prompts highlighted below a conversation,90%]
 +
-System prompts provide context to the model, informing its response. To create a custom system prompt, open the system prompts dropdown menu and click *+ Add new system prompt...*.
-
-* Select a _quick prompt_ at the bottom of the chat window to get help writing a prompt for a specific purpose, such as summarizing an alert or converting a query from a legacy SIEM to {elastic-sec}.
+* System Prompts and Quick Prompts can also be configured from the corresponding tabs on the **Security AI settings** page.
 +
-[role="screenshot"]
-image::images/quick-prompts.png[Quick prompts highlighted below a conversation,90%]
+image::images/assistant-settings-system-prompts.png[The Security AI settings menu's System Prompts tab,90%]
 +
-Quick prompt availability varies based on context — for example, the **Alert summarization** quick prompt appears when you open AI Assistant while viewing an alert. To customize existing quick prompts and create new ones, click *Add Quick prompt*.
+* Quick Prompt availability varies based on context—for example, the **Alert summarization** Quick Prompt appears when you open AI Assistant while viewing an alert. To customize existing Quick Prompts and create new ones, click *Add Quick Prompt*.
 
 * In an active conversation, you can use the inline actions that appear on messages to incorporate AI Assistant's responses into your workflows:
 
@@ -104,22 +102,16 @@ TIP: AI Assistant can remember particular information you tell it to remember. F
 [discrete]
 [[configure-ai-assistant]]
 == Configure AI Assistant
-The *Settings* menu (image:images/icon-settings.png[Settings icon,17,17]) allows you to configure default conversations, quick prompts, system prompts, and data anonymization.
-
-[role="screenshot"]
-image::images/assistant-settings-menu.png[AI Assistant's settings menu, open to the Conversations tab]
-
-The *Settings* menu has the following tabs:
+The *Security AI settings* page allows you to configure AI Assistant. To access it, use the global search field to search for "AI Assistant for Security".
 
-* **Conversations:** When you open AI Assistant from certain pages, such as Timeline or Alerts, it defaults to the relevant conversation type. Choose the default system prompt for each conversation type, the connector, and model (if applicable). The **Streaming** setting controls whether AI Assistant's responses appear word-by-word (streamed), or as a complete block of text. Streaming is currently only available for OpenAI models.
-* **Quick Prompts:** Modify existing quick prompts or create new ones. To create a new quick prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the quick prompt's text. 
-* **System Prompts:** Edit existing system prompts or create new ones. To create a new system prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the system prompt's text. Under *Contexts*, select where the system prompt should appear.
-+
-NOTE: To delete a custom prompt, open the *Name* drop-down menu, hover over the prompt you want to delete, and click the *X* that appears. You cannot delete the default prompts.
+It has the following tabs:
 
+* **Conversations:** When you open AI Assistant from certain pages, such as **Timeline** or **Alerts**, it defaults to the relevant conversation type. For each conversation type, choose the default System Prompt, the default connector, and the default model (if applicable). The **Streaming** setting controls whether AI Assistant's responses appear word-by-word (streamed), or as a complete block of text. Streaming is currently only available for OpenAI models.
+* **Connectors:** Manage all LLM connectors.
+* **System Prompts:** Edit existing System Prompts or create new ones. To create a new System Prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the System Prompt's text. Under *Contexts*, select where the System Prompt should appear.
+* **Quick Prompts:** Modify existing Quick Prompts or create new ones. To create a new Quick Prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the Quick Prompt's text. 
 * **Anonymization:** Select fields to include as plaintext, to obfuscate, and to not send when you provide events to AI Assistant as context. <<ai-assistant-anonymization, Learn more>>.
-
-* **Knowledge base:** Provide additional context to AI Assistant so it can answer questions about {esql} and alerts in your environment. <<ai-assistant-knowledge-base, Learn more>>.
+* **Knowledge base:** Provide additional context to AI Assistant. <<ai-assistant-knowledge-base, Learn more>>.
 
 [discrete]
 [[ai-assistant-anonymization]]
@@ -131,7 +123,9 @@ NOTE: To delete a custom prompt, open the *Name* drop-down menu, hover over the
 To modify Anonymization settings, you need the **Elastic AI Assistant: All** privilege, with **Customize sub-feature privileges** enabled.
 --
 
-The **Anonymization** tab of the AI Assistant settings menu allows you to define default data anonymization behavior for events you send to AI Assistant. Fields with **Allowed** toggled on are included in events provided to AI Assistant. **Allowed** fields with **Anonymized** set to **Yes** are included, but with their values obfuscated.
+The **Anonymization** tab of the Security AI settings menu allows you to define default data anonymization behavior for events you send to AI Assistant. Fields with **Allowed** toggled on are included in events provided to AI Assistant. **Allowed** fields with **Anonymized** set to **Yes** are included, but with their values obfuscated.
+
+NOTE: You can access anonymization settings directly from the **Attack Discovery** page by clicking the settings (image:images/icon-settings.png[Settings icon,17,17]) button next to the model selection dropdown menu.
 
 [role="screenshot"]
 image::images/assistant-anonymization-menu.png[AI Assistant's settings menu, open to the Anonymization tab]
@@ -143,49 +137,17 @@ The *Show anonymized* toggle controls whether you see the obfuscated or plaintex
 When you include a particular event as context, such as an alert from the Alerts page, you can adjust anonymization behavior for the specific event. Be sure the anonymization behavior meets your specifications before sending a message with the event attached.
 
 [discrete]
-[[ai-assistant-knowledge-base]]
+[[ai-assistant-page-knowledge-base]]
 === Knowledge base
-beta::[]
-
-The **Knowledge base** tab of the AI Assistant settings menu allows you to enable AI Assistant to answer questions about the Elastic Search Query Language ({esql}), and about alerts in your environment. To use knowledge base, you must <<ml-requirements, enable machine learning>>.
-
-[discrete]
-[[rag-for-esql]]
-==== Knowledge base for {esql}
 
-NOTE: {esql} is enabled by default in {kib}. It can be
-disabled using the `enableESQL` setting from the
-{kibana-ref}/advanced-options.html[Advanced Settings]. This will hide the {esql} user interface from various applications. However, users will be able to access existing {esql} artifacts like saved searches and visualizations.
-
-IMPORTANT: {esql} queries generated by AI Assistant might require additional validation. To ensure they're correct, refer to the {ref}/esql-language.html[{esql} documentation]. 
-
-When this feature is enabled, AI Assistant can help you write an {esql} query for a particular use case, or answer general questions about {esql} syntax and usage. To enable AI Assistant to answer questions about {esql}:
-
-. Turn on the knowledge base by clicking **Setup**. If the **Setup** button doesn't appear, knowledge base is already enabled.
-. Click *Save*. The knowledge base is now active. A quick prompt for {esql} queries becomes available, which provides a good starting point for your {esql} conversations and questions. 
-
-NOTE: AI Assistant's knowledge base gets additional context from {ml-docs}/ml-nlp-elser.html#download-deploy-elser[Elastic Learned Sparse EncodeR (ELSER)].
-
-[discrete]
-[[rag-for-alerts]]
-==== Knowledge base for alerts
-When this feature is enabled, AI Assistant will receive multiple alerts as context for each of your prompts. It will receive alerts from the last 24 hours that have a status of `open` or `acknowledged`, ordered first by risk score, then by recency. Building block alerts are excluded. This enables it to answer questions about multiple alerts in your environment, rather than just the individual alerts you choose to include as context. 
-
-To enable RAG for alerts:
-
-. Turn on the knowledge base by clicking **Setup**. If the **Setup** button doesn't appear, knowledge base is already enabled.
-. Use the slider to select the number of alerts to send to AI Assistant. Click **Save**.
-+
-[role="screenshot"]
-image::images/knowledge-base-settings.png["AI Assistant's settings menu open to the Knowledge Base tab",75%]
-
-NOTE: Including a large number of alerts may cause your request to exceed the maximum token length of your third-party generative AI provider. If this happens, try selecting a lower number of alerts to send.
+The **Knowledge base** tab of the **Security AI settings** page allows you to enable AI Assistant to remember specified information, and use it as context to improve response quality. To learn more, refer to <<ai-assistant-knowledge-base>>.
 
 [discrete]
 [[ai-assistant-queries]]
+[[rag-for-esql]]
 ### Get the most from your queries
 
-Elastic AI Assistant helps you take full advantage of the {elastic-sec} platform to improve your security operations. Its ability to assist you depends on the specificity and detail of your questions. The more context and detail you provide, the more tailored and useful its responses will be. 
+Elastic AI Assistant allows you to take full advantage of the {elastic-sec} platform to improve your security operations. It can help you write an {esql} query for a particular use case, or answer general questions about how to use the platform. Its ability to assist you depends on the specificity and detail of your questions. The more context and detail you provide, the more tailored and useful its responses will be. 
 
 To maximize its usefulness, consider using more detailed prompts or asking for additional information. For instance, after asking for an {esql} query example, you could ask a follow-up question like, “Could you give me some other examples?” You can also ask for clarification or further exposition, for example "Please provide comments explaining the query you just gave."
 

docs/AI-for-security/api/anonymization-fields-api-find.asciidoc

@@ -1,6 +1,12 @@
 [[anonymization-fields-api-find]]
 === Find anonymization fields
 
+.New API Reference
+[sidebar]
+--
+For the most up-to-date API details, refer to {api-kibana}/group/endpoint-security-ai-assistant-api[AI Assistant APIs].
+--
+
 Retrieve a list of anonymization fields that can be included in the LLM context.
 
 [discrete]

docs/AI-for-security/api/assistant-api-overview.asciidoc

@@ -2,4 +2,10 @@
 [role="xpack"]
 == Elastic AI Assistant API
 
+.New API Reference
+[sidebar]
+--
+For the most up-to-date API details, refer to {api-kibana}/group/endpoint-security-ai-assistant-api[AI Assistant APIs].
+--
+
 The Elastic AI Assistant API allows you to interact with and manage Elastic AI Assistant.

docs/AI-for-security/api/bulk-actions-anonymization-fields-api.asciidoc

@@ -1,6 +1,12 @@
 [[bulk-actions-anonymization-fields-api]]
 === Bulk anonymization field actions
 
+.New API Reference
+[sidebar]
+--
+For the most up-to-date API details, refer to {api-kibana}/group/endpoint-security-ai-assistant-api[AI Assistant APIs].
+--
+
 Apply a bulk action (create, update, or delete) to multiple anonymization fields. The bulk action is applied to all anonymization fields that match the filter or to the list of anonymization fields by their IDs.
 
 [discrete]