docs(agentic-trust): Analyze AI agents guide

docs/components/agentic-orchestration/agentic-orchestration-overview.md

@@ -7,7 +7,7 @@ keywords: ["agentic orchestration"]
 
 import OverviewImg from './img/ao-overview-hero.png';
 import AoGrid from '../react-components/\_ao-card';
-import { getStartedCards, fundamentalCards, recommendationCards } from '../react-components/\_ao-card-data';
+import { getStartedCards, fundamentalCards, fundamentalCards_monitoring, recommendationCards } from '../react-components/\_ao-card-data';
 
 <h3 class="subheading">Orchestrate and integrate artificial intelligence (AI) agents into your end-to-end processes.</h3>
 
@@ -38,7 +38,11 @@ Get started with Camunda agentic orchestration by building and running your firs
 
 Understand the fundamental concepts of Camunda agentic orchestration.
 
-<AoGrid ao={fundamentalCards} columns={3}/>
+<AoGrid ao={fundamentalCards} columns={2}/>
+
+Learn how to monitor and troubleshoot AI agents in real time with Operate, and analyze and improve performance over time with Optimize.
+
+<AoGrid ao={fundamentalCards_monitoring} columns={2}/>
 
 ## Explore further resources
 

docs/components/agentic-orchestration/analyze-ai-agents.md

@@ -0,0 +1,178 @@
+---
+id: analyze-ai-agents
+title: Analyze your AI agents with Optimize
+sidebar_label: Analyze your AI agents
+description: "Analyze and improve the performance of your AI agent executions using Optimize."
+keywords: ["agentic ai", "AI agents", "Optimize"]
+---
+
+import ScriptTask from './img/script-task.png';
+import Heatmap from './img/heatmap.png';
+import PieChart from './img/pie-chart.png';
+import Dashboard from './img/dashboard-optimize.png';
+
+Analyze and improve the performance of your AI agent executions using Optimize.
+
+## About
+
+In this guide, you will:
+
+- Understand what data Optimize can analyze for AI agents.
+- Create reports and heatmaps, including token and tool usage.
+- Build a dashboard to track AI agent behavior over time.
+
+After completing this guide, you will be able to analyze AI agent executions in Optimize and build dashboards that track usage and performance trends.
+
+## Prerequisites
+
+- You have access to [Optimize](/components/optimize/what-is-optimize.md).
+- You have deployed and run the [AI Agent Chat Quick Start](https://marketplace.camunda.com/en-US/apps/587865) model blueprint. This is needed for Optimize to fetch execution data to analyze. Consider running it using different prompts to trigger various AI agent tools.
+
+:::important
+This guide is a follow-up to [Build your first AI agent](../../guides/getting-started-agentic-orchestration.md), in which you use the same example AI agent process. We recommend completing that guide first. However, you can also apply this guide to other AI agent process implementations.
+:::
+
+## Step 1: Make your data available
+
+To make data available to Optimize, make sure it's scoped at the process level. If it's scoped to a lower level, for example, within a connector or tool-execution scope, extract it into process variables.
+
+:::important
+Optimize can only use variable data at the **process level**.
+:::
+
+### Example: Collect token usage
+
+In the AI Agent Chat Quick Start example, token usage data is not available at the process level, but in a nested scope. To surface those metrics, you can add a script task after the AI agent execution that copies those values into process variables.
+
+How you extract these values depends on how your AI agent implementation exposes token usage. The important aspect is that the target variables exist at the **process level** when the instance finishes.
+
+To do so:
+
+1. Add a [script task](/components/modeler/bpmn/script-tasks/script-tasks.md).
+1. Configure its **Properties** as follows:
+   - In **General**, **Name**: Gather metrics.
+   - In **Implementation**, select **FEEL expression**.
+   - In **Script**,
+     - **Result variable**: `tokenUsage`.
+     - **FEEL expression**: `agent.context.metrics.tokenUsage.inputTokenCount`.
+   - In **Output mapping**, add two process variables:
+     - `inputTokenUsage` with **Variable assignment value**: `agent.context.metrics.tokenUsage.inputTokenCount`.
+     - `outputTokenUsage` with **Variable assignment value**: `agent.context.metrics.tokenUsage.outputTokenCount`.
+
+You should see something like the following:
+
+<img src={ScriptTask} alt="Script task configuration for collecting token usage"/>
+
+## Step 2: Examine data in Optimize
+
+By leveraging data collected during process execution, you can use Optimize to examine it through reports and dashboards and identify areas for improvement in your AI agent processes.
+
+1. Open Optimize.
+1. Go to the **Dashboards** tab.
+1. Select your AI agent process, **AI Agent Chat With Tools**, in the **Process dashboards and KPIs** section.
+1. Verify that Optimize shows data for your executed process instances in the **Business Operations** section, including your AI agent process model diagram and other statistics below.
+1. Explore the other metrics shown below in the **Business Reporting** and **Process Improvement** sections.
+
+See the Optimize [getting started](/components/optimize/improve-processes-with-optimize.md) guide for more details on what you can do with Optimize for business intelligence.
+
+## Step 3: Create reports for token usage
+
+You can create reports for token usage across process instances and over time.
+
+To do so:
+
+1. Go to the **Collections** tab.
+1. Select **Report** from the **Create new** dropdown.
+1. Select **AI Agent Chat With Tools** from the **Select one or more processes** dropdown. You can fetch data for all process model versions or customize it.
+1. Choose a blank template.
+1. Enable **Update preview automatically** to make it easier to see the report results as you configure it.
+1. In the **Report setup** section, select **Variable** in the **View** option, and then select **tokenUsage**.
+1. Click the pencil icon and select an aggregation that matches your goal. For example:
+   - **Sum** to track total tokens across instances.
+   - **Average** to track typical usage per process instance.
+1. Save the report with a descriptive name. For example, **Token usage**.
+
+:::note
+You can create similar reports, targeting other goals and process variables, such as `inputTokenUsage` or `outputTokenUsage`.
+:::
+
+### Example: Set a target threshold
+
+If you have a token budget, you can set a target in the report.
+
+1. Follow [the previous steps 1 through 6](#step-3-create-reports-for-token-usage).
+1. In the **Visualization** settings, click the gear icon, and then enable **Set target** to configure a target value, for example, a maximum token usage threshold.
+1. Set the target to less than 10,000 tokens.
+1. Save the report with a descriptive name. For example, **Token usage with threshold**.
+
+## Step 4: Create reports for tool usage
+
+You can create reports for tool usage across process instances and over time.
+
+### Example: Create a heatmap
+
+Use a heatmap to understand how long your AI agent spends in each task.
+
+1. Go to the **Collections** tab.
+1. Select **Report** from the **Create new** dropdown.
+1. Select **Flow node** as the **View**.
+1. Select **Duration** as the **Measure**.
+1. (Optional) Filter the report by selecting **Flow node selection** in the **Filter flow nodes** dropdown. For example, select only tool tasks within the AI Agent connector.
+1. In the **Visualization** settings, select **Heatmap**. Then click the gear icon and enable the tooltip to show absolute values.
+1. Save the report with a descriptive name. For example, **Tool usage heatmap**.
+
+<p align="center">
+   <img src={Heatmap} alt="Tool usage heatmap" width="80%"/>
+</p>
+
+### Example: Create a report for tool call counts
+
+Create a bar chart to see how many times each tool is called.
+
+1. Go to the **Collections** tab.
+1. Select **Report** from the **Create new** dropdown.
+1. Select **Flow node** as **View**.
+1. Select **Count** as the **Measure**.
+1. (Optional) Filter the report by selecting **Flow node selection** in the **Filter flow nodes** dropdown. For example, select only tool tasks within the AI Agent connector.
+1. In the **Visualization** settings, select **Bar chart** or **Pie chart**. Then click the gear icon and enable both tooltips to show absolute and relative values.
+1. Save the report with a descriptive name. For example, **Tool usage**.
+
+<p align="center">
+  <img src={PieChart} alt="Tool usage pie chart" width="50%"/>
+</p>
+
+### Example: Track trends over time
+
+Use a timeline report to analyze trends over time. For example, you can see how many times a tool is called per day over a one-week period.
+
+1. Go to the **Collections** tab.
+1. Select **Report** from the **Create new** dropdown.
+1. Select **Flow node** as **View**.
+1. Select **Count** as the **Measure**.
+1. In **Group by**, select **Start date**. Then choose your preferred interval, for example, **Week**.
+1. (Optional) Filter the report by selecting **Flow node selection** in the **Filter flow nodes** dropdown. For example, select only tool tasks within the AI Agent connector.
+1. In the **Visualization** settings, select **Bar chart** or **Line chart**. Then click the gear icon and enable both tooltips to show absolute and relative values.
+1. Save the report with a descriptive name. For example, **Tool usage over time**.
+
+## Step 5: Build a dashboard
+
+You can create a dashboard, which is a collection of reports that you can view together, for your AI agent process.
+To do so:
+
+1. Go to the **Collections** tab.
+1. Select **Dashboard** from the **Create new** dropdown.
+1. Click the plus icon to add tiles for the reports you created.
+1. In the **Optimize report** section, add reports as needed.
+1. Arrange the tiles to customize your dashboard layout.
+1. Save the dashboard with a descriptive name. For example, **AI Agent Chat Quick Start dashboard**.
+
+<p align="center">
+  <img src={Dashboard} alt="Dashboard overview" width="80%"/>
+</p>
+
+## Next steps
+
+Now that you know how to analyze your AI agents, you can:
+
+- [Monitor your AI agents](/components/agentic-orchestration/monitor-ai-agents.md) with Operate.
+- Learn more about [Camunda agentic orchestration](/components/agentic-orchestration/agentic-orchestration-overview.md) and the [AI Agent connector](/components/connectors/out-of-the-box-connectors/agentic-ai-aiagent.md).

docs/components/agentic-orchestration/monitor-ai-agents.md

@@ -1,15 +1,18 @@
 ---
 id: monitor-ai-agents
-title: Monitor your AI agents
+title: Monitor your AI agents with Operate
 sidebar_label: Monitor your AI agents
-description: "Monitor and troubleshoot your AI agent process instances in Camunda 8 using Operate and Optimize."
-keywords: ["agentic ai", "AI agents"]
+description: "Monitor and troubleshoot your AI agent process instances in real time using Operate."
+keywords: ["agentic ai", "AI agents", "Operate"]
 ---
 
 import Tabs from "@theme/Tabs";
 import TabItem from "@theme/TabItem";
+import ProcessInstance from './img/process-instance-overview.png';
+import InstanceHistory from './img/instance-history.png';
+import Variables from './img/variables.png';
 
-Monitor and troubleshoot your AI agent process instances in Camunda 8 using Operate and Optimize.
+Monitor and troubleshoot your AI agent process instances in real time using Operate.
 
 ## About
 
@@ -18,7 +21,6 @@ In this guide, you will:
 - Inspect an AI agent process instance in Operate.
 - Understand agent’s tool usage and metadata such as tool call inputs and results.
 - Analyze the agent context and how it is stored.
-- Examine process data in Optimize.
 
 :::note
 Operate enables inspection of execution paths, tool usage, and agent metadata. However, certain runtime artifacts, such as document storage contents, may require additional configuration.
@@ -28,11 +30,11 @@ After completing this guide, you will be able to inspect, debug, and monitor AI
 
 ## Prerequisites
 
+- You have access to [Operate](/components/operate/operate-introduction.md).
 - You have the [AI Agent Chat Quick Start](https://marketplace.camunda.com/en-US/apps/587865) model blueprint deployed in [Modeler](/components/modeler/about-modeler.md).
-- You are familiar with [Operate](/components/operate/operate-introduction.md).
 
 :::important
-This guide is a follow-up to [Build your first AI agent](../../guides/getting-started-agentic-orchestration.md), where you will use the same example AI agent process. Therefore, the same [prerequisites apply](../..//guides/getting-started-agentic-orchestration.md#prerequisites). However, it can be applied to other AI agent process implementations.
+This guide is a follow-up to [Build your first AI agent](../../guides/getting-started-agentic-orchestration.md), where you will use the same example AI agent process. It is recommended going through that guide first. However, it can be applied to other AI agent process implementations.
 :::
 
 ## Step 1: Run your AI agent process
@@ -49,20 +51,25 @@ For example:
 2. Locate the process instance created by your prompt. See [View a deployed process](/components/operate/userguide/basic-operate-navigation.md#view-a-deployed-process) for more details.
 3. Open your process instance view by clicking on its process instance key.
 
-At this point, you should see the process progressing through your model.
+At this point, you should see the process progressing through your model:
+
+<img src={ProcessInstance} alt="Process instance overview"/>
 
 ## Step 3: Understand what Operate shows
 
 With Operate, you can track the agent activity and see which tool tasks are called.
 
-1. To show how many times each BPMN element is triggered, select **Execution count**. For this particular prompt example, you can see:
+1. To show how many times each BPMN element is triggered, select **Execution count** in the **Instance History** section. For this particular prompt example, you can see:
    - The AI Agent connector was triggered once.
    - Within it, the agent executed the **Fetch URL** tool. This aligns with your prompt example of extracting information from a website.
 
 2. Select the **Fetch URL** tool element:
-   - In the bottom-left pane, you can see where the element belongs in the execution tree.
-   - In the bottom-right pane, the element details are displayed, including the [**Variables**](components/concepts/variables.md) and [**Input/Output Mappings**](/components/concepts/variables.md#inputoutput-variable-mappings) columns, among others.
-     However, the actual tool inputs and results are stored in a **parent scope** and are accessible via the element's inner instance in the execution tree.
+   - In the bottom-left pane, you can see where the element belongs in the execution tree:
+     <img src={InstanceHistory} alt="Fetch URL element details"/>
+
+- In the bottom-right pane, the element details are displayed, including the [**Variables**](components/concepts/variables.md) and [**Input/Output Mappings**](/components/concepts/variables.md#inputoutput-variable-mappings) columns, among others.
+  However, the actual tool inputs and results are stored in a **parent scope** and are accessible via the element's inner instance in the execution tree.
+  <img src={Variables} alt="Fetch URL element details"/>
 
 ## Step 4: Inspect tool calls
 
@@ -128,7 +135,7 @@ Here's the example conversation stored in the agent's context:
     "content": [
       {
         "type": "text",
-        "text": "You are a helpful, generic chat agent which can answer a wide amount of questions based on your knowledge and an optional set of available tools.\n\nIf tools are provided, you should prefer them instead of guessing an answer. You can call the same tool multiple times by providing different input values. Don't guess any tools which were not explicitely configured. If no tool matches the request, try to generate an answer. If you're not able to find a good answer, return with a message stating why you're not able to.\n\nIf you are prompted to interact with a person, never guess contact details, but use available user/person lookup tools instead and return with an error if you're not able to look up appropriate data.\n\nThinking, step by step, before you execute your tools, you think using the template `<thinking><context></context><reflection></reflection></thinking>`"
+        "text": "You are a helpful, generic chat agent which can answer a wide amount of questions based on your knowledge and an optional set of available tools.\n\nIf tools are provided, you should prefer them instead of guessing an answer. You can call the same tool multiple times by providing different input values. Don't guess any tools which were not explicitly configured. If no tool matches the request, try to generate an answer. If you're not able to find a good answer, return with a message stating why you're not able to.\n\nIf you are prompted to interact with a person, never guess contact details, but use available user/person lookup tools instead and return with an error if you're not able to look up appropriate data.\n\nThinking, step by step, before you execute your tools, you think using the template `<thinking><context></context><reflection></reflection></thinking>`"
       }
     ]
   },
@@ -184,19 +191,7 @@ See [Memory](/components/connectors/out-of-the-box-connectors/agentic-ai-aiagent
 
 :::
 
-## Step 7: Examine process data in Optimize
-
-By leveraging data collected during process execution, you can use [Optimize](/components/optimize/what-is-optimize.md) to explore reports and process dashboards, and identify areas for improvement in your AI agent processes.
-
-1. Open Optimize.
-2. Go to the **Dashboards** tab.
-3. Select your AI agent process, **AI Agent Chat With Tools**.
-4. Verify that Optimize shows data for the recently executed process instance in the **Business Operations** section, including the running process model diagram and other statistics below.
-5. Explore other metrics provided in the **Business Reporting** and **Process Improvement** sections.
-
-See [Getting started](/components/optimize/improve-processes-with-optimize.md) for more details on what you can do with Optimize for business intelligence.
-
-## Step 8: Review the results
+## Step 7: Review the results
 
 Go back to Operate. In the **User Feedback** element, you will see the execution count in green. This means the process instance execution is stopped there and waiting for action.
 
@@ -214,5 +209,5 @@ In this case, the required action is to provide feedback on the agent results. T
 
 Now that you know how to monitor your AI agents, you can:
 
+- [Analyze your AI agents](./analyze-ai-agents.md) with Optimize.
 - Learn more about [Camunda agentic orchestration](/components/agentic-orchestration/agentic-orchestration-overview.md) and the [AI Agent connector](/components/connectors/out-of-the-box-connectors/agentic-ai-aiagent.md).
-- Explore other [AI agent model blueprints](https://marketplace.camunda.com/en-US/listing?q=ai&cat=107793&locale=en-US) from Camunda marketplace.

docs/components/react-components/_ao-card-data.js

@@ -2,7 +2,9 @@ import IconAoImg from "../assets/icon-play.png";
 import IconAoDesignImg from "../assets/icon-docs.png";
 import IconAoAgentImg from "../assets/icon-agentic.png";
 import IconAoLlmImg from "../assets/icon-llm.png";
+import IconOptimize from "../assets/icon-optimize.png";
 import IconConnectorImg from "../connectors/img/icon-connectors.png";
+import IconOperate from "../assets/icon-operate.png";
 
 // Get started cards on introduction page
 export const getStartedCards = [
@@ -29,12 +31,22 @@ export const fundamentalCards = [
     image: IconAoDesignImg,
     description: "Plan and design your agentic orchestration solutions.",
   },
+];
+
+// Fundamentals cards on introduction page
+export const fundamentalCards_monitoring = [
   {
     link: "../monitor-ai-agents/",
     title: "Monitor your AI agents",
-    image: IconAoAgentImg,
+    image: IconOperate,
     description: "Monitor your AI agents with Operate.",
   },
+  {
+    link: "../analyze-ai-agents/",
+    title: "Analyze your AI agents",
+    image: IconOptimize,
+    description: "Analyze your AI agents with Optimize.",
+  },
 ];
 
 // Recommendation cards on introduction page