-
Notifications
You must be signed in to change notification settings - Fork 229
docs(agentic-orchestration): add monitor agents #7983
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
54be879
6f7915a
9dd6a38
35f4a81
def9cbb
8503b69
e642210
2a97c43
1f6e0b2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,208 @@ | ||
| --- | ||
| id: monitor-agents-operate | ||
| title: Monitor your AI agents with Operate | ||
| sidebar_label: Monitor your AI agents | ||
| description: "Monitor and troubleshoot your AI agent process instances in Camunda 8 using Operate." | ||
| keywords: ["agentic ai", "AI agents"] | ||
| --- | ||
|
|
||
| import Tabs from "@theme/Tabs"; | ||
| import TabItem from "@theme/TabItem"; | ||
|
|
||
| Monitor and troubleshoot your AI agent process instances in Camunda 8 using Operate. | ||
|
|
||
| ## About | ||
|
|
||
| In this guide, you will: | ||
|
|
||
| - Inspect an AI agent process instance in Operate. | ||
| - Understand agent’s tool usage and metadata such as tool call inputs and results. | ||
| - Analyze the agent context and how it is stored. | ||
|
|
||
| :::note | ||
| Operate enables inspection of execution paths, tool usage, and agent metadata. However, certain runtime artifacts, such as document storage contents, may require additional configuration. | ||
| ::: | ||
|
|
||
| After completing this guide, you will be able to inspect, debug, and monitor AI agent executions in Camunda 8. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - You have the [AI Agent Chat Quick Start](https://marketplace.camunda.com/en-US/apps/587865) model blueprint deployed in [Modeler](/components/modeler/about-modeler.md). | ||
| - You are familiar with [Operate](/components/operate/operate-introduction.md). | ||
|
|
||
| :::important | ||
| This guide is a follow-up to [Build your first AI agent](../../guides/getting-started-agentic-orchestration.md), where you will use the same example AI agent process. Therefore, the same [prerequisites apply](../..//guides/getting-started-agentic-orchestration.md#prerequisites). However, it can be applied to other AI agent process implementations. | ||
| ::: | ||
|
|
||
| ## Step 1: Run your AI agent process | ||
|
|
||
| Run your process instance using a prompt to trigger the AI Agent connector. | ||
|
|
||
| For example, enter "Visit _docs.camunda.io_ and tell me about it" in the **How can I help you today?** field, and click **Start instance**. | ||
|
|
||
| ## Step 2: Open the process instance in Operate | ||
|
|
||
| 1. Open [Operate](/components/operate/operate-introduction.md). If working locally with Self-Managed, open it in your browser at http://localhost:8080/operate. | ||
| 2. Locate the process instance created by your prompt. See [View a deployed process](/components/operate/userguide/basic-operate-navigation.md#view-a-deployed-process) for more details. | ||
| 3. Open your process instance view by clicking on its process instance key. | ||
|
|
||
| At this point, you should see the process progressing through your model. | ||
|
|
||
| ## Step 3: Understand what Operate shows | ||
|
|
||
| With Operate, you can track the agent activity and see which tool tasks are called. | ||
|
|
||
| 1. To show how many times each BPMN element is triggered, select **Execution count**. | ||
|
|
||
| For this particular prompt example, you can see: | ||
|
|
||
| - The AI Agent connector was triggered once. | ||
| - Within it, the agent executed the **Fetch URL** tool. This aligns with your prompt example: Visit a website URL and extract information from it. | ||
|
|
||
| 2. Select the **Fetch URL** tool element. | ||
|
|
||
| - In the bottom-left pane, you can see where the element belongs in the execution tree. | ||
| - In the bottom-right pane, the element details are displayed, including the [**Variables**](components/concepts/variables.md) and [**Input/Output Mappings**](/components/concepts/variables.md#inputoutput-variable-mappings) columns, among others. | ||
| However, the actual tool inputs and results are stored in a **parent scope** and are accessible via the element's inner instance in the execution tree. | ||
|
|
||
| ## Step 4: Inspect tool calls | ||
|
|
||
| Each tool execution produces an inner instance where you can find: | ||
|
|
||
| - The inputs passed into the tool. | ||
| - The results. | ||
|
|
||
| To see the **Fetch URL** tool input and results: | ||
|
|
||
| 1. In the execution tree, select the **AI_Agent#innerInstance** parent element of the **Fetch URL** tool. You will see: | ||
|
|
||
| - The **toolCall** variable (the _input_). In its value field, you can see the URL you asked the agent to fetch information from. | ||
| - The **toolCallResult** variable (the _results_). See [Tool call responses](/components/connectors/out-of-the-box-connectors/agentic-ai-aiagent-tool-definitions.md#tool-call-responses) for more details. | ||
|
|
||
| 2. To better inspect the results, click the pencil icon to enter edit mode for **toolCallResult**. | ||
| 3. Click the two-squares icon to open the JSON editor modal. With this, you can inspect the full payload of the variable value. The result is a reference to a document where the actual output is stored. You will learn more about this in [Step 6](#step-6-understand-how-agent-memory-is-stored). | ||
|
|
||
| Here's the **toolCallResult** value: | ||
|
|
||
| ```json | ||
| { | ||
| "storeId": "gcp", | ||
| "documentId": "1aa0e4f6-7cee-4c19-ac2e-cebee3c7b151", | ||
| "contentHash": "649a8575bf8ced157fbbba7acc02ba33058bb2e3f40fb37b0f303730fb64fba0", | ||
| "metadata": { | ||
| "contentType": "text/html; charset=utf-8", | ||
| "size": 33722, | ||
| "fileName": "1aa0e4f6-7cee-4c19-ac2e-cebee3c7b151" | ||
| }, | ||
| "camunda.document.type": "camunda" | ||
| } | ||
| ``` | ||
|
|
||
| :::note | ||
| If a tool is executed more than once, select the desired tool invocation in **Instance History**, then open the corresponding inner instance to view the actual inputs and results. | ||
| ::: | ||
|
|
||
| ## Step 5: Analyze the agent context | ||
|
|
||
| Within the AI Agent connector, you can examine the agent context. | ||
| To view it: | ||
|
|
||
| 1. Select the **AI Agent** element in the execution tree. | ||
| 2. To better inspect the value, click the pencil icon to enter edit mode for the **agentContext** variable. | ||
| 3. Click the two-squares icon to open the JSON editor modal. With this, you can inspect the full payload of the variable value. | ||
|
|
||
| In the JSON payload, you can find information about: | ||
|
|
||
| - Defined tools. | ||
| - The conversation, including your prompts and agent's replies. | ||
| - Tool calls invoked by the agent. | ||
| - Tool call inputs and results. | ||
| - Additional metadata, such as reasoning traces and token usage. | ||
|
|
||
| Here's the example conversation stored in the agent's context: | ||
|
|
||
| ```json | ||
| "type": "in-process", | ||
| "conversationId": "130bbad7-b316-4cac-9653-a77e37e11f8b", | ||
| "messages": [ | ||
| { | ||
| "role": "system", | ||
| "content": [ | ||
| { | ||
| "type": "text", | ||
| "text": "You are a helpful, generic chat agent which can answer a wide amount of questions based on your knowledge and an optional set of available tools.\n\nIf tools are provided, you should prefer them instead of guessing an answer. You can call the same tool multiple times by providing different input values. Don't guess any tools which were not explicitely configured. If no tool matches the request, try to generate an answer. If you're not able to find a good answer, return with a message stating why you're not able to.\n\nIf you are prompted to interact with a person, never guess contact details, but use available user/person lookup tools instead and return with an error if you're not able to look up appropriate data.\n\nThinking, step by step, before you execute your tools, you think using the template `<thinking><context></context><reflection></reflection></thinking>`" | ||
| } | ||
| ] | ||
| }, | ||
| { | ||
| "role": "user", | ||
| "content": [ | ||
| { | ||
| "type": "text", | ||
| "text": "Visit https://docs.camunda.io/ and tell me about it" | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "timestamp": "2026-02-23T11:48:03.90923378Z" | ||
| } | ||
| }, | ||
| { | ||
| "role": "assistant", | ||
| "content": [ | ||
| { | ||
| "type": "text", | ||
| "text": "<thinking>\n<context>\nThe user wants me to visit a specific URL (https://docs.camunda.io/) and tell them about it. I have a tool called \"Fetch_URL\" that can fetch the contents of a given URL. This seems like a perfect match for the request.\n</context>\n<reflection>\nI should use the Fetch_URL tool to get the contents of https://docs.camunda.io/ and then analyze what I find to provide information about it to the user.\n</reflection>\n</thinking>" | ||
| } | ||
| ], | ||
| "toolCalls": [ | ||
| { | ||
| "id": "tooluse_kBcf61pS50OS31KYQnRFzy", | ||
| "name": "Fetch_URL", | ||
| "arguments": { | ||
| "url": "https://docs.camunda.io/" | ||
| } | ||
| } | ||
| ], | ||
| ``` | ||
|
|
||
| ## Step 6: Understand how agent memory is stored | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Suggestion: Can we explicitly clarify the trade-off between visibility and scalability? “In Process” → better Operate visibility, limited scalability for long conversations.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This was added as a note box at the end of the Step 6 |
||
|
|
||
| In Modeler, within the AI Agent sub-process, you can define how the conversation memory is stored using the **Memory storage type** field. | ||
|
|
||
| By default, agent memory uses the **In Process** type, which stores it as part of the agent context. | ||
| With this option, you can view it in Operate within the agent context, as you did in the previous step, [Analyze the agent context](#step-5-analyze-the-agent-context). | ||
|
|
||
| With the **Camunda Document Storage** option instead: | ||
|
|
||
| - You can't view the full conversation and chain-of-thought traces in Operate. Operate only show a **document reference** and metadata. | ||
| - Use this option for long conversations, where Operate variable limits might be exceeded. | ||
|
|
||
| See [Memory](/components/connectors/out-of-the-box-connectors/agentic-ai-aiagent-subprocess.md#memory) for more details. | ||
|
|
||
| :::note Agent memory storage | ||
|
|
||
| - Use **In Process** for testing and debugging scenarios: Better visibility in Operate. | ||
| - Use **Camunda Document Storage** for production scenarios: Better scalability and runtime behavior for long contexts. | ||
|
|
||
| ::: | ||
|
|
||
| ## Step 7: Review the results | ||
|
|
||
| In the **User Feedback** element, you will see the execution count in green. This means the process instance execution is stopped there and waiting for action. | ||
|
|
||
| In this case, the required action is to provide feedback on the agent results. To do so: | ||
|
|
||
| 1. Select the **User Feedback** element. | ||
| 2. Open [Tasklist](/components/tasklist/introduction-to-tasklist.md). | ||
| 3. Select the User Feedback task and assign to yourself by clicking **Assign to me**. | ||
| 4. Analyze the result. You will see an overview of the website URL requested in the prompt. | ||
| 5. You can follow up with more prompts to continue testing your AI agent. | ||
| 6. Select the **Are you satisfied with the result?** checkbox when you want to finish the process, then click **Complete task**. | ||
| 7. Go back to Operate. You will see the process instance is now completed, and the end event has been triggered. | ||
|
|
||
| ## Next steps | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Suggestion: Right before Next Steps, based on enterprise production use cases, can we add a short section titled: Production Monitoring Recommendations For example:
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. SGTM, but I’d need more input on this. Also, regarding judge agents: they aren’t included in the example presented here, so I would keep this for another guide, or cover it when we provide it as part of a model blueprint that we can use as an example. |
||
|
|
||
| Now that you know how to monitor your AI agents with Operate, you can: | ||
|
|
||
| - Learn more about [Camunda agentic orchestration](/components/agentic-orchestration/agentic-orchestration-overview.md) and the [AI Agent connector](/components/connectors/out-of-the-box-connectors/agentic-ai-aiagent.md). | ||
| - Explore other [AI agent model blueprints](https://marketplace.camunda.com/en-US/listing?q=ai&cat=107793&locale=en-US) from Camunda marketplace. | ||
Uh oh!
There was an error while loading. Please reload this page.