docs(agentic-trust): Analyze AI agents guide#8383
Conversation
afgambin
added
agentic orchestration
component:docs
|
👋 🤖 🤔 Hello, @christinaausley! Did you make your changes in all the right places? These files were changed only in docs/. You might want to duplicate these changes in versioned_docs/version-8.8/.
You may have done this intentionally, but we wanted to point it out in case you didn't. You can read more about the versioning within our docs in our documentation guidelines. |
camunda-docs-pr-automation
bot
moved this from 🏗 In Progress
to 👀 In Review
in Documentation Team
| ```json | ||
| { | ||
| "storeId": "gcp", | ||
| "documentId": "1aa0e4f6-7cee-4c19-ac2e-cebee3c7b151", | ||
| "contentHash": "649a8575bf8ced157fbbba7acc02ba33058bb2e3f40fb37b0f303730fb64fba0", |
There was a problem hiding this comment.
It's unrelated to this PR, but this example with the document response might be a bit hard to grasp. Maybe we can use another tool call result here showing the actual tool call result in plain text? Or a Fetch URL call where "store response" is not enabled?
|
|
||
| How you extract these values depends on how your AI agent implementation exposes token usage. The important aspect is that the target variables exist at the **process level** when the instance finishes. | ||
|
|
||
| To do so: |
There was a problem hiding this comment.
Getting the token counts from the agent context only works with the AI Agent Process when the "Include agent context" option is enabled, because otherwise the context won't be part of the agent response - should we mention this somewhere?
| - In **Script**, | ||
| - **Result variable**: `tokenUsage`. | ||
| - **FEEL expression**: `agent.context.metrics.tokenUsage.inputTokenCount`. | ||
| - In **Output mapping**, add two process variables: | ||
| - `inputTokenUsage` with **Variable assignment value**: `agent.context.metrics.tokenUsage.inputTokenCount`. | ||
| - `outputTokenUsage` with **Variable assignment value**: `agent.context.metrics.tokenUsage.outputTokenCount`. |
There was a problem hiding this comment.
Disclaimer: I don't know much about Optimize, so I might miss something here.
I'm not sure if I completely understand why we refer to inputTokenCount twice here, once in the result variable and once in the output mapping. Is this because we need to create 2 distinct global variables, but the script task needs an implementation?
Could we potentially create a result variable tokenUsage (so a FEEL implementation exists) mapped to agent.context.metrics.tokenUsage and 2 output mappings:
inputTokenUsage- >tokenUsage.inputTokenUsageoutputTokenUsage->tokenUsage.outputTokenUsage
christinaausley
left a comment
christinaausley
left a comment
There was a problem hiding this comment.
Really great work here! Very minor adjustments in my latest commit, and two non-blocking comments.
| - **FEEL expression**: `agent.context.metrics.tokenUsage.inputTokenCount`. | ||
| - In **Output mapping**, add two process variables: | ||
| - `inputTokenUsage` with **Variable assignment value**: `agent.context.metrics.tokenUsage.inputTokenCount`. | ||
| - `outputTokenUsage` with **Variable assignment value**: `agent.context.metrics.tokenUsage.outputTokenCount`. |
There was a problem hiding this comment.
tokenUsage is set to inputTokenCount, which may read like total token usage later in the guide? If the intent is total usage, we could consider renaming the variable or adjusting the example? Optional.
|
|
||
| 1. Follow [the previous steps 1 through 6](#step-3-create-reports-for-token-usage). | ||
| 1. In the **Visualization** settings, click the gear icon, and then enable **Set target** to configure a target value, for example, a maximum token usage threshold. | ||
| 1. Set the target to less than 10,000 tokens. |
There was a problem hiding this comment.
If Optimize uses a specific target operator or threshold mode here, we could consider naming it explicitly instead of implying "less than" behavior but I'm not sure of the mechanics here -- like would "Set the target to 10,000 tokens or to the threshold that matches your budget." even really work? Feel free to disregard but just a thought.
|
The preview environment relating to the commit b48a7d5 has successfully been deployed. You can access it at https://preview.docs.camunda.cloud/pr-8383/ |
4 participants
Description
This PR adds a new guide documenting how to analyze AI agents with Optimize. It is part of this epic.
When should this change go live?
bugorsupportlabel)available & undocumentedlabel)holdlabel)low priolabel)PR Checklist
{type}(scope): {description}commit message(s)/docsdirectory (version 8.9)./versioned_docsdirectory.@camunda/tech-writersunless working with an embedded writer.