Observability: Workflow traceability between executors and LLM calls #44479
Description
- Package Name: azure-ai-projects
- Package Version: 2.0.0b2
- Operating System:
- Python Version:
Description
There is currently no clear way to trace which LLM (agent) calls correspond to which executor.process invocation within a workflow run. While workflow execution steps and executor spans are visible, LLM-related spans appear as generic entries (e.g., AI unknown – Working: Processing request / Generating response) with no explicit linkage back to the executor or workflow step that triggered them.
This makes it difficult to debug, analyze latency, or attribute cost and failures to specific executors in complex or sequential workflows.
To Reproduce
Steps to reproduce the behavior:
Run a multi-step or sequential workflow that invokes one or more agents/LLMs via executors.
Open the workflow run trace view.
Observe multiple executor.process spans alongside AI unknown LLM spans.
Attempt to identify which LLM call belongs to which executor invocation.
Expected behavior
Each LLM / agent call should be traceable back to the specific executor (and ideally workflow step / node) that initiated it.
This would allow users to clearly answer: “Which executor triggered this LLM call?”
Screenshots
See attached trace screenshot showing multiple executor.process spans and uncorrelated AI unknown LLM spans.
Additional context
Today, LLM spans appear disconnected from the workflow execution graph, making performance debugging and workflow observability difficult—especially when multiple executors invoke the same agent in the same run. Improving trace correlation would significantly help with debugging, optimization, and cost attribution in agentic workflows.