adding tracing for agents (Azure#50119)

* adding tracing for agents

* initial streaming tracing

* updates

* updates and review comment related changes

* adding tests

* updating changelog

* adding test recording, samples and updating readme

* updating tests

* updating tests

* review changes

* fixing readme

* updating readme based on review comments

Author: M-Hietala

sdk/ai/Azure.AI.Agents.Persistent/CHANGELOG.md

@@ -4,6 +4,8 @@
 
 ### Features Added
 
+- Tracing for Agents
+
 ### Breaking Changes
 
 ### Bugs Fixed

sdk/ai/Azure.AI.Agents.Persistent/README.md

@@ -35,6 +35,9 @@ managing search indexes, evaluating generative AI performance, and enabling Open
     - [Function call](#function-call)
     - [Azure function Call](#azure-function-call)
     - [OpenAPI](#create-agent-with-openapi)
+    - [Tracing](#tracing)
+      - [Azure Monitor Tracing](#tracing-to-azure-monitor)
+      - [Console Tracing](#tracing-to-console)
 - [Troubleshooting](#troubleshooting)
 - [Next steps](#next-steps)
 - [Contributing](#contributing)
@@ -866,6 +869,63 @@ Assert.AreEqual(
     run.LastError?.Message);
 ```
 `
+
+#### Tracing
+
+You can add an Application Insights Azure resource to your Azure AI Foundry project. See the Tracing tab in your AI Foundry project. If one was enabled, you use the Application Insights connection string, configure your Agents, and observe the full execution path through Azure Monitor. Typically, you might want to start tracing before you create an Agent.
+
+Tracing requires enabling OpenTelemetry support. One way to do this is to set the `AZURE_EXPERIMENTAL_ENABLE_ACTIVITY_SOURCE` environment variable value to `true`. You can also enable the feature with the following code:
+```C# Snippet:EnableActivitySourceToGetTraces
+AppContext.SetSwitch("Azure.Experimental.EnableActivitySource", true);
+```
+
+To enabled content recording, set the `AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED` environment variable to `true`. Content in this context refers to chat message content, function call tool related function names, function parameter names and values. Alternatively, you can control content recording with the following code:
+```C# Snippet:DisableContentRecordingForTraces
+AppContext.SetSwitch("Azure.Experimental.TraceGenAIMessageContent", false);
+```
+Set the value to `true` to enable content recording.
+
+##### Tracing to Azure Montior
+First, set the `APPLICATIONINSIGHTS_CONNECTION_STRING` environment variable to point to your Azure Monitor resource.
+
+For tracing to Azure Monitor from your application, the preferred option is to use Azure.Monitor.OpenTelemetry.AspNetCore. Install the package with [NuGet](https://www.nuget.org/ ):
+```dotnetcli
+dotnet add package Azure.Monitor.OpenTelemetry.AspNetCore
+```
+
+More information about using the Azure.Monitor.OpenTelemetry.AspNetCore package can be found [here](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/monitor/Azure.Monitor.OpenTelemetry.AspNetCore/README.md ).
+
+Another option is to use Azure.Monitor.OpenTelemetry.Exporter package. Install the pacakge with [NuGet](https://www.nuget.org/ )
+```dotnetcli
+dotnet add package Azure.Monitor.OpenTelemetry.Exporter
+```
+
+Here is an example how to set up tracing to Azure monitor using Azure.Monitor.OpenTelemetry.Exporter:
+```C# Snippet:AgentsTelemetrySetupTracingToAzureMonitor
+var tracerProvider = Sdk.CreateTracerProviderBuilder()
+    .AddSource("Azure.AI.Agents.Persistent.*")
+    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("AgentTracingSample"))
+    .AddAzureMonitorTraceExporter().Build();
+```
+
+##### Tracing to Console
+
+For tracing to console from your application, install the OpenTelemetry.Exporter.Console with [NuGet](https://www.nuget.org/ ):
+
+```dotnetcli
+dotnet add package OpenTelemetry.Exporter.Console
+```
+
+
+Here is an example how to set up tracing to console:
+```C# Snippet:AgentsTelemetrySetupTracingToConsole
+var tracerProvider = Sdk.CreateTracerProviderBuilder()
+                .AddSource("Azure.AI.Agents.Persistent.*") // Add the required sources name
+                .SetResourceBuilder(OpenTelemetry.Resources.ResourceBuilder.CreateDefault().AddService("AgentTracingSample"))
+                .AddConsoleExporter() // Export traces to the console
+                .Build();
+```
+
 ## Troubleshooting
 
 Any operation that fails will throw a [RequestFailedException][RequestFailedException]. The exception's `code` will hold the HTTP response status code. The exception's `message` contains a detailed message that may be helpful in diagnosing the issue:

sdk/ai/Azure.AI.Agents.Persistent/assets.json

@@ -2,5 +2,5 @@
   "AssetsRepo": "Azure/azure-sdk-assets",
   "AssetsRepoPrefixPath": "net",
   "TagPrefix": "net/ai/Azure.AI.Agents.Persistent",
-  "Tag": "net/ai/Azure.AI.Agents.Persistent_8e1aef00a7"
+  "Tag": "net/ai/Azure.AI.Agents.Persistent_2b964340aa"
 }

sdk/ai/Azure.AI.Agents.Persistent/src/Custom/Internal/ContinuationTokenPageable.cs

@@ -11,12 +11,23 @@
 using System.Threading.Tasks;
 using System.Text.Json;
 using System.Runtime.CompilerServices;
+using Azure.AI.Agents.Persistent.Telemetry;
 
 namespace Azure.AI.Agents.Persistent
 {
+    // Enum to specify the type of items in the pageable
+    internal enum ContinuationItemType
+    {
+        Undefined,
+        ThreadMessage,
+        RunStep
+    }
+
     internal class ContinuationTokenPageable<T>: Pageable<T>
     {
         private readonly ContinuationTokenPageableImpl<T> _impl;
+        private readonly ContinuationItemType _itemType;
+
         public ContinuationTokenPageable(
             Func<int?, string, HttpMessage> createPageRequest,
             Func<JsonElement, T> valueFactory,
@@ -26,9 +37,14 @@ public ContinuationTokenPageable(
             RequestContext requestContext,
             string itemPropertyName = "data",
             string hasMoreField = "has_more",
-            string continuationTokenName = "last_id"
+            string continuationTokenName = "last_id",
+            ContinuationItemType itemType = ContinuationItemType.Undefined,
+            string threadId = null,
+            string runId = null,
+            Uri endpoint = null
         )
         {
+            _itemType = itemType;
             _impl = new(
                 createPageRequest: createPageRequest,
                 valueFactory: valueFactory,
@@ -38,7 +54,11 @@ public ContinuationTokenPageable(
                 requestContext: requestContext,
                 itemPropertyName: itemPropertyName,
                 hasMoreField: hasMoreField,
-                continuationTokenName: continuationTokenName
+                continuationTokenName: continuationTokenName,
+                itemType: itemType,
+                threadId: threadId,
+                runId: runId,
+                endpoint: endpoint
             );
         }
 
@@ -48,6 +68,8 @@ public ContinuationTokenPageable(
     internal class ContinuationTokenPageableAsync<T> : AsyncPageable<T>
     {
         private readonly ContinuationTokenPageableImpl<T> _impl;
+        private readonly ContinuationItemType _itemType;
+
         public ContinuationTokenPageableAsync(
             Func<int?, string, HttpMessage> createPageRequest,
             Func<JsonElement, T> valueFactory,
@@ -57,9 +79,14 @@ public ContinuationTokenPageableAsync(
             RequestContext requestContext,
             string itemPropertyName = "data",
             string hasMoreField = "has_more",
-            string continuationTokenName = "last_id"
+            string continuationTokenName = "last_id",
+            ContinuationItemType itemType = ContinuationItemType.Undefined,
+            string threadId = null,
+            string runId = null,
+            Uri endpoint = null
         )
         {
+            _itemType = itemType;
             _impl = new(
                 createPageRequest: createPageRequest,
                 valueFactory: valueFactory,
@@ -69,7 +96,11 @@ public ContinuationTokenPageableAsync(
                 requestContext: requestContext,
                 itemPropertyName: itemPropertyName,
                 hasMoreField: hasMoreField,
-                continuationTokenName: continuationTokenName
+                continuationTokenName: continuationTokenName,
+                itemType: itemType,
+                threadId: threadId,
+                runId: runId,
+                endpoint: endpoint
             );
         }
 
@@ -91,6 +122,10 @@ internal class ContinuationTokenPageableImpl<T>
         private readonly string _itemPropertyName;
         private readonly string _hasMoreField;
         private readonly string _continuationTokenName;
+        private readonly ContinuationItemType _itemType;
+        private readonly string _threadId;
+        private readonly string _runId;
+        private readonly Uri _endpoint;
 
         public ContinuationTokenPageableImpl(
             Func<int?, string, HttpMessage> createPageRequest,
@@ -101,7 +136,11 @@ public ContinuationTokenPageableImpl(
             RequestContext requestContext,
             string itemPropertyName,
             string hasMoreField,
-            string continuationTokenName
+            string continuationTokenName,
+            ContinuationItemType itemType = ContinuationItemType.Undefined,
+            string threadId = null,
+            string runId = null,
+            Uri endpoint = null
         )
         {
             _createPageRequest = createPageRequest;
@@ -114,6 +153,10 @@ string continuationTokenName
             _itemPropertyName = itemPropertyName;
             _hasMoreField = hasMoreField;
             _continuationTokenName = continuationTokenName;
+            _itemType = itemType;
+            _threadId = threadId;
+            _runId = runId;
+            _endpoint = endpoint;
         }
 
         public async IAsyncEnumerator<T> GetAsyncEnumerator(CancellationToken cancellationToken = default)
@@ -186,45 +229,93 @@ private Response GetNextResponse(int? pageSizeHint, string continuationToken)
                 return null;
             }
 
-            using DiagnosticScope scope = _clientDiagnostics.CreateScope(_scopeName);
-            scope.Start();
-            try
+            // Use GenAI scope for paging if item type is specified
+            OpenTelemetryScope genAIScope = null;
+            DiagnosticScope? diagnosticsScope = null;
+            if (_itemType == ContinuationItemType.ThreadMessage)
+            {
+                genAIScope = OpenTelemetryScope.StartListMessages(_threadId, _runId, _endpoint);
+            }
+            else if (_itemType == ContinuationItemType.RunStep)
             {
-                _pipeline.Send(message, _cancellationToken);
-                return GetResponse(message);
+                genAIScope = OpenTelemetryScope.StartListRunSteps(_threadId, _runId, _endpoint);
             }
-            catch (Exception e)
+            else
             {
-                scope.Failed(e);
-                throw;
+                diagnosticsScope = _clientDiagnostics.CreateScope(_scopeName);
+            }
+
+            using (genAIScope)
+            {
+                using (diagnosticsScope)
+                {
+                    diagnosticsScope?.Start();
+                    try
+                    {
+                        _pipeline.Send(message, _cancellationToken);
+                        var response = GetResponse(message);
+                        genAIScope?.RecordPagedResponse(response);
+                        return response;
+                    }
+                    catch (Exception e)
+                    {
+                        diagnosticsScope?.Failed(e);
+                        genAIScope?.RecordError(e);
+                        throw;
+                    }
+                }
             }
         }
 
         private async ValueTask<Response> GetNextResponseAsync(int? pageSizeHint, string continuationToken, CancellationToken cancellationToken)
         {
             var message = CreateMessage(pageSizeHint, continuationToken);
 
-            using DiagnosticScope scope = _clientDiagnostics.CreateScope(_scopeName);
-            scope.Start();
-            try
+            // Use GenAI scope for paging if item type is specified
+            OpenTelemetryScope genAIScope = null;
+            DiagnosticScope? diagnosticsScope = null;
+            if (_itemType == ContinuationItemType.ThreadMessage)
             {
-                if (cancellationToken.CanBeCanceled && _cancellationToken.CanBeCanceled)
-                {
-                    using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, _cancellationToken);
-                    await _pipeline.SendAsync(message, cts.Token).ConfigureAwait(false);
-                }
-                else
-                {
-                    var ct = cancellationToken.CanBeCanceled ? cancellationToken : _cancellationToken;
-                    await _pipeline.SendAsync(message, ct).ConfigureAwait(false);
-                }
-
-                return GetResponse(message);
+                genAIScope = OpenTelemetryScope.StartListMessages(_threadId, _runId, _endpoint);
+            }
+            else if (_itemType == ContinuationItemType.RunStep)
+            {
+                genAIScope = OpenTelemetryScope.StartListRunSteps(_threadId, _runId, _endpoint);
             }
-            catch (Exception e)
+            else
             {
-                scope.Failed(e);
-                throw;
+                diagnosticsScope = _clientDiagnostics.CreateScope(_scopeName);
+            }
+
+            using (genAIScope)
+            {
+                using (diagnosticsScope)
+                {
+                    diagnosticsScope?.Start();
+                    try
+                    {
+                        if (cancellationToken.CanBeCanceled && _cancellationToken.CanBeCanceled)
+                        {
+                            using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, _cancellationToken);
+                            await _pipeline.SendAsync(message, cts.Token).ConfigureAwait(false);
+                        }
+                        else
+                        {
+                            var ct = cancellationToken.CanBeCanceled ? cancellationToken : _cancellationToken;
+                            await _pipeline.SendAsync(message, ct).ConfigureAwait(false);
+                        }
+
+                        var response = GetResponse(message);
+                        genAIScope?.RecordPagedResponse(response);
+                        return response;
+                    }
+                    catch (Exception e)
+                    {
+                        diagnosticsScope?.Failed(e);
+                        genAIScope?.RecordError(e);
+                        throw;
+                    }
+                }
             }
         }