Merge pull request #1356 from lgayhardt/release-ignite-ai-studio

[Azure AI Svcs] AI Studio: Visualize your traces

Author: v-dirichards

articles/ai-studio/concepts/trace.md

@@ -0,0 +1,71 @@
+---
+title: Tracing in Azure AI Inference SDK
+titleSuffix: Azure AI Studio
+description: This article provides an overview of tracing with the Azure AI Inference SDK.
+manager: scottpolly
+ms.service: azure-ai-studio
+ms.topic: conceptual
+ms.date: 11/19/2024
+ms.reviewer: truptiparkar
+ms.author: lagayhar  
+author: lgayhardt
+---
+
+# Tracing in Azure AI Inference SDK overview
+
+[!INCLUDE [feature-preview](../includes/feature-preview.md)]
+
+Tracing is a powerful tool that offers developers an in-depth understanding of the execution process of their generative AI applications. It provides a detailed view of the execution flow of the application. This essential information proves critical while debugging complex applications or optimizing performance.
+
+Tracing with the Azure AI Inference SDK offers enhanced visibility and simplified troubleshooting for LLM-based applications, effectively supporting development, iteration, and production monitoring. Tracing follows the OpenTelemetry semantic conventions, capturing and visualizing the internal execution details of any AI application, enhancing the overall development experience.
+
+## Key features
+
+- **Enhanced Observability**: Offers clear insights into the Gen AI Application lifecycle.
+- **User-Centric Design**: Simplifies telemetry enablement, focusing on improving developer workflow and productivity.
+- **Seamless Instrumentation**: Seamlessly instruments Azure AI Inference API for enabling telemetry.
+- **OTEL based tracing for User-defined functions**: Allows adding local variables and intermediate results to trace decorator for detailed tracing capabilities for user defined functions.
+- **Secure Data Handling**: Provides options to prevent sensitive or large data logging as per open telemetry standards.
+- **Feedback Logging**: Users can collect & attach user feedback and evaluative data to enrich trace data with qualitative insights.
+
+## Concepts
+
+### Traces
+
+Traces record specific events or the state of an application during execution. It can include data about function calls, variable values, system events and more. Whether your application is a monolith with a single database or a sophisticated mesh of services, traces are essential to understanding the full "path" a request takes in your application. To learn more, see [OpenTelemetry Traces](https://opentelemetry.io/docs/concepts/signals/traces/).
+
+### Semantic conventions
+
+OpenTelemetry defines Semantic Conventions, sometimes called Semantic attributes, that specify common names for different kinds of operations and data. The benefit of using Semantic conventions is in following a common naming scheme that can be standardized across a codebase, libraries, and platforms. By adhering to these conventions, Azure AI ensures that trace data is consistent and can be easily interpreted by observability tools. This consistency is crucial for effective monitoring, debugging, and optimization of Gen AI applications. To learn more, see [OpenTelemetry's Semantic Conventions for Generative AI systems](https://opentelemetry.io/docs/specs/semconv/gen-ai/).
+
+### Spans
+
+Spans are the building blocks of traces. Each span represents a single operation within a trace, capturing the start and end time, and any attributes or metadata associated with the operation. Spans can be nested to represent hierarchical relationships, allowing developers to see the full call stack and understand the sequence of operations. To learn more, see [OpenTelemetry's Spans](https://opentelemetry.io/docs/concepts/signals/traces/#spans).
+
+### Attributes
+
+Attributes are key-value pairs that provide additional information about a trace or span. Attributes can be used to record contextual data such as function parameters, return values, or custom annotations. This metadata enriches the trace data, making it more informative and useful for analysis.
+
+Attributes have the following rules that each language SDK implements:
+
+- Keys must be non-null string values.
+- Values must be a non-null string, boolean, floating point value, integer, or an array of these values.
+
+To learn more, see [OpenTelemetry's Attributes](https://opentelemetry.io/docs/concepts/signals/traces/#attributes).
+
+### Trace exporters
+
+Trace exporters are responsible for sending trace data to a backend system for storage and analysis. Azure AI supports exporting traces to various observability platforms, including Azure Monitor and other OpenTelemetry-compatible backends.
+
+### Trace visualization
+
+Trace visualization refers to the graphical representation of trace data. Azure AI integrates with visualization tools like Azure AI Studio Tracing, Aspire dashboard, and Prompty Trace viewer  to provide developers with an intuitive way to explore and analyze traces, helping them to quickly identify issues and understand the behavior of their applications.
+
+## Conclusion
+
+Azure AI's tracing capabilities are designed to empower developers with the tools they need to gain deep insights into their AI applications. By providing a robust, intuitive, and scalable tracing feature, Azure AI helps reduce debugging time, enhance application reliability, and improve overall performance. With a focus on user experience and system observability, Azure AI's tracing solution is set to revolutionize the way developers interact with and understand their Gen AI applications.
+
+## Related content
+
+- [Trace your application with Azure AI Inference SDK](../how-to/develop/trace-local-sdk.md)
+- [Visualize your traces](../how-to/develop/visualize-traces.md)

articles/ai-studio/how-to/develop/visualize-traces.md

@@ -0,0 +1,108 @@
+---
+title: Visualize your traces
+titleSuffix: Azure AI Studio
+description: This article provides instructions on how to visualize your traces.
+manager: scottpolly
+ms.service: azure-ai-studio
+ms.topic: how-to
+ms.date: 11/19/2024
+ms.reviewer: amipatel
+ms.author: lagayhar  
+author: lgayhardt
+---
+
+# Visualize your traces
+
+[!INCLUDE [feature-preview](../../includes/feature-preview.md)]
+
+After instrumenting your application to log traces, let's walkthrough how you can view your traces in both local and cloud solutions to debug your application.
+
+## View your traces for local debugging
+
+To enable traces locally, you have two options:  
+
+1. Using **Prompty**, you can trace your application with the **Azure AI Inference SDK**, which offers enhanced visibility and simplified troubleshooting for LLM-based applications. This method follows the OpenTelemetry specification, capturing and visualizing the internal execution details of any AI application, thereby enhancing the overall development experience. To learn more, see [Debugging Prompty](https://prompty.ai/docs/getting-started/debugging-prompty).
+2. **Aspire Dashboard** : A free & open-source OpenTelemetry dashboard for deep insights into your apps on your local development machine. To learn more, see [Aspire Dashboard](https://aspiredashboard.com/#start ).
+
+## View your traces in Azure AI Foundry portal
+
+Before you can log to Azure AI Foundry portal, attach an Application Insights resource to your project.
+
+1. Navigate to your project in [Azure AI Foundry portal](https://ai.azure.com/).
+1. Select the **Tracing** page on the left hand side.
+1. Select **Create New** to attach a new Application Insights resource to your project.
+1. Supply a name and select **Create**.
+
+:::image type="content" source="../../media/trace/visualize/tracing-setup-overview.gif" alt-text="Animation of going to tracing and creating an Application Insight resource." lightbox="../../media/trace/visualize/tracing-setup-overview.gif":::
+
+Next, install the `opentelemetry` SDK:
+
+```python
+%pip install azure-monitor-opentelemetry
+```
+
+Now enable tracing with output to the console:
+
+```python
+import os
+from azure.monitor.opentelemetry import configure_azure_monitor
+
+os.environ['AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED'] = 'true'
+# Enable Azure Monitor tracing
+application_insights_connection_string = project.telemetry.get_connection_string()
+if not application_insights_connection_string:
+    print("Application Insights was not enabled for this project.")
+    print("Enable it via the 'Tracing' tab in your AI Studio project page.")
+    exit()
+    
+configure_azure_monitor(connection_string=application_insights_connection_string)
+```
+
+Finally, run an inferencing call. The call is logged to Azure AI Studio. This code prints a link to the traces.
+
+```python
+response = chat.complete(
+    model="gpt-4o-mini",
+    messages=[
+        {"role": "system", "content": "You are an AI assistant that is a travel planning expert especially with National Parks."},
+        {"role": "user", "content": "Hey, can you recommend me trails I should go on when I visit Mount Rainier?"},
+    ]
+)
+
+print("View traces at:")
+print(f"https://ai.azure.com/tracing?wsid=/subscriptions/{project.scope['subscription_id']}/resourceGroups/{project.scope['resource_group_name']}/providers/Microsoft.MachineLearningServices/workspaces/{project.scope['project_name']}")
+```
+
+Select the link and begin viewing traces in Azure AI Studio!
+
+### Debug and filter traces
+
+In your project, you can filter your traces as you see fit.
+
+By selecting a trace, I can step through each span and identify issues while observing how my application is responding.
+
+:::image type="content" source="../../media/trace/visualize/debug-filter-tracing.gif" alt-text="Animation of filtering traces in the portal." lightbox="../../media/trace/visualize/debug-filter-tracing.gif":::
+
+### Update your attached Application Insights resource
+
+ To update the Application Insights resource that is attached to your project, go to **Manage data source** and **Edit** to switch to a new Application Insights resource.
+
+:::image type="content" source="../../media/trace/visualize/tracing-manage-data-source.png" alt-text="Screenshot of manage data sources pop-up highlighting the edit button." lightbox="../../media/trace/visualize/tracing-manage-data-source.png":::
+
+## View your traces in Azure Monitor
+
+If you logged traces using the previous code snippet, then you're all set to view your traces in Azure Monitor Application Insights. You can open in Application Insights from **Manage data source** and use the **End-to-end transaction details view** to further investigate.
+
+For more information on how to send Azure AI Inference traces to Azure Monitor and create Azure Monitor resource, see [Azure Monitor OpenTelemetry documentation](/azure/azure-monitor/app/opentelemetry-enable).
+
+### View your generative AI spans and traces
+
+From Azure AI studio project, you can also open your custom dashboard that provides you with insights specifically to help you monitor your generative AI application.
+
+In this Azure Workbook, you can view your Gen AI spans and jump into the Azure Monitor **End-to-end transaction details view** view to deep dive and investigate.
+
+Learn more about using this workbook to monitor your application, see [Azure Workbook documentation](/azure/azure-monitor/visualize/workbooks-create-workbook).
+
+## Related content
+
+- [Trace your application with Azure AI Inference SDK](./trace-local-sdk.md)

articles/ai-studio/media/trace/visualize/debug-filter-tracing.gif

@@ -305,9 +305,15 @@ items:
         href: how-to/develop/langchain.md
       - name: Develop with LlamaIndex
         href: how-to/develop/llama-index.md
-      - name: Trace your application with prompt flow
-        href: how-to/develop/trace-local-sdk.md
         displayName: code,sdk
+    - name: Trace generative AI apps
+      items:
+       - name: Tracing overview
+         href: concepts/trace.md 
+       - name: Trace your application with Azure AI Inference SDK
+         href: how-to/develop/trace-local-sdk.md
+       - name: Visualize your traces
+         href: how-to/develop/visualize-traces.md
   - name: Evaluate generative AI apps
     items:
     - name: Evaluations concepts