AITK - Add Tracing (#8657)

ms-johnalex · web-flow · commit 691713623b11 · 2025-07-30T20:03:31.000+02:00
* Initial upload

* updated code samples and toc

* updated based on feedback

* Update based on feedback
diff --git a/docs/intelligentapps/agentbuilder.md b/docs/intelligentapps/agentbuilder.md
@@ -191,6 +191,16 @@ To view the Python code, follow these steps:
 
    > To authenticate with the model, you usually need an API key from the provider. To access models hosted by GitHub, [generate a personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) (PAT) in your GitHub settings.
 
+## What you learned
+
+In this article, you learned how to:
+
+- Use the AI Toolkit for VS Code to test and debug your agents.
+- Discover, configure, and build MCP servers to connect your agents to external APIs and services.
+- Set up function calling to connect your agents to external APIs and services.
+- Implement structured output to deliver predictable results from your agents.
+- Integrate prompt engineering into your application with generated code snippets.
+
 ## Next steps
 
 - [Run an evaluation job](/docs/intelligentapps/evaluation.md) for the popular evaluators
diff --git a/docs/intelligentapps/bulkrun.md b/docs/intelligentapps/bulkrun.md
@@ -84,6 +84,17 @@ You can:
 - **Edit Column Name**: Change the name of any column in the dataset.
 - **Add Ground Truth Column**: Add a column for ground truth values to compare with AI responses.
 
+## What you learned
+
+In this article, you learned how to:
+
+- Generate a synthetic dataset for bulk runs.
+- Import and export datasets in CSV format.
+- Run evaluations on bulk run results.
+- Mark responses as good or bad to keep a record of manual evaluations.
+- View details of responses and navigate between queries in the dataset.
+- Manage data columns for better analysis.
+
 ## Next steps
 
 - [Run an evaluation](/docs/intelligentapps/evaluation.md) with the popular evaluators
diff --git a/docs/intelligentapps/evaluation.md b/docs/intelligentapps/evaluation.md
@@ -176,3 +176,15 @@ def measure_the_response_if_human_like_or_not(query, **kwargs):
         "reason": "This is a placeholder for the evaluator's reason."
     }
 ```
+
+## What you learned
+
+In this article, you learned how to:
+
+- Create and run evaluation jobs in AI Toolkit for VS Code.
+- Monitor the status of evaluation jobs and view their results.
+- Compare evaluation results between different versions of prompts and agents.
+- View version history for prompts and agents.
+- Use built-in evaluators to measure performance with various metrics.
+- Create custom evaluators to extend the built-in evaluation capabilities.
+- Use LLM-based and code-based evaluators for different evaluation scenarios.
diff --git a/docs/intelligentapps/finetune.md b/docs/intelligentapps/finetune.md
@@ -299,3 +299,19 @@ For example:
   "ACA_APP_ENDPOINT": "<your-aca-endpoint>"
 }
 ```
+
+## What you learned
+
+In this article, you learned how to:
+
+- Set up the AI Toolkit for VS Code to support fine-tuning and inference in Azure Container Apps.
+- Create a fine-tuning project in AI Toolkit for VS Code.
+- Configure the fine-tuning workflow, including dataset selection and training parameters.
+- Run the fine-tuning workflow to adapt a pre-trained model to your specific dataset.
+- View the results of the fine-tuning process, including metrics and logs.
+- Use the sample notebook for model inference and testing.
+- Export and share the fine-tuning project with others.
+- Re-evaluate a model using different datasets or training parameters.
+- Handle failed jobs and adjust configurations for re-runs.
+- Understand the supported models and their requirements for fine-tuning.
+- Use the AI Toolkit for VS Code to manage fine-tuning projects, including provisioning Azure resources, running fine-tuning jobs, and deploying models for inference.
diff --git a/docs/intelligentapps/images/overview/tracing.png b/docs/intelligentapps/images/overview/tracing.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0e2ce30421d7a3d8565aa1bce4076875f9094cf8dd345215a4a60ffc4e45241
+size 188836
diff --git a/docs/intelligentapps/images/tracing/trace_details.png b/docs/intelligentapps/images/tracing/trace_details.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0e2ce30421d7a3d8565aa1bce4076875f9094cf8dd345215a4a60ffc4e45241
+size 188836
diff --git a/docs/intelligentapps/images/tracing/trace_list.png b/docs/intelligentapps/images/tracing/trace_list.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d500ae64b68f9edf834e59c5e889b4c4e74c34cbe27cd79f7c841f4dd990f4c
+size 170839
diff --git a/docs/intelligentapps/images/tracing/trace_start.png b/docs/intelligentapps/images/tracing/trace_start.png
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05f64ccf04f060c3caad2fd4c53d61f705e44a8999894876766aae99fb460616
+size 138009
diff --git a/docs/intelligentapps/modelconversion.md b/docs/intelligentapps/modelconversion.md
@@ -211,6 +211,20 @@ The default runtime is: `C:\Users\{user_name}\.aitk\bin\model_lab_runtime\Python
 
 Go to the History board. Select **Export** to share the model project with others. This copies the model project without history folder. If you want to share models with others, select the corresponding jobs. This copies the selected history folder containing the model and its configuration.
 
+## What you learned
+
+In this article, you learned how to:
+
+- Create a model conversion project in AI Toolkit for VS Code.
+- Configure the conversion workflow, including quantization and evaluation settings.
+- Run the conversion workflow to transform a pre-built model into an optimized ONNX model.
+- View the results of the conversion, including metrics and logs.
+- Use the sample notebook for model inference and testing.
+- Export and share the model project with others.
+- Re-evaluate a model using different execution providers or datasets.
+- Handle failed jobs and adjust configurations for re-runs.
+- Understand the supported models and their requirements for conversion and quantization.
+
 ## See also
 
 - [How to manually setup GPU conversion](/docs/intelligentapps/reference/ManualConversionOnGPU.md)
diff --git a/docs/intelligentapps/models.md b/docs/intelligentapps/models.md
@@ -167,3 +167,19 @@ You can manage your models in the **MY MODELS** section of the AI Toolkit view.
 ## License and sign-in
 
 Some models require a publisher or hosting-service license and account to sign-in. In that case, before you can run the model in the [model playground](/docs/intelligentapps/playground.md), you are prompted to provide this information.
+
+## What you learned
+
+In this article, you learned how to:
+
+- Explore and manage generative AI models in AI Toolkit.
+- Find models from various sources, including GitHub, ONNX, OpenAI, Anthropic, Google, Ollama, and custom endpoints.
+- Add models to your toolkit and deploy them to Azure AI Foundry.
+- Add custom models, including Ollama and OpenAI compatible models, and test them in the playground or agent builder.
+- Use the model catalog to view available models and select the best fit for your AI application needs.
+- Use filters and search to find models quickly.
+- Browse models by category, such as Popular, GitHub, ONNX, and Ollama.
+- Convert and add custom ONNX models using the model conversion tool.
+- Manage models in MY MODELS, including editing, deleting, refreshing, and viewing details.
+- Start and stop the ONNX server and copy endpoints for local models.
+- Handle license and sign-in requirements for some models before testing them.
diff --git a/docs/intelligentapps/overview.md b/docs/intelligentapps/overview.md
@@ -1,13 +1,13 @@
 ---
 ContentId: 164299e8-d27d-40b9-8b8d-a6e05df8ac69
-DateApproved: 07/14/2025
+DateApproved: 07/22/2025
 MetaDescription: Build, test, and deploy AI applications with AI Toolkit for Visual Studio Code. Features model playground, prompt engineering, batch evaluation, fine-tuning, and multi-modal support for LLMs and SLMs.
 ---
 # AI Toolkit for Visual Studio Code
 
 AI Toolkit for Visual Studio Code is a comprehensive extension that empowers developers and AI engineers to build, test, and deploy intelligent applications using generative AI models. Whether you're working locally or in the cloud, AI Toolkit provides an integrated development environment for the complete AI application lifecycle.
 
-The toolkit offers seamless integration with popular AI models from providers like OpenAI, Anthropic, Google, and GitHub, while also supporting local models through ONNX and Ollama. From model discovery and experimentation to prompt engineering and deployment, AI Toolkit streamlines your AI development workflow within VS Code.
+The AI Toolkit offers seamless integration with popular AI models from providers like OpenAI, Anthropic, Google, and GitHub, while also supporting local models through ONNX and Ollama. From model discovery and experimentation to prompt engineering and deployment, AI Toolkit streamlines your AI development workflow within VS Code.
 
 ## Key features
 
@@ -20,6 +20,7 @@ The toolkit offers seamless integration with popular AI models from providers li
 | [Model Evaluation](/docs/intelligentapps/evaluation) | Comprehensive model assessment using datasets and standard metrics. Measure performance with built-in evaluators (F1 score, relevance, similarity, coherence) or create custom evaluation criteria. | ![Screenshot showing the Model Evaluation interface with metrics and performance analysis tools](./images/overview/eval.png) |
 | [Fine-tuning](/docs/intelligentapps/finetune) | Customize and adapt models for specific domains and requirements. Train models locally with GPU support or leverage Azure Container Apps for cloud-based fine-tuning. | ![Screenshot showing the Fine-tuning interface with model adaptation and training controls](./images/overview/fine-tune.png) |
 | [Model Conversion](/docs/intelligentapps/modelconversion) | Convert, quantize, and optimize machine learning models for local deployment. Transform models from Hugging Face and other sources to run efficiently on Windows with CPU, GPU, or NPU acceleration. | ![Screenshot showing the Model Conversion interface with tools for optimizing and transforming AI models](./images/overview/conversion.png) |
+| [Tracing](/docs/intelligentapps/tracing) | Monitor and analyze the performance of your AI applications. Collect and visualize trace data to gain insights into model behavior and performance. | ![Screenshot showing the Tracing interface with tools for monitoring AI applications](./images/overview/tracing.png) |
 
 ## Who is AI Toolkit for?
 
diff --git a/docs/intelligentapps/playground.md b/docs/intelligentapps/playground.md
@@ -117,3 +117,17 @@ For each model you are comparing, you can:
 - Copy the response text to the clipboard.
 - Configure the model parameters for the selected model.
 - Continue the chat with the selected model.
+
+## What you learned
+
+In this article, you learned how to:
+
+- Use the AI Toolkit playground to interact with generative AI models.
+- Test models in the playground, configure system prompts and parameters, and explore multi-modal capabilities by attaching files.
+- Use the Compare feature to evaluate the performance of different models on the same prompt and view responses side by side.
+- Use the chat prompt input box to send prompts, attach files or images, perform web searches, and switch between modes for AI features.
+- View model responses, copy or regenerate responses, and render code snippets, images, markdown, and LaTeX.
+- Track token count in model responses to understand cost and performance.
+- Use the Model Preferences panel to select models, set system prompts, and adjust inference parameters.
+- Use the tool bar to create new playground sessions, view and manage chat history, automatically name sessions, compare models, and view generated code.
+- Manage chat history, including viewing, renaming, deleting, and switching between sessions.
diff --git a/docs/intelligentapps/tracing.md b/docs/intelligentapps/tracing.md
diff --git a/docs/toc.json b/docs/toc.json
