You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# What is Azure OpenAI in Azure AI Foundry portal?
18
18
19
-
Azure OpenAI Service provides REST API access to OpenAI's powerful language models. Azure OpenAI Studio was previously where you went to access and work with the Azure OpenAI Service. This studio is now integrated into Azure AI Foundry portal.
19
+
Azure OpenAI Service provides REST API access to OpenAI's powerful language models. Azure OpenAI Studio was previously where you went to access and work with the Azure OpenAI Service. This studio is now integrated into [Azure AI Foundry portal](https://ai.azure.com).
20
20
21
21
## Access Azure OpenAI Service in Azure AI Foundry portal
Copy file name to clipboardExpand all lines: articles/ai-foundry/concepts/a-b-experimentation.md
+7-11Lines changed: 7 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,14 +1,15 @@
1
1
---
2
2
title: A/B experiments for AI applications
3
+
titleSuffix: Azure AI Foundry
3
4
description: Learn about conducting A/B experiments for AI applications.
4
-
author: s-polly
5
-
ms.author: scottpolly
5
+
author: lgayhardt
6
+
ms.author: lagayhar
6
7
ms.reviewer: skohlmeier
7
8
ms.service: azure-ai-foundry
8
9
ms.custom:
9
10
- ignite-2024
10
11
ms.topic: concept-article
11
-
ms.date: 11/22/2024
12
+
ms.date: 02/27/2025
12
13
13
14
#CustomerIntent: As an AI application developer, I want to learn about A/B experiments so that I can evaluate and improve my applications.
14
15
---
@@ -18,23 +19,21 @@ ms.date: 11/22/2024
18
19
> [!IMPORTANT]
19
20
>Items marked (preview) in this article are currently in public or private preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
20
21
21
-
In the field of AI application development, A/B experimentation has emerged as a critical practice. It allows for continuous evaluation of AI applications, balancing business impact, risk, and cost. While offline and online evaluations provide some insights, they need to be supplemented with A/B experimentation to ensure the use of right metrics for measuring success. A/B experimentation involves comparing two versions of a feature, prompt, or model using feature flags or dynamic configuration to determine which performs better. This method is essential for several reasons:
22
+
In the field of AI application development, A/B experimentation has emerged as a critical practice. It allows for continuous evaluation of AI applications, balancing business impact, risk, and cost. While offline and online evaluations provide some insights, they need to be supplemented with A/B experimentation to ensure the use of right metrics for measuring success. A/B experimentation involves comparing two versions of a feature, prompt, or model using feature flags or dynamic configuration to determine which performs better. This method is essential for several reasons:
22
23
23
24
-**Enhancing Model Performance** - A/B experimentation allows developers to systematically test different versions of AI models, algorithms, or features to identify the most effective version. With controlled experiments, you can measure the effect of changes on key performance metrics, such as accuracy, user engagement, and response time. This iterative process enables you to identify the best model, helps fine-tuning and ensures that your models deliver the best possible results.
24
25
-**Reducing Bias and Improving Fairness** - AI models can inadvertently introduce biases, leading to unfair outcomes. A/B experimentation helps identify and mitigate these biases by comparing the performance of different model versions across diverse user groups. This ensures that the AI applications are fair and equitable, providing consistent performance for all users.
25
26
-**Accelerating Innovation** - A/B experimentation fosters a culture of innovation by encouraging continuous experimentation and learning. You can quickly validate new ideas and features, reducing the time and resources spent on unproductive approaches. This accelerates the development cycle and allows teams to bring innovative AI solutions to market faster.
26
27
-**Optimizing User Experience** - User experience is paramount in AI applications. A/B experimentation enables you to experiment with different user interface designs, interaction patterns, and personalization strategies. By analyzing user feedback and behavior, you can optimize the user experience, making AI applications more intuitive and engaging.
27
28
-**Data-Driven Decision Making** - A/B experimentation provides a robust framework for data-driven decision making. Instead of relying on intuition or assumptions, you can base your decisions on empirical evidence. This leads to more informed and effective strategies for improving AI applications.
28
29
29
-
30
30
## How does A/B experimentation fit into the AI application lifecycle?
31
31
32
-
33
32
A/B experimentation and offline evaluation are both essential components in the development of AI applications, each serving unique purposes that complement each other.
34
33
35
34
Offline evaluation involves testing AI models using test datasets to measure their performance on various metrics such as fluency and coherence. After selecting a model in the Azure AI Model Catalog or GitHub Model marketplace, offline preproduction evaluation is crucial for initial model validation during integration testing, allowing you to identify potential issues and make improvements before deploying the model or application to production.
36
35
37
-
However, offline evaluation has its limitations. It can't fully capture the complex interactions that occur in real-world scenarios. This is where A/B experimentation comes into play. By deploying different versions of the AI model or UX features to live users, A/B experimentation provides insights into how the model and application performs in real-world conditions. This helps you understand user behavior, identify unforeseen issues, and measure the impact of changes on model evaluation metrics, operational metrics (for example, latency) and business metrics (for example, account sign-ups, conversions, etc.).
36
+
However, offline evaluation has its limitations. It can't fully capture the complex interactions that occur in real-world scenarios. This is where A/B experimentation comes into play. By deploying different versions of the AI model or UX features to live users, A/B experimentation provides insights into how the model and application performs in real-world conditions. This helps you understand user behavior, identify unforeseen issues, and measure the impact of changes on model evaluation metrics, operational metrics (for example, latency), and business metrics (for example, account sign-ups, conversions, etc.).
38
37
39
38
As shown in the diagram, while offline evaluation is essential for initial model validation and refinement, A/B experimentation provides the real-world testing needed to ensure the AI application performs effectively and fairly in practice. Together, they form a comprehensive approach to developing robust, safe, and user-friendly AI applications.
40
39
@@ -52,8 +51,7 @@ We're significantly simplifying the evaluation and A/B experimentation process w
52
51
53
52
## Azure AI Partners
54
53
55
-
56
-
You're also welcome to use your own A/B experimentation provider to run experiments on your AI applications. There are several solutions to choose from available in the Azure Marketplace:
54
+
You're also welcome to use your own A/B experimentation provider to run experiments on your AI applications. There are several solutions to choose from available in Azure Marketplace:
57
55
58
56
### Statsig
59
57
@@ -67,8 +65,6 @@ You're also welcome to use your own A/B experimentation provider to run experime
67
65
### LaunchDarkly
68
66
[LaunchDarkly](https://azuremarketplace.microsoft.com/marketplace/apps/aad.launchdarkly?tab=Overview) is a feature management and experimentation platform built with software developers in mind. It enables you to manage feature flags on a large scale, run A/B tests and experiments, and progressively deliver software to ship with confidence.
Copy file name to clipboardExpand all lines: articles/ai-foundry/concepts/connections.md
+10-10Lines changed: 10 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,15 +9,15 @@ ms.custom:
9
9
- build-2024
10
10
- ignite-2024
11
11
ms.topic: conceptual
12
-
ms.date: 11/21/2024
13
-
ms.reviewer: sgilley
12
+
ms.date: 02/21/2025
13
+
ms.reviewer: meerakurup
14
14
ms.author: sgilley
15
15
author: sdgilley
16
16
---
17
17
18
18
# Connections in Azure AI Foundry portal
19
19
20
-
Connections in Azure AI Foundry portal are a way to authenticate and consume both Microsoft and non-Microsoft resources within your Azure AI Foundry projects. For example, connections can be used for prompt flow, training data, and deployments. [Connections can be created](../how-to/connections-add.md) exclusively for one project or shared with all projects in the same hub.
20
+
Connections in [Azure AI Foundry](https://ai.azure.com) are a way to authenticate and consume both Microsoft and non-Microsoft resources within your Azure AI Foundry projects. For example, connections can be used for prompt flow, training data, and deployments. [Connections can be created](../how-to/connections-add.md) exclusively for one project or shared with all projects in the same hub.
21
21
22
22
## Connections to Azure AI services
23
23
@@ -31,30 +31,30 @@ As another example, you can [create a connection](../how-to/connections-add.md)
31
31
32
32
## Connections to non-Microsoft services
33
33
34
-
Azure AI Foundry supports connections to non-Microsoft services, including the following:
35
-
- The [API key connection](../how-to/connections-add.md) handles authentication to your specified target on an individual basis. This is the most common non-Microsoft connection type.
36
-
- The [custom connection](../how-to/connections-add.md) allows you to securely store and access keys while storing related properties, such as targets and versions. Custom connections are useful when you have many targets that or cases where you wouldn't need a credential to access. LangChain scenarios are a good example where you would use custom service connections. Custom connections don't manage authentication, so you'll have to manage authentication on your own.
34
+
Azure AI Foundry supports connections to non-Microsoft services, including:
35
+
- The [API key connection](../how-to/connections-add.md) handles authentication to your specified target on an individual basis. API key is the most common non-Microsoft connection type.
36
+
- The [custom connection](../how-to/connections-add.md) allows you to securely store and access keys while storing related properties, such as targets and versions. Custom connections are useful when you have many targets that or cases where you wouldn't need a credential to access. LangChain scenarios are a good example where you would use custom service connections. Custom connections don't manage authentication, so you have to manage authentication on your own.
37
37
38
38
## Connections to datastores
39
39
40
40
> [!IMPORTANT]
41
-
> Data connections cannot be shared across projects. They are created exclusively in the context of one project.
41
+
> Data connections can't be shared across projects. They're created exclusively in the context of one project.
42
42
43
43
Creating a data connection allows you to access external data without copying it to your project. Instead, the connection provides a reference to the data source.
44
44
45
45
A data connection offers these benefits:
46
46
47
47
- A common, easy-to-use API that interacts with different storage types including Microsoft OneLake, Azure Blob, and Azure Data Lake Gen2.
48
48
- Easier discovery of useful connections in team operations.
49
-
-For credential-based access (service principal/SAS/key), Azure AI Foundry connection secures credential information. This way, you won't need to place that information in your scripts.
49
+
-Credential-based access (service principal/SAS/key). Azure AI Foundry connection secures credential information so you don't need to place that information in your scripts.
50
50
51
51
When you create a connection with an existing Azure storage account, you can choose between two different authentication methods:
52
52
53
53
-**Credential-based**: Authenticate data access with a service principal, shared access signature (SAS) token, or account key. Users with *Reader* project permissions can access the credentials.
54
54
-**Identity-based**: Use your Microsoft Entra ID or managed identity to authenticate data access.
55
55
56
56
> [!TIP]
57
-
> When using an identity-based connection, Azure role-based access control (Azure RBAC) is used to determine who can access the connection. You must assign the correct Azure RBAC roles to your developers before they can use the connection. For more information, see [Scenario: Connections using Microsoft Entra ID](rbac-ai-foundry.md#scenario-connections-using-microsoft-entra-id-authentication).
57
+
> When you use an identity-based connection, Azure role-based access control (Azure RBAC) determines who can access the connection. You must assign the correct Azure RBAC roles to your developers before they can use the connection. For more information, see [Scenario: Connections using Microsoft Entra ID](rbac-ai-foundry.md#scenario-connections-using-microsoft-entra-id-authentication).
58
58
59
59
60
60
The following table shows the supported Azure cloud-based storage services and authentication methods:
@@ -82,7 +82,7 @@ A Uniform Resource Identifier (URI) represents a storage location on your local
82
82
83
83
## Key vaults and secrets
84
84
85
-
Connections allow you to securely store credentials, authenticate access, and consume data and information. Secrets associated with connections are securely persisted in the corresponding Azure Key Vault, adhering to robust security and compliance standards. As an administrator, you can audit both shared and project-scoped connections on a hub level (link to connection rbac).
85
+
Connections allow you to securely store credentials, authenticate access, and consume data and information. Secrets associated with connections are securely persisted in the corresponding Azure Key Vault, adhering to robust security and compliance standards. As an administrator, you can audit both shared and project-scoped connections on a hub level.
86
86
87
87
Azure connections serve as key vault proxies, and interactions with connections are direct interactions with an Azure key vault. Azure AI Foundry connections store API keys securely, as secrets, in a key vault. The key vault [Azure role-based access control (Azure RBAC)](./rbac-ai-foundry.md) controls access to these connection resources. A connection references the credentials from the key vault storage location for further use. You won't need to directly deal with the credentials after they're stored in the hub's key vault. You have the option to store the credentials in the YAML file. A CLI command or SDK can override them. We recommend that you avoid credential storage in a YAML file, because a security breach could lead to a credential leak.
Copy file name to clipboardExpand all lines: articles/ai-foundry/concepts/content-filtering.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -86,7 +86,7 @@ The configurability feature allows customers to adjust the settings, separately
86
86
87
87
<sup>1</sup> For Azure OpenAI models, only customers who have been approved for modified content filtering have full content filtering control, including configuring content filters at severity level high only or turning off content filters. Apply for modified content filters via these forms: [Azure OpenAI Limited Access Review: Modified Content Filters](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUMlBQNkZMR0lFRldORTdVQzQ0TEI5Q1ExOSQlQCN0PWcu), and [Modified Abuse Monitoring](https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUOE9MUTFMUlpBNk5IQlZWWkcyUEpWWEhGOCQlQCN0PWcu).
88
88
89
-
Customers are responsible for ensuring that applications integrating Azure OpenAI comply with the [Code of Conduct](/legal/cognitive-services/openai/code-of-conduct?context=%2Fazure%2Fai-services%2Fopenai%2Fcontext%2Fcontext).
89
+
Customers are responsible for ensuring that applications integrating Azure OpenAI comply with the [Code of Conduct](/legal/ai-code-of-conduct?context=%2Fazure%2Fai-services%2Fopenai%2Fcontext%2Fcontext).
0 commit comments