You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Batch Endpoints can deploy models to run inference over large amounts of data, including OpenAI models. In this example, you learn how to create a batch endpoint to deploy ADA-002 model from OpenAI to compute embeddings at scale but you can use the same approach for completions and chat completions models. It uses Microsoft Entra authentication to grant access to the Azure OpenAI resource.
20
+
To run inference over large amounts of data, you can use batch endpoints to deploy models, including OpenAI models. In this article, you see how to create a batch endpoint to deploy an ADA-002 model from OpenAI to compute embeddings at scale. You can use the same approach for completions and chat completions models.
21
21
22
-
## About this example
22
+
The examples in this article use Microsoft Entra authentication to grant access to the Azure OpenAI resource. The model is registered in MLflow format. It uses the OpenAI flavor, which provides support for calling the OpenAI service at scale.
23
23
24
-
In this example, we're going to compute embeddings over a dataset using ADA-002 model from OpenAI. We will register the particular model in MLflow format using the OpenAI flavor which has support to orchestrate all the calls to the OpenAI service at scale.
24
+
To follow along with the example steps, see the Jupyter notebook [Score OpenAI models in batch using Batch Endpoints](https://github.com/Azure/azureml-examples/blob/main/sdk/python/endpoints/batch/deploy-models/openai-embeddings/deploy-and-test.ipynb).
Use the following command to go to the folder for this example:
35
+
36
+
# [Azure CLI](#tab/cli)
29
37
30
38
```azurecli
31
39
cd endpoints/batch/deploy-models/openai-embeddings
32
40
```
33
41
34
-
### Follow along in Jupyter Notebooks
35
-
36
-
You can follow along this sample in the following notebooks. In the cloned repository, open the notebook: [deploy-and-test.ipynb](https://github.com/Azure/azureml-examples/blob/main/sdk/python/endpoints/batch/deploy-models/openai-embeddings/deploy-and-test.ipynb).
cd endpoints/batch/deploy-models/openai-embeddings
46
+
```
41
47
48
+
---
42
49
43
-
### Ensure you have an OpenAI deployment
50
+
##Create an OpenAI resource
44
51
45
-
The example shows how to run OpenAI models hosted in Azure OpenAI Service. To successfully do it, you need an OpenAI resource correctly deployed in Azure and a deployment for the model you want to use.
52
+
This article shows you how to run OpenAI models hosted in Azure OpenAI Service. To follow the steps, you need an OpenAI resource that's deployed in Azure. For information about creating an Azure OpenAI Service resource, see [Create a resource](../ai-services/openai/how-to/create-resource.md#create-a-resource).
46
53
47
54
:::image type="content" source="./media/how-to-use-batch-model-openai-embeddings/aoai-deployments.png" alt-text="An screenshot showing the Azure OpenAI studio with the list of model deployments available.":::
48
55
49
-
Take note of the OpenAI resource being used. We use the name to construct the URL of the resource. Save the URL for later use on the tutorial.
56
+
The name of your OpenAI resource forms part of the resource URL. Use the following command to save that URL for use in later steps.
### Ensure you have a compute cluster where to deploy the endpoint
72
+
##Create a compute cluster
66
73
67
-
Batch endpoints use compute cluster to run the models. In this example, we use a compute cluster called **batch-cluster**. We create the compute cluster here but you can skip this step if you already have one:
74
+
Batch endpoints use a compute cluster to run models. Use the following code to create a compute cluster called **batch-cluster-lp**. If you already have a compute cluster, you can skip this step.
68
75
69
76
# [Azure CLI](#tab/cli)
70
77
71
78
```azurecli
72
-
COMPUTE_NAME="batch-cluster"
73
-
az ml compute create -n batch-cluster --type amlcompute --min-instances 0 --max-instances 5
79
+
COMPUTE_NAME="batch-cluster-lp"
80
+
az ml compute create -n batch-cluster-lp --type amlcompute --min-instances 0 --max-instances 5
You can access the Azure OpenAI resource in two ways:
85
92
86
-
*Using Microsoft Entra authentication (recommended).
87
-
*Using an access key.
93
+
* Microsoft Entra authentication (recommended)
94
+
*An access key
88
95
89
-
Using Microsoft Entra is recommended because it helps you avoid managing secrets in the deployments.
96
+
Using Microsoft Entra is recommended because it helps you avoid managing secrets in deployments.
90
97
91
98
# [Microsoft Entra authentication](#tab/ad)
92
99
93
-
You can configure the identity of the compute to have access to the Azure OpenAI deployment to get predictions. In this way, you don't need to manage permissions for each of the users using the endpoint. To configure the identity of the compute cluster get access to the Azure OpenAI resource, follow these steps:
100
+
You can configure the identity of the compute instance to have access to the Azure OpenAI deployment to get predictions. In this way, you don't need to manage permissions for each endpoint user. To give the identity of the compute cluster access to the Azure OpenAI resource, follow these steps:
94
101
95
-
1.Ensure or assign an identity to the compute cluster your deployment uses. In this example, we use a compute cluster called **batch-cluster** and we assign a systemassigned managed identity, but you can use other alternatives.
102
+
1.Assign an identity to the compute cluster that your deployment uses. This example uses a compute cluster called **batch-cluster-lp** and a system-assigned managed identity, but you can use other alternatives. If your compute cluster already has an assigned identity, you can skip this step.
96
103
97
104
```azurecli
98
-
COMPUTE_NAME="batch-cluster"
105
+
COMPUTE_NAME="batch-cluster-lp"
99
106
az ml compute update --name $COMPUTE_NAME --identity-type system_assigned
100
107
```
101
108
@@ -120,7 +127,7 @@ You can configure the identity of the compute to have access to the Azure OpenAI
120
127
121
128
# [Access keys](#tab/keys)
122
129
123
-
You can get an access key and configure the batch deployment to use the access key to get predictions. Grab the access key from your account and keep it for future reference in this tutorial.
130
+
You can configure the batch deployment to use the OpenAI resource access key to get predictions. Copy the access key from your account, and keep it for later steps.
124
131
125
132
---
126
133
@@ -154,7 +161,7 @@ Model deployments in batch endpoints can only deploy registered models. You can
@@ -261,7 +268,7 @@ Model deployments in batch endpoints can only deploy registered models. You can
261
268
> [!TIP]
262
269
> Notice the `environment_variables` section where we indicate the configuration for the OpenAI deployment. The value for `OPENAI_API_BASE` will be set later in the creation command so you don't have to edit the YAML configuration file.
Copy file name to clipboardExpand all lines: articles/machine-learning/includes/azureml-batch-clone-samples.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ author: ccrestana
6
6
ms.author: cacrest
7
7
---
8
8
9
-
The example in this article is based on code samples contained in the [azureml-examples](https://github.com/azure/azureml-examples) repository. To run the commands locally without having to copy/paste YAML and other files, first clone the repo and then change directories to the folder:
9
+
The example in this article is based on code samples contained in the [azureml-examples](https://github.com/azure/azureml-examples) repository. To run the commands locally without having to copy or paste YAML and other files, use the following commands to clone the repository and go to the folder for your coding language:
Copy file name to clipboardExpand all lines: articles/machine-learning/includes/azureml-batch-prereqs.md
+7-9Lines changed: 7 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,14 +6,12 @@ author: ccrestana
6
6
ms.author: cacrest
7
7
---
8
8
9
-
- An Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the [free or paid version of Azure Machine Learning](https://azure.microsoft.com/free/).
9
+
- An Azure subscription. If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.
10
10
- An Azure Machine Learning workspace. To create a workspace, see [Manage Azure Machine Learning workspaces](../how-to-manage-workspace.md).
11
-
- Ensure that you have the following permissions in the Machine Learning workspace:
12
-
13
-
- Create or manage batch endpoints and deployments: Use an Owner, Contributor, or Custom role that allows `Microsoft.MachineLearningServices/workspaces/batchEndpoints/*`.
14
-
- Create Azure Resource Manager deployments in the workspace resource group: Use an Owner, Contributor, or Custom role that allows `Microsoft.Resources/deployments/write` in the resource group where the workspace is deployed.
15
-
16
-
- Install the following software to work with Machine Learning:
11
+
- The following permissions in the Azure Machine Learning workspace:
12
+
- For creating or managing batch endpoints and deployments: Use an Owner, Contributor, or Custom role that has been assigned the `Microsoft.MachineLearningServices/workspaces/batchEndpoints/*` actions.
13
+
- For creating Azure Resource Manager deployments in the workspace resource group: Use an Owner, Contributor, or Custom role that has been assigned the `Microsoft.Resources/deployments/write` action in the resource group where the workspace is deployed.
14
+
- The following software to work with Azure Machine Learning:
17
15
18
16
# [Azure CLI](#tab/cli)
19
17
@@ -23,7 +21,7 @@ ms.author: cacrest
23
21
az extension add -n ml
24
22
```
25
23
26
-
Pipeline component deployments for Batch Endpoints are introduced in version 2.7 of the `ml` extension for the Azure CLI. Use the `az extension update --name ml` command to get the latest version.
24
+
Pipeline component deployments for batch endpoints are introduced in version 2.7 of the `ml` extension for the Azure CLI. Use the `az extension update --name ml` command to get the latest version.
27
25
28
26
# [Python](#tab/python)
29
27
@@ -39,7 +37,7 @@ ms.author: cacrest
39
37
40
38
### Connect to your workspace
41
39
42
-
The workspace is the top-level resource for Machine Learning. It provides a centralized place to work with all artifacts you create when you use Machine Learning. In this section, you connect to the workspace where you perform your deployment tasks.
40
+
The workspace is the top-level resource for Azure Machine Learning. It provides a centralized place to work with all artifacts you create when you use Azure Machine Learning. In this section, you connect to the workspace where you perform your deployment tasks.
0 commit comments