You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Using serverless GPUs in Azure Container Apps (preview)
15
15
16
16
Azure Container Apps provides access to GPUs on-demand without you having to manage the underlying infrastructure. As a serverless feature, you only pay for GPUs in use. When enabled, the number of GPUs used for your app rises and falls to meet the load demands of your application. Serverless GPUs enable you to seamlessly run your workloads with automatic scaling, optimized cold start, per-second billing with scale down to zero when not in use, and reduced operational overhead.
17
17
18
-
Serverless GPUs are only supported for Consumption workload profiles. The feature is not supported for Consumption-only environments.
18
+
Serverless GPUs are only supported for Consumption workload profiles. The feature isn't supported for Consumption-only environments.
19
19
20
20
> [!NOTE]
21
21
> Access to GPUs is only available after you request GPU quotas. You can submit your GPU quota request via a [customer support case](/azure/azure-portal/supportability/how-to-create-azure-support-request).
@@ -93,7 +93,7 @@ Serverless GPUs are run on consumption GPU workload profiles. You manage a consu
93
93
94
94
## Improve GPU cold start
95
95
96
-
You can improve cold start on your GPU-enabled containers by enabling artifact streaming on your Azure Container Registry. For more details, see [enable artifact streaming](./https://learn.microsoft.com/en-us/azure/container-registry/container-registry-artifact-streaming?pivots=development-environment-azure-cli#pushimport-the-image-and-generate-the-streaming-artifact----azure-cli).
96
+
You can improve cold start on your GPU-enabled containers by enabling artifact streaming on your Azure Container Registry. For more information, see [enable artifact streaming](/azure/container-registry/container-registry-artifact-streaming?pivots=development-environment-azure-cli).
97
97
98
98
> [!NOTE]
99
99
> To use artifact streaming, your container images must be hosted in a premium Azure Container Registry.
# Tutorial: Deploy an NVIDIA LLAMA3 NIM to Azure Container Apps
15
14
16
-
NVIDIA Inference Microservices (NIMs) are optimized, containerized AI inference microservices designed to simplify and accelerate the deployment of AI models across various environments. By leveraging Azure Container Apps with serverless GPUs, you can run these NIMs efficiently without managing the underlying infrastructure.
15
+
NVIDIA Inference Microservices (NIMs) are optimized, containerized AI inference microservices designed to simplify and accelerate the deployment of AI models. When you use Azure Container Apps with serverless GPUs, you can run these NIMs efficiently without having to manage the underlying infrastructure.
17
16
18
-
In this tutorial, you'll deploy a Llama3 NVIDIA NIM to Azure Container Apps using serverless GPUs.
17
+
In this tutorial, you learn to deploy a Llama3 NVIDIA NIM to Azure Container Apps using serverless GPUs.
18
+
19
+
This tutorial uses a premium instance of Azure Container Registry to improve cold start performance when working with serverless GPUs. If you don't want to use a premium Azure Container Registry, make sure to modify the `az acr create` command in this tutorial to set `--sku` to `basic`.
19
20
20
21
## Prerequisites
21
22
22
-
- An Azure account with an active subscription.
23
-
- If you don't have one, you [can create one for free](https://azure.microsoft.com/free/).
24
-
- Install the [Azure CLI](/cli/azure/install-azure-cli).
25
-
- Have a NVIDIA NGC API Key. Obtain an API key from the [NVIDIA NGC website](https://catalog.ngc.nvidia.com).
23
+
| Resource | Description |
24
+
|---|---|
25
+
| Azure account | An Azure account with an active subscription.<br><br>If you don't have one, you [can create one for free](https://azure.microsoft.com/free/). |
26
+
| Azure CLI | Install the [Azure CLI](/cli/azure/install-azure-cli). |
27
+
| NVIDIA NGC API key | You can get an API key from the [NVIDIA GPU Cloud (NGC) website](https://catalog.ngc.nvidia.com). |
> This tutorial uses a premium Azure Container Registry to improve cold start performance when working with serverless GPUs. If you don't want to use a premium Azure Container Registry, modify the following command and set `--sku` to `basic`.
62
+
63
+
```azurecli
64
+
az acr create \
65
+
--resource-group $RESOURCE_GROUP \
66
+
--name $ACR_NAME \
67
+
--location $LOCATION \
68
+
--sku premium
69
+
```
70
+
71
+
## Pull, tag, and push your image
72
+
73
+
Next, pull the image from NVIDIA GPU Cloud and push to Azure Container Registry.
74
+
56
75
> [!NOTE]
57
-
> This tutorial uses a premium Azure Contianer Registry as it is recommended when using serverless GPUs for improved cold start performance. If you do not wish to use a premium Azure Container Registry, modify the below command, so --sku is set to Basic.
76
+
> NVIDIA NICs each has their own hardware requirements. Make sure the GPU type you select supports the [NIM](link) of your choice. The Llama3 NIM used in this tutorial can run on NVIDIA A100 GPUs.
After you run this command, the sign in process prompts you to enter a username. Enter **$oauthtoken** for your user name value.
85
+
86
+
Then you're prompted for a password. Enter your NVIDIA NGC API key here. Once authenticated to the NVIDIA registry, you can authenticate to the Azure registry.
87
+
88
+
1. Authenticate to Azure Container Registry.
89
+
90
+
```bash
91
+
az acr login --name $ACR_NAME
92
+
```
93
+
94
+
1. Pull the Llama3 NIM image.
95
+
96
+
```azurecli
97
+
docker pull nvcr.io/nim/meta/$CONTAINER_AND_TAG
98
+
```
99
+
100
+
1. Tag the image.
101
+
102
+
```azurecli
103
+
docker tag nvcr.io/nim/meta/$CONTAINER_AND_TAG$ACR_NAME.azurecr.io/$CONTAINER_AND_TAG
## Enable artifact streaming (recommended but optional)
62
113
63
-
## Pull the image from NGC and push to ACR
114
+
Many of the NIM images are large, and your container app can take a long timeto start if you don't enable artifact streaming. Use the following steps to enable artifact streaming.
64
115
65
116
> [!NOTE]
66
-
> NVIDIA NICs each have their own hardware requirements. [Make sure the NIM](link) you select is supported by the GPU types available in Azure Container Apps. The Llama3 NIM used in this tutorial can run on NVIDIA A100 GPUs.
117
+
> The following commands can take a few minutes to complete.
67
118
68
-
1.Authenticate with both the NVIDIA and azure container registries
119
+
1. Enable artifact streaming on your container registry.
69
120
70
-
```bash
71
-
docker login nvcr.io
72
-
Username: $oauthtoken
73
-
Password: <PASTE_API_KEY_HERE>
74
-
```
121
+
```azurecli
122
+
az acr artifact-streaming update \
123
+
--name $ACR_NAME \
124
+
--repository llama31_8b_ins \
125
+
--enable-streaming True
126
+
```
75
127
76
-
```bash
77
-
az acr login --name $ACR_NAME
78
-
```
128
+
1. Enable artifact streaming on the container image.
79
129
80
-
1. Pull the Llama3 NIM image and push it to your Azure Container Registry
Many of the NIM images are large, and your container app may take a long time to start if you don't enable artifact streaming. To enable artifact streaming, follow these steps:
150
+
1. Add the GPU workload profile to your environment.
100
151
101
-
```azurecli
102
-
az acr artifact-streaming create --image jupyter/all-spark-notebook:latest
103
-
```
152
+
```azurecli
153
+
az containerapp env workload-profile add \
154
+
--resource-group $RESOURCE_GROUP \
155
+
--name $CONTAINERAPPS_ENVIRONMENT \
156
+
--workload-profile-type $GPU_TYPE \
157
+
--workload-profile-name LLAMA_PROFILE
158
+
```
104
159
105
-
```azurecli
106
-
az acr artifact-streaming update --repository jupyter/all-spark-notebook --enable-streaming true
107
-
```
160
+
1. Create the container app.
108
161
109
-
```azurecli
110
-
az acr artifact-streaming operation show --image jupyter/all-spark-notebook:newtag
111
-
```
162
+
```azurecli
163
+
az containerapp create \
164
+
--name $CONTAINER_APP_NAME \
165
+
--resource-group $RESOURCE_GROUP \
166
+
--environment $CONTAINERAPPS_ENVIRONMENT \
167
+
--image $ACR_NAME.azurecr.io/$CONTAINER_AND_TAG \
168
+
--cpu 24 \
169
+
--memory 220 \
170
+
--gpu "NVIDIAA100" \
171
+
--secrets ngc-api-key=<PASTE_NGC_API_KEY_HERE> \
172
+
--env-vars NGC_API_KEY=secretref:ngc-api-key \
173
+
--registry-server $ACR_NAME.azurecr.io \
174
+
--registry-username <ACR_USERNAME> \
175
+
--registry-password <ACR_PASSWORD> \
176
+
--query properties.configuration.ingress.fqdn
177
+
```
112
178
113
-
Note: Tis may take a few minutes.
179
+
This command returns the URL of your container app. Set this value aside in a text editor for use in a following command.
114
180
115
-
## Create your container app with the NGC API Key
181
+
## Verify the application works
116
182
117
-
```azurecli
118
-
az containerapp env create \
119
-
--name $CONTAINERAPPS_ENVIRONMENT \
120
-
--resource-group $RESOURCE_GROUP \
121
-
--location $LOCATION \
122
-
--workload-profiles enabled
123
-
```
183
+
You can verify a successful deployment by sending a request `POST` request to your application.
184
+
185
+
Before you run this command, make sure you replace the `<YOUR_CONTAINER_APP_URL>` URL with your container app URL returned from the previous command.
124
186
125
-
az containerapp env workload-profile add \
126
-
--resource-group $RESOURCE_GROUP \
127
-
--name $CONTAINERAPPS_ENVIRONMENT \
128
-
--workload-profile-type $GPU_TYPE \
129
-
--workload-profile-name <WORKLOAD_PROFILE_NAME> \
130
-
131
-
az containerapp secret set \
132
-
--name $CONTAINER_APP_NAME \
133
-
--resource-group $RESOURCE_GROUP \
134
-
--secrets ngc-api-key=<PASTE_NGC_API_KEY_HERE>
135
-
136
-
```azurecli ///need add workload profile and verify
## (Optional) Improving performance with volume mounts
166
-
For even faster cold start times, many of the NIMs provide a volume mount path to mount a cache directory. This cache directory can be used to store the model weights and other files that the NIM needs to run. To set up a volume mount for the Llama3 NIM, see this article.
200
+
201
+
For even faster cold start times, many of the NIMs provide a volume mount path to mount a cache directory. You can use this cache directory to store the model weights and other files that the NIM needs to run. To set up a volume mount for the Llama3 NIM, see this article.
167
202
168
203
## Clean up resources
169
204
170
205
If you're not going to continue to use this application, run the following command to delete the resource group along with all the resources created in this tutorial.
171
206
172
207
>[!CAUTION]
173
-
> The following command deletes the specified resource group and all resources contained within it. If resources outside the scope of this tutorial exist in the specified resource group, they will also be deleted.
174
-
175
-
# [Bash](#tab/bash)
208
+
> The following command deletes the specified resource group and all resources contained within it. This command also deletes any resources outside the scope of this tutorial that exist in this resource group.
0 commit comments