Skip to content

Commit c96cf02

Browse files
denreadenrea
authored
Merge pull request #279656 from msakande/specify-models-online-deployment
New article: specify-models-for online-deployment
2 parents 2e75eb3 + 505c59e commit c96cf02

11 files changed

+173
-123
lines changed

‎articles/machine-learning/concept-endpoints-online.md

Lines changed: 5 additions & 119 deletions
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@ The following table describes the key attributes of a deployment:
121121
|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
122122
| Name | The name of the deployment. |
123123
| Endpoint name | The name of the endpoint to create the deployment under. |
124-
| Model<sup>1</sup> | The model to use for the deployment. This value can be either a reference to an existing versioned model in the workspace or an inline model specification. For more information on how to track and specify the path to your model, see [Identify model path with respect to `AZUREML_MODEL_DIR`](#identify-model-path-with-respect-to-azureml_model_dir). |
124+
| Model<sup>1</sup> | The model to use for the deployment. This value can be either a reference to an existing versioned model in the workspace or an inline model specification. For more information on how to track and specify the path to your model, see [Specify model to deploy for use in online endpoint](concept-online-deployment-model-specification.md). |
125125
| Code path | The path to the directory on the local development environment that contains all the Python source code for scoring the model. You can use nested directories and packages. |
126126
| Scoring script | The relative path to the scoring file in the source code directory. This Python code must have an `init()` function and a `run()` function. The `init()` function will be called after the model is created or updated (you can use it to cache the model in memory, for example). The `run()` function is called at every invocation of the endpoint to do the actual scoring and prediction. |
127127
| Environment<sup>1</sup> | The environment to host the model and code. This value can be either a reference to an existing versioned environment in the workspace or an inline environment specification. __Note:__ Microsoft regularly patches the base images for known security vulnerabilities. You'll need to redeploy your endpoint to use the patched image. If you provide your own image, you're responsible for updating it. For more information, see [Image patching](concept-environments.md#image-patching). |
@@ -135,129 +135,15 @@ The following table describes the key attributes of a deployment:
135135
136136
To learn how to deploy online endpoints using the CLI, SDK, studio, and ARM template, see [Deploy an ML model with an online endpoint](how-to-deploy-online-endpoints.md).
137137

138-
### Identify model path with respect to `AZUREML_MODEL_DIR`
139-
140-
When deploying your model to Azure Machine Learning, you need to specify the location of the model you wish to deploy as part of your deployment configuration. In Azure Machine Learning, the path to your model is tracked with the `AZUREML_MODEL_DIR` environment variable. By identifying the model path with respect to `AZUREML_MODEL_DIR`, you can deploy one or more models that are stored locally on your machine or deploy a model that is registered in your Azure Machine Learning workspace.
141-
142-
For illustration, we reference the following local folder structure for the first two cases where you deploy a single model or deploy multiple models that are stored locally:
143-
144-
:::image type="content" source="media/how-to-deploy-online-endpoints/multi-models-1.png" alt-text="A screenshot showing a folder structure containing multiple models." lightbox="media/how-to-deploy-online-endpoints/multi-models-1.png":::
145-
146-
#### Use a single local model in a deployment
147-
148-
To use a single model that you have on your local machine in a deployment, specify the `path` to the `model` in your deployment YAML. Here's an example of the deployment YAML with the path `/Downloads/multi-models-sample/models/model_1/v1/sample_m1.pkl`:
149-
150-
```yml
151-
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
152-
name: blue
153-
endpoint_name: my-endpoint
154-
model:
155-
  path: /Downloads/multi-models-sample/models/model_1/v1/sample_m1.pkl
156-
code_configuration:
157-
  code: ../../model-1/onlinescoring/
158-
  scoring_script: score.py
159-
environment:
160-
  conda_file: ../../model-1/environment/conda.yml
161-
  image: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest
162-
instance_type: Standard_DS3_v2
163-
instance_count: 1
164-
```
165-
166-
After you create your deployment, the environment variable `AZUREML_MODEL_DIR` will point to the storage location within Azure where your model is stored. For example, `/var/azureml-app/azureml-models/81b3c48bbf62360c7edbbe9b280b9025/1` will contain the model `sample_m1.pkl`.
167-
168-
Within your scoring script (`score.py`), you can load your model (in this example, `sample_m1.pkl`) in the `init()` function:
169-
170-
```python
171-
def init():
172-
model_path = os.path.join(str(os.getenv("AZUREML_MODEL_DIR")), "sample_m1.pkl")
173-
model = joblib.load(model_path)
174-
```
175-
176-
#### Use multiple local models in a deployment
177-
178-
Although the Azure CLI, Python SDK, and other client tools allow you to specify only one model per deployment in the deployment definition, you can still use multiple models in a deployment by registering a model folder that contains all the models as files or subdirectories.
179-
180-
In the previous example folder structure, you notice that there are multiple models in the `models` folder. In your deployment YAML, you can specify the path to the `models` folder as follows:
181-
182-
```yml
183-
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
184-
name: blue
185-
endpoint_name: my-endpoint
186-
model:
187-
  path: /Downloads/multi-models-sample/models/
188-
code_configuration:
189-
  code: ../../model-1/onlinescoring/
190-
  scoring_script: score.py
191-
environment:
192-
  conda_file: ../../model-1/environment/conda.yml
193-
  image: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest
194-
instance_type: Standard_DS3_v2
195-
instance_count: 1
196-
```
197-
198-
After you create your deployment, the environment variable `AZUREML_MODEL_DIR` will point to the storage location within Azure where your models are stored. For example, `/var/azureml-app/azureml-models/81b3c48bbf62360c7edbbe9b280b9025/1` will contain the models and the file structure.
199-
200-
For this example, the contents of the `AZUREML_MODEL_DIR` folder will look like this:
201-
202-
:::image type="content" source="media/how-to-deploy-online-endpoints/multi-models-2.png" alt-text="A screenshot of the folder structure of the storage location for multiple models." lightbox="media/how-to-deploy-online-endpoints/multi-models-2.png":::
203-
204-
Within your scoring script (`score.py`), you can load your models in the `init()` function. The following code loads the `sample_m1.pkl` model:
205-
206-
```python
207-
def init():
208-
model_path = os.path.join(str(os.getenv("AZUREML_MODEL_DIR")), "models","model_1","v1", "sample_m1.pkl ")
209-
model = joblib.load(model_path)
210-
```
211-
212-
For an example of how to deploy multiple models to one deployment, see [Deploy multiple models to one deployment (CLI example)](https://github.com/Azure/azureml-examples/blob/main/cli/endpoints/online/custom-container/minimal/multimodel) and [Deploy multiple models to one deployment (SDK example)](https://github.com/Azure/azureml-examples/blob/main/sdk/python/endpoints/online/custom-container/online-endpoints-custom-container-multimodel.ipynb).
213-
214-
> [!TIP]
215-
> If you have more than 1500 files to register, consider compressing the files or subdirectories as .tar.gz when registering the models. To consume the models, you can uncompress the files or subdirectories in the `init()` function from the scoring script. Alternatively, when you register the models, set the `azureml.unpack` property to `True`, to automatically uncompress the files or subdirectories. In either case, uncompression happens once in the initialization stage.
216-
217-
#### Use models registered in your Azure Machine Learning workspace in a deployment
218-
219-
To use one or more models, which are registered in your Azure Machine Learning workspace, in your deployment, specify the name of the registered model(s) in your deployment YAML. For example, the following deployment YAML configuration specifies the registered `model` name as `azureml:local-multimodel:3`:
220-
221-
```yml
222-
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
223-
name: blue
224-
endpoint_name: my-endpoint
225-
model: azureml:local-multimodel:3
226-
code_configuration:
227-
  code: ../../model-1/onlinescoring/
228-
  scoring_script: score.py
229-
environment:
230-
  conda_file: ../../model-1/environment/conda.yml
231-
  image: mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest
232-
instance_type: Standard_DS3_v2
233-
instance_count: 1
234-
```
235-
236-
For this example, consider that `local-multimodel:3` contains the following model artifacts, which can be viewed from the **Models** tab in the Azure Machine Learning studio:
237-
238-
:::image type="content" source="media/how-to-deploy-online-endpoints/multi-models-3.png" alt-text="A screenshot of the folder structure showing the model artifacts of the registered model." lightbox="media/how-to-deploy-online-endpoints/multi-models-3.png":::
239-
240-
After you create your deployment, the environment variable `AZUREML_MODEL_DIR` will point to the storage location within Azure where your models are stored. For example, `/var/azureml-app/azureml-models/local-multimodel/3` will contain the models and the file structure. `AZUREML_MODEL_DIR` will point to the folder containing the root of the model artifacts.
241-
Based on this example, the contents of the `AZUREML_MODEL_DIR` folder will look like this:
242-
243-
:::image type="content" source="media/how-to-deploy-online-endpoints/multi-models-4.png" alt-text="A screenshot of the folder structure showing multiple models." lightbox="media/how-to-deploy-online-endpoints/multi-models-4.png":::
244-
245-
Within your scoring script (`score.py`), you can load your models in the `init()` function. For example, load the `diabetes.sav` model:
246-
247-
```python
248-
def init():
249-
model_path = os.path.join(str(os.getenv("AZUREML_MODEL_DIR"), "models", "diabetes", "1", "diabetes.sav")
250-
model = joblib.load(model_path)
251-
```
252-
253-
254138
### Virtual machine quota allocation for deployment
255139

256140
[!INCLUDE [quota-allocation-online-deployment](includes/quota-allocation-online-deployment.md)]
257141

258-
Azure Machine Learning provides a [shared quota](how-to-manage-quotas.md#azure-machine-learning-shared-quota) pool from which all users can access quota to perform testing for a limited time. When you use the studio to deploy Llama models (from the model catalog) to a managed online endpoint, Azure Machine Learning allows you to access this shared quota for a short time.
142+
#### Shared quota pool
143+
144+
Also, [!INCLUDE [machine-learning-shared-quota](includes/machine-learning-shared-quota.md)]
259145

260-
To deploy a _Llama-2-70b_ or _Llama-2-70b-chat_ model, however, you must have an [Enterprise Agreement subscription](../cost-management-billing/manage/create-enterprise-subscription.md) before you can deploy using the shared quota. For more information on how to use the shared quota for online endpoint deployment, see [How to deploy foundation models using the studio](how-to-use-foundation-models.md#deploying-using-the-studio).
146+
To deploy Llama-2, Phi, Nemotron, Mistral, Dolly, and Deci-DeciLM models from the model catalog via the shared quota, you must have an [Enterprise Agreement subscription](../cost-management-billing/manage/create-enterprise-subscription.md). For more information on how to use the shared quota for online endpoint deployment, see [How to deploy foundation models using the studio](how-to-use-foundation-models.md#shared-quota).
261147

262148
For more information on quotas and limits for resources in Azure Machine Learning, see [Manage and increase quotas and limits for resources with Azure Machine Learning](how-to-manage-quotas.md).
263149

0 commit comments

Comments
 (0)