You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Once you've trained machine learning models or pipelines, or you've found models from the model catalog that suit your needs, you need to deploy them to production so that others can use them for _inference_. Inference is the process of applying new input data to a machine learning model or pipeline to generate outputs. While these outputs are typically referred to as "predictions," inferencing can be used to generate outputs for other machine learning tasks, such as classification and clustering. In Azure Machine Learning, you perform inferencing by using __endpoints__.
24
+
After you train machine learning models or pipelines, or find suitable models from the model catalog, you need to deploy them to production for others to use for _inference_. Inference is the process of applying new input data to a machine learning model or pipeline to generate outputs. While these outputs are typically called "predictions," inference can generate outputs for other machine learning tasks, such as classification and clustering. In Azure Machine Learning, you perform inference by using __endpoints__.
25
25
26
26
## Endpoints and deployments
27
27
28
-
An **endpoint** is a stable and durable URL that can be used to request or invoke a model. You provide the required inputs to the endpoint and get the outputs back. Azure Machine Learning allows you to implement standard deployments, online endpoints, and batch endpoints. An endpoint provides:
28
+
An **endpoint** is a stable and durable URL that can be used to request or invoke a model. You provide the required inputs to the endpoint and receive the outputs. Azure Machine Learning supports standard deployments, online endpoints, and batch endpoints. An endpoint provides:
29
29
30
-
-a stable and durable URL (like _endpoint-name.region.inference.ml.azure.com_),
31
-
-an authentication mechanism, and
32
-
-an authorization mechanism.
30
+
-A stable and durable URL (such as _endpoint-name.region.inference.ml.azure.com_)
31
+
-An authentication mechanism
32
+
-An authorization mechanism
33
33
34
-
A **deployment** is a set of resources and computes required for hosting the model or component that does the actual inferencing. An endpoint contains a deployment, and for online and batch endpoints, one endpoint can contain several deployments. The deployments can host independent assets and consume different resources, based on the needs of the assets. Furthermore, an endpoint has a routing mechanism that can direct requests to any of its deployments.
34
+
A **deployment** is a set of resources and compute required to host the model or component that performs the actual inference. An endpoint contains a deployment. For online and batch endpoints, one endpoint can contain several deployments. The deployments can host independent assets and consume different resources based on the needs of the assets. An endpoint also has a routing mechanism that can direct requests to any of its deployments.
35
35
36
-
On one hand, some types of endpoints in Azure Machine Learning consume dedicated resources on their deployments. For these endpoints to run, you must have compute quota on your Azure subscription. On the other hand, certain models support a serverless deployment—allowing them to consume no quota from your subscription. For serverless deployment, you're billed based on usage.
36
+
Some types of endpoints in Azure Machine Learning consume dedicated resources on their deployments. For these endpoints to run, you must have compute quota on your Azure subscription. However, certain models support a serverless deployment, which allows them to consume no quota from your subscription. For serverless deployments, you're billed based on usage.
37
37
38
38
### Intuition
39
39
40
-
Suppose you're working on an application that predicts the type and color of a car, given its photo. For this application, a user with certain credentials makes an HTTP request to a URL and provides a picture of a car as part of the request. In return, the user gets a response that includes the type and color of the car as string values. In this scenario, the URL serves as an __endpoint__.
40
+
Suppose you're working on an application that predicts the type and color of a car from a photo. For this application, a user with certain credentials makes an HTTP request to a URL and provides a picture of a car as part of the request. In return, the user receives a response that includes the type and color of the car as string values. In this scenario, the URL serves as an __endpoint__.
41
41
42
42
:::image type="content" source="media/concept-endpoints/concept-endpoint.png" alt-text="A diagram showing the concept of an endpoint." border="false":::
43
43
44
-
Furthermore, say that a data scientist, Alice, is working on implementing the application. Alice knows a lot about TensorFlow and decides to implement the model using a Keras sequential classifier with a RestNet architecture from the TensorFlow Hub. After testing the model, Alice is happy with its results and decides to use the model to solve the car prediction problem. The model is large in size and requires 8 GB of memory with 4 cores to run. In this scenario, Alice's model and the resources, such as the code and the compute, that are required to run the model make up a __deployment under the endpoint__.
44
+
Now suppose that a data scientist, Alice, is implementing the application. Alice has extensive TensorFlow experience and decides to implement the model using a Keras sequential classifier with a ResNet architecture from the TensorFlow Hub. After testing the model, Alice is satisfied with its results and decides to use the model to solve the car prediction problem. The model is large and requires 8 GB of memory with 4 cores to run. In this scenario, Alice's model and the resources—such as the code and the compute—that are required to run the model make up a __deployment under the endpoint__.
45
45
46
46
:::image type="content" source="media/concept-endpoints/concept-deployment.png" alt-text="A diagram showing the concept of a deployment." border="false":::
47
47
48
-
Let's imagine that after a couple of months, the organization discovers that the application performs poorly on images with less than ideal illumination conditions. Bob, another data scientist, knows a lot about data augmentation techniques that help a model build robustness on that factor. However, Bob feels more comfortable using Torch to implement the model and trains a new model with Torch. Bob wants to try this model in production gradually until the organization is ready to retire the old model. The new model also shows better performance when deployed to GPU, so the deployment needs to include a GPU. In this scenario, Bob's model and the resources, such as the code and the compute, that are required to run the model make up __another deployment under the same endpoint__.
48
+
After a few months, the organization discovers that the application performs poorly on images with poor lighting conditions. Bob, another data scientist, has expertise in data augmentation techniques that help models build robustness for this factor. However, Bob prefers using PyTorch to implement the model and trains a new model with PyTorch. Bob wants to test this model in production gradually until the organization is ready to retire the old model. The new model also performs better when deployed to GPU, so the deployment needs to include a GPU. In this scenario, Bob's model and the resources—such as the code and the compute—that are required to run the model make up __another deployment under the same endpoint__.
49
49
50
50
:::image type="content" source="media/concept-endpoints/concept-deployment-routing.png" alt-text="A diagram showing the concept of an endpoint with multiple deployments." border="false":::
51
51
52
52
## Endpoints: standard deployment, online, and batch
53
53
54
-
Azure Machine Learning allows you to implement[standard deployments](how-to-deploy-models-serverless.md), [online endpoints](concept-endpoints-online.md), and [batch endpoints](concept-endpoints-batch.md).
54
+
Azure Machine Learning supports[standard deployments](how-to-deploy-models-serverless.md), [online endpoints](concept-endpoints-online.md), and [batch endpoints](concept-endpoints-batch.md).
55
55
56
-
_standard deployment_ and _online endpoints_ are designed for real-time inference. Whenever you invoke the endpoint, the results are returned in the endpoint's response. Standard deployments don't consume quota from your subscription; rather, they're billed with Standard billing.
56
+
_Standard deployments_ and _online endpoints_ are designed for real-time inference. When you invoke the endpoint, the results are returned in the endpoint's response. Standard deployments don't consume quota from your subscription; instead, they're billed with standard billing.
57
57
58
-
_Batch endpoints_ are designed for long-running batch inference. Whenever you invoke a batch endpoint, you generate a batch job that performs the actual work.
58
+
_Batch endpoints_ are designed for long-running batch inference. When you invoke a batch endpoint, you generate a batch job that performs the actual work.
59
59
60
60
### When to use standard deployment, online, and batch endpoints
61
61
62
-
__standard deployment__:
62
+
__Standard deployment__:
63
63
64
64
Use [standard deployments](how-to-deploy-models-serverless.md) to consume large foundational models for real-time inferencing off-the-shelf or for fine-tuning such models. Not all models are available for deployment to standard deployments. We recommend using this deployment mode when:
65
65
66
66
> [!div class="checklist"]
67
67
> * Your model is a foundational model or a fine-tuned version of a foundational model that is available for standard deployments.
68
68
> * You can benefit from a quota-less deployment.
69
-
> * You don't need to customize the inferencing stack used to run the model.
69
+
> * You don't need to customize the inference stack used to run the model.
70
70
71
71
__Online endpoints__:
72
72
@@ -76,7 +76,7 @@ Use [online endpoints](concept-endpoints-online.md) to operationalize models for
76
76
> * Your model is a foundational model or a fine-tuned version of a foundational model, but it's not supported in standard deployment.
77
77
> * You have low-latency requirements.
78
78
> * Your model can answer the request in a relatively short amount of time.
79
-
> * Your model's inputs fit on the HTTP payload of the request.
79
+
> * Your model's inputs fit in the HTTP payload of the request.
80
80
> * You need to scale up in terms of number of requests.
81
81
82
82
__Batch endpoints__:
@@ -93,11 +93,11 @@ Use [batch endpoints](concept-endpoints-batch.md) to operationalize models or pi
93
93
94
94
### Comparison of standard deployment, online, and batch endpoints
95
95
96
-
All standard deployment, online, and batch endpoints are based on the idea of endpoints, therefore, you can transition easily from one to the other. Online and batch endpoints are also capable of managing multiple deployments for the same endpoint.
96
+
All standard deployments, online endpoints, and batch endpoints are based on the idea of endpoints, therefore, you can transition easily from one to the other. Online and batch endpoints are also capable of managing multiple deployments for the same endpoint.
97
97
98
98
#### Endpoints
99
99
100
-
The following table shows a summary of the different features available to standard deployment, online, and batch endpoints at the endpoint level.
100
+
The following table shows a summary of the different features available to standard deployments, online endpoints, and batch endpoints at the endpoint level.
@@ -117,9 +117,9 @@ The following table shows a summary of the different features available to stand
117
117
118
118
#### Deployments
119
119
120
-
The following table shows a summary of the different features available to standard deployment, online, and batch endpoints at the deployment level. These concepts apply to each deployment under the endpoint (for online and batch endpoints), and apply to standard deployment (where the concept of deployment is built into the endpoint).
120
+
The following table shows a summary of the different features available to standard deployments, online endpoints, and batch endpoints at the deployment level. These concepts apply to each deployment under the endpoint (for online and batch endpoints), and apply to standard deployments (where the concept of deployment is built into the endpoint).
| Cost basis<sup>5</sup> | Per token | Per deployment: compute instances running | Per job: compute instanced consumed in the job (capped to the maximum number of instances of the cluster) |
134
+
| Cost basis<sup>5</sup> | Per token | Per deployment: compute instances running | Per job: compute instances consumed in the job (capped to the maximum number of instances of the cluster) |
135
135
| Local testing of deployments | No | Yes | No |
136
136
137
137
138
138
<sup>2</sup> *Inference server* refers to the serving technology that takes requests, processes them, and creates responses. The inference server also dictates the format of the input and the expected outputs.
139
139
140
140
<sup>3</sup> *Autoscaling* is the ability to dynamically scale up or scale down the deployment's allocated resources based on its load. Online and batch deployments use different strategies for autoscaling. While online deployments scale up and down based on the resource utilization (like CPU, memory, requests, etc.), batch endpoints scale up or down based on the number of jobs created.
141
141
142
-
<sup>4</sup> Both online and batch deployments charge by the resources consumed. In online deployments, resources are provisioned at deployment time. In batch deployment, resources aren't consumed at deployment time but at the time that the job runs. Hence, there's no cost associated with the batch deployment itself. Likewise, queued jobs don't consume resources either.
142
+
<sup>4</sup> Both online and batch deployments charge by the resources consumed. In online deployments, resources are provisioned at deployment time. In batch deployments, resources aren't consumed at deployment time but at the time that the job runs. Hence, there's no cost associated with the batch deployment itself. Likewise, queued jobs don't consume resources either.
143
143
144
144
## Developer interfaces
145
145
146
146
Endpoints are designed to help organizations operationalize production-level workloads in Azure Machine Learning. Endpoints are robust and scalable resources, and they provide the best capabilities to implement MLOps workflows.
147
147
148
148
You can create and manage batch and online endpoints with several developer tools:
![prmerger-automator[bot]](https://avatars.githubusercontent.com/u/22479449?v=4&size=40){kind=avatar}
0 commit comments