You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/machine-learning/how-to-use-batch-azure-data-factory.md
+9-6Lines changed: 9 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,18 +19,21 @@ ms.custom: devplatv2
19
19
20
20
Big data requires a service that can orchestrate and operationalize processes to refine these enormous stores of raw data into actionable business insights. The [Azure Data Factory](../data-factory/introduction.md) managed cloud service handles these complex hybrid extract-transform-load (ETL), extract-load-transform (ELT), and data integration projects.
21
21
22
-
Azure Data Factory allows you to create pipelines that can orchestrate multiple data transformations and manage them as a single unit. Batch endpoints are an excellent candidate to become a step in such processing workflow. In this article, learn how to use batch endpoints in Azure Data Factory activities by relying on the Web Invoke activity and the REST API.
22
+
Azure Data Factory allows you to create pipelines that can orchestrate multiple data transformations and manage them as a single unit. Batch endpoints are an excellent candidate to become a step in such processing workflow.
23
+
24
+
In this article, learn how to use batch endpoints in Azure Data Factory activities by relying on the Web Invoke activity and the REST API.
23
25
24
26
> [!TIP]
25
-
> When you use data pipelines in Fabric, you can invoke batch endpoint directly using the Azure Machine Learning activity. We recommend using Fabric for data orchestration whenever possible to take advantage of the newest capabilities. The Azure Machine Learning activity in Azure Data Factory can only work with assets from Azure Machine Learning V1. For more information, see [Run Azure Machine Learning models from Fabric, using batch endpoints (preview)](how-to-use-batch-fabric.md).
27
+
> When you use data pipelines in Fabric, you can invoke batch endpoint directly using the Azure Machine Learning activity. We recommend using Fabric for data orchestration whenever possible to take advantage of the newest capabilities.
28
+
> The Azure Machine Learning activity in Azure Data Factory can only work with assets from Azure Machine Learning V1. For more information, see [Run Azure Machine Learning models from Fabric, using batch endpoints (preview)](how-to-use-batch-fabric.md).
26
29
27
30
## Prerequisites
28
31
29
32
- A model deployed as a batch endpoint. Use the *heart condition classifier* created in [Using MLflow models in batch deployments](how-to-mlflow-batch.md).
30
33
- An Azure Data Factory resource. To create a data factory, follow the steps in [Quickstart: Create a data factory by using the Azure portal](../data-factory/quickstart-create-data-factory-portal.md).
31
34
- After creating your data factory, browse to it in the Azure portal and select **Launch Studio**:
32
35
33
-
:::image type="content" source="./media/how-to-use-batch-adf/data-factory-home-page.png" alt-text="Screenshot of the home page for the Azure Data Factory, labeled Open Azure Data Factory Studio and Launch studio highlighted.":::
36
+
:::image type="content" source="./media/how-to-use-batch-adf/data-factory-home-page.png" lightbox="./media/how-to-use-batch-adf/data-factory-home-page.png" alt-text="Screenshot of the home page for the Azure Data Factory, labeled Open Azure Data Factory Studio and Launch studio highlighted.":::
34
37
35
38
## Authenticate against batch endpoints
36
39
@@ -96,11 +99,11 @@ The pipeline requires you to configure the following parameters:
96
99
97
100
# [Use a service principal](#tab/sp)
98
101
99
-
:::image type="content" source="./media/how-to-use-batch-adf/pipeline-diagram.png" alt-text="Diagram that shows th high level structure of the pipeline you're creating.":::
102
+
:::image type="content" source="./media/how-to-use-batch-adf/pipeline-diagram.png" lightbox="./media/how-to-use-batch-adf/pipeline-diagram.png" alt-text="Diagram that shows th high level structure of the pipeline you're creating.":::
100
103
101
104
The pipeline contains the following activities:
102
105
103
-
-**Authorize**: A Web Activity that uses the service principal created in [Authenticating against batch endpoints](#authenticating-against-batch-endpoints) to obtain an authorization token. This token is used to invoke the endpoint later.
106
+
-**Authorize**: A Web Activity that uses the service principal created in [Authenticating against batch endpoints](#authenticate-against-batch-endpoints) to obtain an authorization token. This token is used to invoke the endpoint later.
104
107
-**Run Batch-Endpoint**: A Web Activity that uses the batch endpoint URI to invoke it. It passes the input data URI where the data is located and the expected output file.
105
108
-**Wait for job**: It's a loop activity that checks the status of the created job and waits for its completion, either as **Completed** or **Failed**. This activity uses the following activities:
106
109
@@ -156,7 +159,7 @@ To create this pipeline in your existing Azure Data Factory and invoke batch end
156
159
157
160
:::image type="content" source="./media/how-to-use-batch-adf/pipeline-params.png" alt-text="Screenshot of the pipeline parameters expected for the resulting pipeline that uses a service principal.":::
158
161
159
-
---
162
+
---
160
163
161
164
> [!WARNING]
162
165
> Ensure that your batch endpoint has a default deployment configured before you submit a job to it. The created pipeline invokes the endpoint. A default deployment needs to be created and configured.
0 commit comments