You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It's common to use pipeline components to develop complex machine learning pipelines. You can group multiple steps into a component that you can use as a single step to do tasks like data preprocessing or model training.
24
+
It's common to use pipeline components to develop complex machine learning pipelines. You can also group multiple steps into a component that you can use as a single step to do tasks like data preprocessing or model training.
25
25
26
26
This article shows you how to nest multiple steps in components to build complex Azure Machine Learning pipeline jobs. You can develop and test these multistep components standalone, which helps you share your work and collaborate better with team members.
27
27
28
-
By using multi-step pipeline components, you can focus on developing subtasks and easily integrate them with the entire pipeline job. A pipeline component has a well-defined input and output interface, so multistep pipeline component users don't need to know the implementation details of the component.
28
+
By using multistep pipeline components, you can focus on developing subtasks and easily integrate them with the entire pipeline job. A pipeline component has a well-defined input and output interface, so multistep pipeline component users don't need to know the implementation details of the component.
29
29
30
-
Both pipeline components and pipeline jobs contain groups of steps or components, but defining a pipeline differs from defining a pipeline job in the following ways:
30
+
Both pipeline components and pipeline jobs contain groups of steps or components, but defining a pipeline component differs from defining a pipeline job in the following ways:
31
31
32
-
- Pipeline components define only the interface of inputs and outputs. When you define a pipeline component, you explicitly set the input and output types, but don't directly assign values to them.
33
-
- Pipeline components don't have runtime settings, so you can't hardcode a compute or a data node in a pipeline component. Instead you must promote these nodes as pipeline level inputs and assign values during runtime.
32
+
- Pipeline components define only the input and output interfaces. In a pipeline component, you explicitly set the input and output types, but don't directly assign values to them.
33
+
- Pipeline components don't have runtime settings, so you can't hardcode a compute or data node in a pipeline component. Instead you must promote these nodes as pipeline level inputs and assign values during runtime.
34
34
- Pipeline level settings such as `default_datastore` and `default_compute` are also runtime settings that aren't part of pipeline component definitions.
35
35
36
36
## Prerequisites
@@ -40,7 +40,7 @@ Both pipeline components and pipeline jobs contain groups of steps or components
40
40
41
41
# [Azure CLI](#tab/cliv2)
42
42
43
-
- Install the Azure CLI and the `ml` extension. Follow the installation steps in [Install, set up, and use the CLI (v2)](how-to-configure-cli.md).
43
+
- Install the Azure CLI and the `ml` extension. For more information, see [Install, set up, and use the CLI (v2)](how-to-configure-cli.md). The `ml` extension automatically installs the first time you run an `az ml` command.
44
44
- Understand how to [create and run Azure Machine Learning pipelines and components with the CLI v2](how-to-create-component-pipelines-cli.md).
45
45
46
46
# [Python SDK](#tab/python)
@@ -56,35 +56,35 @@ Both pipeline components and pipeline jobs contain groups of steps or components
56
56
57
57
## Build pipeline jobs with nested components
58
58
59
-
You can define multiple steps as a pipeline component and then use the multi-step component like any other component to build a pipeline job.
59
+
You can define multiple steps as a pipeline component, and then use the multistep component like any other component to build a pipeline job.
60
60
61
61
### Define pipeline components
62
62
63
63
# [Azure CLI](#tab/cliv2)
64
64
65
-
You can use multi-components to build a pipeline component, similar to how you build pipeline jobs with components.
65
+
You can use multiple components to build a pipeline component, similar to how you build pipeline jobs with components.
66
66
67
67
The following example comes from the [pipeline_with_train_eval_pipeline_component](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/pipelines-with-components/pipeline_with_pipeline_component/pipeline_with_train_eval_pipeline_component) example pipeline in the [Azure Machine Learning examples](https://github.com/Azure/azureml-examples) GitHub repository.
68
68
69
-
The example component defines a three-node pipeline job. The two nodes in the example pipeline job each use the locally defined components `train`, `score`, and `eval`. The following code shows the pipeline component:
69
+
The example component defines a three-node pipeline job. The two nodes in the example pipeline job each use the locally defined components `train`, `score`, and `eval`. The following code defines the pipeline component:
You can also find other Azure CLI pipeline component-related examples and information at [pipelines-with-components](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/pipelines-with-components) in the [Azure Machine Learning examples repository](https://github.com/Azure/azureml-examples).
73
+
You can find other Azure CLI pipeline component-related examples and information at [pipelines-with-components](https://github.com/Azure/azureml-examples/tree/main/cli/jobs/pipelines-with-components) in the [Azure Machine Learning examples repository](https://github.com/Azure/azureml-examples).
74
74
75
75
# [Python SDK](#tab/python)
76
76
77
77
You can define a pipeline component using a Python function, which is similar to defining a pipeline job using a function. You can also promote the compute of some steps to use as inputs for the pipeline component.
78
78
79
-
The following Python SDK examples are from the [Build pipeline with subpipeline (pipeline component)](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component/pipeline_with_train_eval_pipeline_component/pipeline_with_train_eval_pipeline_component.ipynb) Machine Learning notebook. Run this notebook to build the example pipeline.
79
+
The following Python SDK examples are from the [Build pipeline with subpipeline (pipeline component)](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component/pipeline_with_train_eval_pipeline_component/pipeline_with_train_eval_pipeline_component.ipynb)Azure Machine Learning notebook. Run this notebook to build the example pipeline.
You can also find other Python SDK v2 pipeline component-related notebooks and information at [Pipeline component](https://github.com/Azure/azureml-examples/tree/main/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component) in the [Azure Machine Learning examples](https://github.com/Azure/azureml-examples) GitHub repository.
83
+
You can find other Python SDK v2 pipeline component-related notebooks and information at [Pipeline component](https://github.com/Azure/azureml-examples/tree/main/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component) in the [Azure Machine Learning examples](https://github.com/Azure/azureml-examples) GitHub repository.
84
84
85
85
# [Studio UI](#tab/ui)
86
86
87
-
To access components in Azure Machine Learning studio, or to share or reuse components across jobs in the workspace, you need to register the components. To register pipeline components, follow the instructions at [Register component in your workspace](how-to-create-component-pipelines-ui.md#register-component-in-your-workspace). After that, you can view and use the components in the studio asset library and components list page.
87
+
To access components in Azure Machine Learning studio, you need to register the components. To register pipeline components, follow the instructions at [Register component in your workspace](how-to-create-component-pipelines-ui.md#register-component-in-your-workspace). After that, you can view and use the components in the studio asset library and components list page.
88
88
89
89
---
90
90
@@ -96,12 +96,12 @@ You reference pipeline components as child jobs in a pipeline job just like you
96
96
97
97
You need to promote any parameters you want to change during runtime as pipeline job inputs. Otherwise, they're hard-coded in the pipeline component. Promoting compute definition to a pipeline level input supports heterogenous pipelines that can use different compute targets in different steps.
98
98
99
-
To submit the pipeline job, you edit the `cpu-cluster` in the `default_compute` section and run the `az ml job create --file pipeline.yml` command.
99
+
To submit the pipeline job, edit the `cpu-cluster` in the `default_compute` section before you run the `az ml job create -f pipeline.yml` command.
>To share or reuse components across jobs in the workspace, you need to register the components. You can use `az ml component create` to register pipeline components.
104
+
>To share or reuse components across jobs in the workspace, you need to register the components. You can use [`az ml component create`](/cli/azure/ml/component#az-ml-component-create) to register pipeline components.
105
105
106
106
# [Python SDK](#tab/python)
107
107
@@ -110,13 +110,13 @@ You can use the pipeline component as a step like other components in the pipeli
>To share or reuse components across jobs in the workspace, you need to register the components. You can use `ml_client.components.create_or_update` to register pipeline components.
113
+
>To share or reuse components across jobs in the workspace, you need to register the components. You can use [`ml_client.components.create_or_update`](/python/api/azure-ai-ml/azure.ai.ml.mlclient#azure-ai-ml-mlclient-create-or-update) to register pipeline components.
114
114
115
115
# [Studio UI](#tab/ui)
116
116
117
-
After you register a pipeline component, you can drag and drop the component into the studio designer canvas and use the UI to build a pipeline job. For detailed instructions, see [Create pipelines using registered components](how-to-create-component-pipelines-ui.md#create-pipeline-using-registered-component).
117
+
After you register a pipeline component, you can drag and drop the component into the studio Designer canvas and use the UI to build a pipeline job. For detailed instructions, see [Create pipelines using registered components](how-to-create-component-pipelines-ui.md#create-pipeline-using-registered-component).
118
118
119
-
The following screenshots use the [nyc_taxi_data_regression_with_pipeline_component](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component/nyc_taxi_data_regression_with_pipeline_component/nyc_taxi_data_regression_with_pipeline_component.ipynb) notebook in the [Azure Machine Learning examples](https://github.com/Azure/azureml-examples) GitHub repository.
119
+
The following screenshots are from the [nyc_taxi_data_regression_with_pipeline_component](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component/nyc_taxi_data_regression_with_pipeline_component/nyc_taxi_data_regression_with_pipeline_component.ipynb) notebook in the [Azure Machine Learning examples](https://github.com/Azure/azureml-examples) GitHub repository.
120
120
121
121
:::image type="content" source="./media/how-to-use-pipeline-component/pipeline-component-authoring.png" alt-text="Screenshot of the designer canvas page to build pipeline job with pipeline component." lightbox= "./media/how-to-use-pipeline-component/pipeline-component-authoring.png":::
0 commit comments