You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/machine-learning/concept-designer.md
+13-13Lines changed: 13 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,15 +8,15 @@ ms.topic: concept-article
8
8
ms.author: lagayhar
9
9
ms.reviewer: lagayhar
10
10
author: lgayhardt
11
-
ms.date: 05/21/2024
11
+
ms.date: 09/09/2025
12
12
ms.custom:
13
13
- designer
14
14
- sfi-image-nochange
15
15
---
16
16
17
17
# What is Designer (v2) in Azure Machine Learning?
18
18
19
-
Designer in Azure Machine Learning studio is a drag-and-drop user interface for building machine learning pipelines in Azure Machine Learning workspaces.
19
+
Designer in Azure Machine Learning studio is a drag-and-drop interface for building machine learning pipelines in Azure Machine Learning workspaces.
20
20
21
21
> [!IMPORTANT]
22
22
> Designer in Azure Machine Learning supports two types of pipelines, which use classic prebuilt (v1) or custom (v2) components. The two component types aren't compatible within pipelines. **This article applies to Designer (v2) with custom components.**
@@ -31,17 +31,17 @@ The following animated GIF shows how you can build a pipeline visually in Design
31
31
32
32
## Asset libraries
33
33
34
-
Designer uses building blocks from Azure Machine Learning asset libraries to create pipelines. The asset libraries include the following pipeline building blocks:
34
+
Designer creates pipelines using building blocks from Azure Machine Learning asset libraries. The asset libraries include these pipeline building blocks:
The **Data**, **Model**, and **Component** tabs on the left side of Designer show assets in your workspace and in all Azure Machine Learning [registries](./how-to-share-models-pipelines-across-workspaces-with-registries.md) that you have access to.
40
+
The **Data**, **Model**, and **Component** tabs on the left side of Designer show assets in your workspace and in all Azure Machine Learning [registries](./how-to-share-models-pipelines-across-workspaces-with-registries.md) that you can access.
41
41
42
42
:::image type="content" source="./media/concept-designer/asset-library.png" alt-text="Screenshot of the asset libraries filtered for one registry." lightbox= "./media/concept-designer/asset-library.png":::
43
43
44
-
To view assets from specific registries, select the **Registry name** filter. The assets you created in your current workspace are in the **Workspace** registry. The assets provided by Azure Machine Learning are in the **azureml** registry.
44
+
To view assets from specific registries, select the **Registry name** filter. Assets you created in your current workspace are in the **Workspace** registry. Assets provided by Azure Machine Learning are in the **azureml** registry.
45
45
46
46
To learn how to create data and component assets in your workspace, see the following articles:
47
47
@@ -50,11 +50,11 @@ To learn how to create data and component assets in your workspace, see the foll
50
50
51
51
## Pipelines
52
52
53
-
You can use Designer to visually build pipelines with your assets. You can either create new pipelines or clone and build on existing pipeline jobs.
53
+
You can use Designer to visually build pipelines with your assets. Create new pipelines or clone and build on existing pipeline jobs.
54
54
55
55
### New pipelines
56
56
57
-
Selecting the **+** symbol under **New pipeline** at the top of the Designer screen creates a new pipeline to build from scratch. Be sure to select the **Custom** option so you can use custom components.
57
+
Select the **+** symbol under **New pipeline** at the top of the Designer screen to create a new pipeline from scratch. Be sure to select the **Custom** option so you can use custom components.
58
58
59
59
:::image type="content" source="./media/concept-designer/new-pipeline.png" alt-text="Screenshot of selecting new pipeline with custom components." lightbox= "./media/concept-designer/new-pipeline.png":::
60
60
@@ -70,18 +70,18 @@ You can edit a pipeline draft anytime by adding or removing components, configur
70
70
71
71
:::image type="content" source="./media/concept-designer/edit-pipeline.png" alt-text="Screenshot of a draft pipeline being edited." lightbox= "./media/concept-designer/edit-pipeline.png":::
72
72
73
-
A valid pipeline draft has the following characteristics:
73
+
A valid pipeline draft has these characteristics:
74
74
75
75
- Data assets can connect only to components.
76
-
- Components can connect only to either data assets or to other components.
76
+
- Components can connect only to data assets or to other components.
77
77
- All required input ports for components must have some connection to the data flow.
78
78
- All required parameters for each component must be set.
79
79
80
-
When you're ready to run your pipeline draft, you save it and submit it as a pipeline job.
80
+
When you're ready to run your pipeline draft, save it and submit it as a pipeline job.
81
81
82
82
### Pipeline jobs
83
83
84
-
Each time you run a pipeline, the pipeline configuration and results are stored in your workspace as a pipeline job. You can resubmit any past pipeline job, inspect it for troubleshooting or auditing, or clone it to create a new pipeline draft for further editing.
84
+
Each time you run a pipeline, the pipeline configuration and results are stored in your workspace as a pipeline job. You can resubmit any past pipeline job, inspect it for troubleshooting or auditing, or clone it to create a new pipeline draft for editing.
85
85
86
86
:::image type="content" source="./media/concept-designer/pipeline-job.png" alt-text="Screenshot of pipeline job list." lightbox= "./media/concept-designer/pipeline-job.png":::
87
87
@@ -91,10 +91,10 @@ You can edit and resubmit your pipelines. After submitting, you can see the line
91
91
92
92
### Cloned pipeline jobs
93
93
94
-
If you want to base a new pipeline on an existing pipeline job in the workspace, you can clone the job into a new pipeline draft to continue editing.
94
+
To base a new pipeline on an existing pipeline job in the workspace, clone the job into a new pipeline draft to continue editing.
95
95
96
96
:::image type="content" source="./media/concept-designer/pipeline-clone.png" alt-text="Screenshot of a pipeline job in the workspace with the clone button highlighted." lightbox= "./media/concept-designer/pipeline-clone.png":::
97
-
After cloning, you can find out which pipeline job the new pipeline was cloned from by selecting **Show lineage**.
97
+
After cloning, you can find which pipeline job the new pipeline was cloned from by selecting **Show lineage**.
98
98
99
99
:::image type="content" source="./media/concept-designer/draft-show-lineage.png" alt-text="Screenshot showing the draft lineage after selecting Show lineage." lightbox= "./media/concept-designer/draft-show-lineage.png":::
An Azure Machine Learning pipeline is an independently executable workflow of a complete machine learning task. An Azure Machine Learning pipeline helps to standardize the best practices of producing a machine learning model, enables the team to execute at scale, and improves the model building efficiency.
29
+
30
+
31
+
An Azure Machine Learning pipeline is a workflow that automates a complete machine learning task. It standardizes best practices, supports team collaboration, and improves efficiency.
30
32
31
33
## Why are Azure Machine Learning pipelines needed?
32
34
33
-
The core of a machine learning pipeline is to split a complete machine learning task into a multistep workflow. Each step is a manageable component that can be developed, optimized, configured, and automated individually. Steps are connected through well-defined interfaces. The Azure Machine Learning pipeline service automatically orchestrates all the dependencies between pipeline steps. This modular approach brings two key benefits:
34
-
-[Standardize the Machine learning operation (MLOps) practice and support scalable team collaboration](#standardize-the-mlops-practice-and-support-scalable-team-collaboration)
35
-
-[Training efficiency and cost reduction](#training-efficiency-and-cost-reduction)
35
+
36
+
-[Standardizes machine learning operations (MLOps) and supports scalable team collaboration](#standardize-the-mlops-practice-and-support-scalable-team-collaboration)
37
+
-[Improves training efficiency and reduces cost](#training-efficiency-and-cost-reduction)
38
+
39
+
A pipeline breaks a machine learning task into steps. Each step is a manageable component that can be developed and automated separately. Azure Machine Learning manages dependencies between steps. This modular approach:
40
+
- Standardizes MLOps and supports team collaboration
41
+
- Improves training efficiency and reduces cost
42
+
-[Standardizes machine learning operations (MLOps) and supports scalable team collaboration](#standardize-the-mlops-practice-and-support-scalable-team-collaboration)
43
+
-[Improves training efficiency and reduces cost](#training-efficiency-and-cost-reduction)
36
44
37
45
### Standardize the MLOps practice and support scalable team collaboration
38
46
39
-
Machine learning operation (MLOps) automates the process of building machine learning models and taking the model to production. This is a complex process. It usually requires collaboration from different teams with different skills. A well-defined machine learning pipeline can abstract this complex process into a multiple steps workflow, mapping each step to a specific task such that each team can work independently.
40
47
41
-
For example, a typical machine learning project includes the steps of data collection, data preparation, model training, model evaluation, and model deployment. Usually, the data engineers concentrate on data steps, data scientists spend most time on model training and evaluation, the machine learning engineers focus on model deployment and automation of the entire workflow. By leveraging machine learning pipeline, each team only needs to work on building their own steps. The best way of building steps is using [Azure Machine Learning component (v2)](concept-component.md), a self-contained piece of code that does one step in a machine learning pipeline. All these steps built by different users are finally integrated into one workflow through the pipeline definition. The pipeline is a collaboration tool for everyone in the project. The process of defining a pipeline and all its steps can be standardized by each company's preferred DevOps practice. The pipeline can be further versioned and automated. If the ML projects are described as a pipeline, then the best MLOps practice is already applied.
48
+
49
+
MLOps automates building and deploying models. Pipelines simplify this process by mapping each step to a specific task, so teams can work independently.
50
+
51
+
52
+
53
+
For example, a project may include data collection, preparation, training, evaluation, and deployment. Data engineers, scientists, and ML engineers each own their steps. Steps are best built as [components](concept-component.md), then integrated into a single workflow. Pipelines can be versioned, automated, and standardized by DevOps practices.
42
54
43
55
### Training efficiency and cost reduction
44
56
45
-
Besides being the tool to put MLOps into practice, the machine learning pipeline also improves large model training's efficiency and reduces cost. Taking modern natural language model training as an example. It requires pre-processing large amounts of data and GPU intensive transformer model training. It takes hours to days to train a model each time. When the model is being built, the data scientist wants to test different training code or hyperparameters and run the training many times to get the best model performance. For most of these trainings, there's usually small changes from one training to another one. It will be a significant waste if every time the full training from data processing to model training takes place. By using machine learning pipeline, it can automatically calculate which steps result is unchanged and reuse outputs from previous training. Additionally, the machine learning pipeline supports running each step on different computation resources. Such that, the memory heavy data processing work and run-on high memory CPU machines, and the computation intensive training can run on expensive GPU machines. By properly choosing which step to run on which type of machines, the training cost can be significantly reduced.
57
+
58
+
59
+
Pipelines also improve efficiency and reduce costs. They reuse outputs from unchanged steps and let you run each step on the best compute resource for the task.
46
60
47
61
## Getting started best practices
48
62
49
-
Depending on what a machine learning project already has, the starting point of building a machine learning pipeline might vary. There are a few typical approaches to building a pipeline.
50
63
51
-
The first approach usually applies to the team that hasn't used pipeline before and wants to take some advantage of pipeline like MLOps. In this situation, data scientists typically have developed some machine learning models on their local environment using their favorite tools. Machine learning engineers need to take data scientists' output into production. The work involves cleaning up some unnecessary code from original notebook or Python code, changes the training input from local data to parameterized values, split the training code into multiple steps as needed, perform unit test of each step, and finally wraps all steps into a pipeline.
52
64
53
-
Once the teams get familiar with pipelines and want to do more machine learning projects using pipelines, they'll find the first approach is hard to scale. The second approach is set up a few pipeline templates, each try to solve one specific machine learning problem. The template predefines the pipeline structure including how many steps, each step's inputs and outputs, and their connectivity. To start a new machine learning project, the team first forks one template repo. The team leader then assigns members which step they need to work on. The data scientists and data engineers do their regular work. When they're happy with their result, they structure their code to fit in the pre-defined steps. Once the structured codes are checked-in, the pipeline can be executed or automated. If there's any change, each member only needs to work on their piece of code without touching the rest of the pipeline code.
65
+
You can build a pipeline in several ways, depending on your starting point.
66
+
67
+
54
68
55
-
Once a team has built a collection of machine learnings pipelines and reusable components, they could start to build the machine learning pipeline from cloning previous pipeline or tie existing reusable component together. At this stage, the team's overall productivity will be improved significantly.
69
+
If you are new to pipelines, start by splitting existing code into steps, parameterizing inputs, and wrapping everything into a pipeline.
70
+
71
+
72
+
73
+
To scale, use pipeline templates for common problems. Teams fork a template, work on assigned steps, and update only their part as needed.
74
+
75
+
76
+
77
+
With reusable pipelines and components, teams can quickly create new workflows by cloning or combining existing pieces.
56
78
57
79
:::moniker range="azureml-api-2"
58
-
Azure Machine Learning offers different methods to build a pipeline. For users who are familiar with DevOps practices, we recommend using [CLI](how-to-create-component-pipelines-cli.md). For data scientists who are familiar with python, we recommend writing pipelines using the [Azure Machine Learning SDK v2](how-to-create-component-pipeline-python.md). For users who prefer to use the UI, they could use the [designer to build pipelines by using registered components](how-to-create-component-pipelines-ui.md).
80
+
81
+
82
+
You can build pipelines using the [CLI](how-to-create-component-pipelines-cli.md), [Python SDK](how-to-create-component-pipeline-python.md), or [Designer UI](how-to-create-component-pipelines-ui.md).
59
83
60
84
61
85
:::moniker-end
62
86
63
87
<aname="compare"></a>
64
88
## Which Azure pipeline technology should I use?
65
89
66
-
The Azure cloud provides several types of pipeline, each with a different purpose. The following table lists the different pipelines and what they're used for:
90
+
91
+
92
+
Azure provides several types of pipelines for different purposes:
0 commit comments