Merge pull request #239398 from lgayhardt/designerpipeline0523

Designer moving files to V1 and pipeline debug

Author: ttorble

articles/machine-learning/.openpublishing.redirection.machine-learning.json

@@ -3839,6 +3839,61 @@
         "source_path_from_root": "/articles/machine-learning/how-to-deploy-batch-with-rest.md",
         "redirect_url": "/azure/machine-learning/how-to-access-data-batch-endpoints-jobs?tabs=rest",
         "redirect_document_id": false
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/tutorial-designer-automobile-price-train-score.md",
+        "redirect_url": "/azure/machine-learning/v1/tutorial-designer-automobile-price-train-score",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/tutorial-designer-automobile-price-deploy.md",
+        "redirect_url": "/azure/machine-learning/v1/tutorial-designer-automobile-price-deploy",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/samples-designer.md",
+        "redirect_url": "/azure/machine-learning/v1/samples-designer",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/algorithm-cheat-sheet.md",
+        "redirect_url": "/azure/machine-learning/v1/algorithm-cheat-sheet",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/how-to-select-algorithms.md",
+        "redirect_url": "/azure/machine-learning/v1/how-to-select-algorithms",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/how-to-use-pipeline-ui.md",
+        "redirect_url": "/azure/machine-learning/how-to-debug-pipeline-failure",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/how-to-designer-transform-data.md",
+        "redirect_url": "/azure/machine-learning/v1/how-to-designer-transform-data",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/how-to-use-pipeline-parameter.md",
+        "redirect_url": "/azure/machine-learning/v1/how-to-use-pipeline-parameter",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/how-to-retrain-designer.md",
+        "redirect_url": "/azure/machine-learning/v1/how-to-retrain-designer",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/how-to-run-batch-predictions-designer.md",
+        "redirect_url": "/azure/machine-learning/v1/how-to-run-batch-predictions-designer",
+        "redirect_document_id": true
+    },
+    {
+        "source_path_from_root": "/articles/machine-learning/how-to-designer-python.md",
+        "redirect_url": "/azure/machine-learning/v1/how-to-designer-python",
+        "redirect_document_id": true
     }
 ]
 }

articles/machine-learning/component-reference/execute-python-script.md

@@ -16,7 +16,7 @@ ms.date: 06/15/2021
 
 This article describes the Execute Python Script component in Azure Machine Learning designer.
 
-Use this component to run Python code. For more information about the architecture and design principles of Python, see [how run Python code in Azure Machine Learning designer](../how-to-designer-python.md).
+Use this component to run Python code. For more information about the architecture and design principles of Python, see [how run Python code in Azure Machine Learning designer](../v1/how-to-designer-python.md).
 
 With Python, you can perform tasks that existing components don't support, such as:
 

articles/machine-learning/concept-designer.md

@@ -1,125 +1,103 @@
 ---
-title: What is the Azure Machine Learning designer?
+title: What is the Azure Machine Learning designer(v2)?
 titleSuffix: Azure Machine Learning
-description: Learn what the Azure Machine Learning designer is and what tasks you can use it for. The drag-and-drop UI enables model training and deployment. 
+description: Learn what the Azure Machine Learning designer is and what tasks you can use it for. The drag-and-drop UI enables customer to build machine learning pipeline. 
 services: machine-learning
 ms.service: machine-learning
 ms.subservice: core
 ms.topic: conceptual
 ms.author: lagayhar
 ms.reviewer: lagayhar
 author: lgayhardt
-ms.date: 08/03/2022
-ms.custom: designer, training
-monikerRange: 'azureml-api-1 || azureml-api-2'
+ms.date: 05/25/2023
+ms.custom: designer
 ---
 
-# What is Azure Machine Learning designer? 
+# What is Azure Machine Learning designer(v2)?
 
-Azure Machine Learning designer is a drag-and-drop interface used to train and deploy models in Azure Machine Learning. This article describes the tasks you can do in the designer.
+Azure Machine Learning designer is a drag-and-drop UI interface to build pipeline in Azure Machine Learning.
 
- - To get started with the designer, see [Tutorial: Train a no-code regression model](tutorial-designer-automobile-price-train-score.md). 
- - To learn about the components available in the designer, see the [Algorithm and component reference](./algorithm-module-reference/module-reference.md).
+As shown in below GIF, you can build a pipeline visually by drag and drop your building blocks and connect them.
 
-![Azure Machine Learning designer example](./media/concept-designer/designer-drag-and-drop.gif)
+:::image type="content" source="./media/concept-designer/designer-drag-and-drop.gif" alt-text="GIF of a building a pipeline in the designer." lightbox= "./media/concept-designer/designer-drag-and-drop.gif":::
 
-The designer uses your Azure Machine Learning [workspace](concept-workspace.md) to organize shared resources such as:
 
-+ [Pipelines](#pipeline)
-+ [Data](#data)
-+ [Compute resources](#compute)
-:::moniker range="azureml-api-2"
-+ [Registered models](concept-model-management-and-deployment.md#register-and-track-machine-learning-models)
-:::moniker-end
-:::moniker range="azureml-api-1"
-+ [Registered models](v1/concept-azure-machine-learning-architecture.md#models)
-:::moniker-end
-+ [Published pipelines](#publish)
-+ [Real-time endpoints](#deploy)
+>[!Note]
+> Designer supports two types of components, classic prebuilt components（v1） and custom components(v2). These two types of components are NOT compatible.
+>
+>Classic prebuilt components provide prebuilt components majorly for data processing and traditional machine learning tasks like regression and classification. This type of component continues to be supported but will not have any new components added.
+>
+>Custom components allow you to wrap your own code as a component. It supports sharing components across workspaces and seamless authoring across Studio, CLI v2, and SDK v2 interfaces.
+>
+>For new projects, we highly suggest you use custom component, which is compatible with AzureML V2 and will keep receiving new updates.
+>
+>This article applies to custom components.
 
-## Model training and deployment
 
-Use a visual canvas to build an end-to-end machine learning workflow. Train, test, and deploy models all in the designer:
+## Assets
 
-+ Drag-and-drop [data assets](#data) and [components](#component) onto the canvas.
-+ Connect the components to create a [pipeline draft](#pipeline-draft).
-+ Submit a [pipeline run](#pipeline-job) using the compute resources in your Azure Machine Learning workspace.
-+ Convert your **training pipelines** to **inference pipelines**.
-+ [Publish](#publish) your pipelines to a REST **pipeline endpoint** to submit a new pipeline that runs with different parameters and data assets.
-    + Publish a **training pipeline** to reuse a single pipeline to train multiple models while changing parameters and data assets.
-    + Publish a **batch inference pipeline** to make predictions on new data by using a previously trained model.
-+ [Deploy](#deploy) a **real-time inference pipeline** to an online endpoint to make predictions on new data in real time.
+The building blocks of pipeline are called assets in Azure Machine Learning, which includes:
+ - [Data](./concept-data.md)
+ - Model
+ - [Component](./concept-component.md)
 
-![Workflow diagram for training, batch inference, and real-time inference in the designer](./media/concept-designer/designer-workflow-diagram.png)
+Designer has an asset library on the left side, where you can access all the assets you need to create your pipeline. It shows both the assets you created in your workspace, and the assets shared in [registry](./how-to-share-models-pipelines-across-workspaces-with-registries.md) that you have permission to access.
 
-## Pipeline
-
-A [pipeline](concept-ml-pipelines.md) consists of data assets and analytical components, which you connect. Pipelines have many uses: you can make a pipeline that trains a single model, or one that trains multiple models. You can create a pipeline that makes predictions in real time or in batch, or make a pipeline that only cleans data. Pipelines let you reuse your work and organize your projects.
-
-### Pipeline draft
+:::image type="content" source="./media/concept-designer/asset-library.png" alt-text="Screenshot of the asset library." lightbox= "./media/concept-designer/asset-library.png":::
 
-As you edit a pipeline in the designer, your progress is saved as a **pipeline draft**. You can edit a pipeline draft at any point by adding or removing components, configuring compute targets, creating parameters, and so on.
-
-A valid pipeline has these characteristics:
 
-* Data assets can only connect to components.
-* components can only connect to either data assets or other components.
-* All input ports for components must have some connection to the data flow.
-* All required parameters for each component must be set.
+To see assets from a specific registry, select the Registry name filter above the asset library. The assets you created in your current workspace are in the registry = workspace. The assets provided by Azure Machine Learning are in the registry = azureml.
 
-When you're ready to run your pipeline draft, you submit a pipeline job.
+Designer only shows the assets that you created and named in your workspace. You won't see any unnamed assets in the asset library. To learn how to create data and component assets, read these articles:
 
-### Pipeline job
+- [How to create data asset](./how-to-create-data-assets.md)
+- [How to create component](./how-to-create-component-pipelines-ui.md#register-component-in-your-workspace)
 
-Each time you run a pipeline, the configuration of the pipeline and its results are stored in your workspace as a **pipeline job**. You can go back to any pipeline job to inspect it for troubleshooting or auditing. **Clone** a pipeline job to create a new pipeline draft for you to edit.
-
-Pipeline jobs are grouped into experiments to organize job history. You can set the experiment for every pipeline job. 
-
-## Data
+## Pipeline
 
-A machine learning data asset makes it easy to access and work with your data. Several [sample data assets](samples-designer.md#datasets) are included in the designer for you to experiment with. You can [register](how-to-create-register-datasets.md) more data assets as you need them.
+Designer is a tool that lets you create pipelines with your assets in a visual way. When you use designer, you'll encounter two concepts related to pipelines: pipeline draft and pipeline jobs.
 
-## Component
+![pipeline-draft-and-pipeline-job-list](./media/concept-designer/pipeline-draft-and-job.png)
 
-A component is an algorithm that you can perform on your data. The designer has several components ranging from data ingress functions to training, scoring, and validation processes.
+:::image type="content" source="./media/concept-designer/pipeline-draft-and-job.png" alt-text="Screenshot of pipeline draft and pipeline job list." lightbox= "./media/concept-designer/pipeline-draft-and-job.png":::
 
-A component may have a set of parameters that you can use to configure the component's internal algorithms. When you select a component on the canvas, the component's parameters are displayed in the Properties pane to the right of the canvas. You can modify the parameters in that pane to tune your model. You can set the compute resources for individual components in the designer. 
+### Pipeline draft
 
-:::image type="content" source="./media/concept-designer/properties.png" alt-text="Component properties":::
+As you edit a pipeline in the designer, your progress is saved as a **pipeline draft**. You can edit a pipeline draft at any point by adding or removing components, configuring compute targets, creating parameters, and so on.
 
+A valid pipeline draft has these characteristics:
 
-For some help navigating through the library of machine learning algorithms available, see [Algorithm & component reference overview](component-reference/component-reference.md). For help with choosing an algorithm, see the [Azure Machine Learning Algorithm Cheat Sheet](algorithm-cheat-sheet.md).
+- Data assets can only connect to components.
+- Components can only connect to either data assets or other components.
+- All required input ports for components must have some connection to the data flow.
+- All required parameters for each component must be set.
 
-## <a name="compute"></a> Compute resources
+When you're ready to run your pipeline draft, you submit a pipeline job.
 
-Use compute resources from your workspace to run your pipeline and host your deployed models as online endpoints or pipeline endpoints (for batch inference). The supported compute targets are:
+### Pipeline job
 
-| Compute target | Training | Deployment |
-| ---- |:----:|:----:|
-| Azure Machine Learning compute | ✓ | |
-| Azure Kubernetes Service | | ✓ |
+Each time you run a pipeline, the configuration of the pipeline and its results are stored in your workspace as a **pipeline job**. You can go back to any pipeline job to inspect it for troubleshooting or auditing. **Clone** a pipeline job creates a new pipeline draft for you to continue editing.
 
-Compute targets are attached to your [Azure Machine Learning workspace](concept-workspace.md). You manage your compute targets in your workspace in the [Azure Machine Learning studio](https://ml.azure.com).
+## Approaches to build pipeline in designer
 
-## Deploy
+### Create new pipeline from scratch
 
-To perform real-time inferencing, you must deploy a pipeline as an [online endpoint](concept-endpoints-online.md). The online endpoint creates an interface between an external application and your scoring model. A call to an online endpoint returns prediction results to the application in real time. To make a call to an online endpoint, you pass the API key that was created when you deployed the endpoint. The endpoint is based on REST, a popular architecture choice for web programming projects.
+You can create a new pipeline and build from scratch. Remember to select the **Custom component** option when you create the pipeline in designer.
 
-Online endpoints must be deployed to an Azure Kubernetes Service cluster.
+:::image type="content" source="./media/how-to-create-component-pipelines-ui/new-pipeline.png" alt-text="Screenshot showing to select custom component." lightbox= "./media/how-to-create-component-pipelines-ui/new-pipeline.png":::
 
-To learn how to deploy your model, see [Tutorial: Deploy a machine learning model with the designer](tutorial-designer-automobile-price-deploy.md).
+### Clone an existing pipeline job
 
-## Publish
+If you would like to work based on an existing pipeline job in the workspace, you can easily clone it into a new pipeline draft to continue editing.
 
-You can also publish a pipeline to a **pipeline endpoint**. Similar to an online endpoint, a pipeline endpoint lets you submit new pipeline jobs from external applications using REST calls. However, you cannot send or receive data in real time using a pipeline endpoint.
+:::image type="content" source="./media/how-to-debug-pipeline-failure/job-detail-clone.png" alt-text="Screenshot of a pipeline job in the workspace with the clone button highlighted." lightbox= "./media/how-to-debug-pipeline-failure/job-detail-clone.png":::
 
-Published pipelines are flexible, they can be used to train or retrain models, [perform batch inferencing](how-to-run-batch-predictions-designer.md), process new data, and much more. You can publish multiple pipelines to a single pipeline endpoint and specify which pipeline version to run.
+After cloning, you can also know which pipeline job it's cloned from by selecting **Show lineage**.
 
-A published pipeline runs on the compute resources you define in the pipeline draft for each component.
+:::image type="content" source="./media/how-to-debug-pipeline-failure/draft-show-lineage.png" alt-text="Screenshot showing the draft lineage after selecting show lineage button." lightbox= "./media/how-to-debug-pipeline-failure/draft-show-lineage.png":::
 
-The designer creates the same [PublishedPipeline](/python/api/azureml-pipeline-core/azureml.pipeline.core.graph.publishedpipeline) object as the SDK.
+You can edit your pipeline and then submit again. After submitting, you can see the lineage between the job you submit and the original job by selecting **Show lineage** in the job detail page.
 
-## Next steps
+## Next step
 
-* Learn the fundamentals of predictive analytics and machine learning with [Tutorial: Predict automobile price with the designer](tutorial-designer-automobile-price-train-score.md)
-* Learn how to modify existing [designer samples](samples-designer.md) to adapt them to your needs.
+- [Create pipeline with components (UI)](./how-to-create-component-pipelines-ui.md)

articles/machine-learning/concept-vulnerability-management.md

@@ -148,7 +148,7 @@ For code-based training experiences, you control which Azure Machine Learning en
 
 * Automated ML jobs run on environments that layer on top of Azure Machine Learning [base docker images](https://github.com/Azure/AzureML-Containers).
 
- * Designer jobs are compartmentalized into [Components](concept-designer.md#component). Each component has its own environment that layers on top of the Azure Machine Learning base docker images. For more information on components, see the [Component reference](./component-reference/component-reference.md).  
+ * Designer jobs are compartmentalized into [Components](./v1/concept-designer.md#component). Each component has its own environment that layers on top of the Azure Machine Learning base docker images. For more information on components, see the [Component reference](./component-reference/component-reference.md).  
 
 ## Next steps
 

articles/machine-learning/how-to-create-component-pipeline-python.md

@@ -293,7 +293,7 @@ You can open the `Link to Azure Machine Learning studio`, which is the job detai
 
 :::image type="content" source="./media/how-to-create-component-pipeline-python/pipeline-ui.png" alt-text="Screenshot of the pipeline job detail page." lightbox ="./media/how-to-create-component-pipeline-python/pipeline-ui.png":::
 
-You can check the logs and outputs of each component by right clicking the component, or select the component to open its detail pane. To learn more about how to debug your pipeline in UI, see [How to use studio UI to build and debug Azure Machine Learning pipelines](how-to-use-pipeline-ui.md).
+You can check the logs and outputs of each component by right clicking the component, or select the component to open its detail pane. To learn more about how to debug your pipeline in UI, see [How to use debug pipeline failure](how-to-debug-pipeline-failure.md).
 
 ## (Optional) Register components to workspace