Merge pull request #232192 from lgayhardt/amlpipelinecomp0323

Pipeline Component

Author: AnnaMHuff

articles/machine-learning/how-to-use-pipeline-component.md

@@ -0,0 +1,92 @@
+---
+title: How to use pipeline component in pipeline 
+titleSuffix: Azure Machine Learning
+description: How to use pipeline component to build nested pipeline job in Azure Machine Learning pipeline using CLI v2 and Python SDK
+services: machine-learning
+ms.service: machine-learning
+ms.subservice: mlops
+ms.topic: how-to
+author: cloga
+ms.author: lochen
+ms.reviewer: lagayhar
+ms.date: 04/12/2023
+ms.custom: sdkv2, cliv2, 
+---
+
+# How to use pipeline component to build nested pipeline job (V2) (preview)
+
+[!INCLUDE [dev v2](../../includes/machine-learning-dev-v2.md)]
+
+When developing a complex machine learning pipeline, it's common to have sub-pipelines that use multi-step  to perform tasks such as data preprocessing and model training. These sub-pipelines can be developed and tested standalone. Pipeline component groups multi-step as a component that can be used as a single step to create complex pipelines. Which will help you share your work and better collaborate with team members.
+
+By using a pipeline component, the author can focus on developing sub-tasks and easily integrate them with the entire pipeline job. Furthermore, a pipeline component has a well-defined interface in terms of inputs and outputs, which means that user of the pipeline component doesn't need to know the implementation details of the component.
+
+In this article, you'll learn how to use pipeline component in Azure Machine Learning pipeline.
+
+[!INCLUDE [machine-learning-preview-generic-disclaimer](../../includes/machine-learning-preview-generic-disclaimer.md)]
+
+## Prerequisites
+
+- Understand how to use Azure Machine Learning pipeline with [CLI v2](how-to-create-component-pipelines-cli.md) and [SDK v2](how-to-create-component-pipeline-python.md).
+- Understand what is [component](concept-component.md) and how to use component in Azure Machine Learning pipeline.
+- Understand what is a [Azure Machine Learning pipeline](concept-ml-pipelines.md)
+
+## The difference between pipeline job and pipeline component
+
+In general, pipeline component is similar to pipeline job. They're both consist of a group of jobs/components. 
+
+Here are some main differences you need aware when defining pipeline component:
+
+- Pipeline component only defines the interface of inputs/outputs, which means when defining a pipeline component you need to explicitly define the type of inputs/outputs instead of directly assigning values to them.
+- Pipeline component can't have runtime settings, you can't hard-code compute, or data node in the pipeline component. Instead you need to promote them as pipeline level inputs and assign values during runtime.
+- Pipeline level settings such as default_datastore and default_compute are also runtime settings. They aren't part of pipeline component definition.
+
+### CLI v2
+
+The example used in this article can be found in [azureml-example repo](https://github.com/Azure/azureml-examples). Navigate to *azureml-examples/cli/jobs/pipelines-with-components/pipeline_with_pipeline_component* to check the example.
+
+You can use multi-components to build a pipeline component. Similar to how you built pipeline job with component. This is two step pipeline component.
+
+:::code language="yaml" source="~/azureml-examples-main/cli/jobs/pipelines-with-components/pipeline_with_pipeline_component/pipeline_with_train_eval_pipeline_component/components/train_pipeline_component.yml" highlight="7-48":::
+
+When reference pipeline component to define child job in a pipeline job, just like reference other type of component. You can provide runtime settings such as default_datastore, default_compute in pipeline job level, any parameter you want to change during run time need promote as pipeline job inputs, otherwise, they'll be hard-code in next pipeline component. We're support to promote compute as pipeline component input to support heterogenous pipeline, which may need different compute target in different steps.
+
+:::code language="yaml" source="~/azureml-examples-main/cli/jobs/pipelines-with-components/pipeline_with_pipeline_component/pipeline_with_train_eval_pipeline_component/pipeline.yml" highlight="11-16,23-25,60":::
+
+### Python SDK
+
+The python SDK example can be found in [azureml-example repo](https://github.com/Azure/azureml-examples). Navigate to *azureml-examples/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component/pipeline_with_train_eval_pipeline_component* to check the example.
+
+You can define a pipeline component using a Python function, which is similar to defining a pipeline job using a function. You can also promote the compute of some step to be used as inputs for the pipeline component.
+
+[!notebook-python[] (~/azureml-examples-main/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component/pipeline_with_train_eval_pipeline_component/pipeline_with_train_eval_pipeline_component.ipynb?name=pipeline-component)]
+
+You can use pipeline component as a step like other components in pipeline job.
+
+[!notebook-python[] (~/azureml-examples-main/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component/pipeline_with_train_eval_pipeline_component/pipeline_with_train_eval_pipeline_component.ipynb?name=pipeline-component-pipeline-job)]
+
+## Pipeline job with pipeline component in studio
+
+You can use `az ml component create` or `ml_client.components.create_or_update` to register pipeline component as a registered component. After that you can view the component in asset library and component list page.
+
+### Using pipeline component to build pipeline job
+
+After you register the pipeline component, you can drag and drop the pipeline component into the designer canvas and use the UI to build pipeline job.
+
+:::image type="content" source="./media/how-to-use-pipeline-component/pipeline-component-authoring.png" alt-text="Screenshot of the designer canvas page to build pipeline job with pipeline component." lightbox= "./media/how-to-use-pipeline-component/pipeline-component-authoring.png":::
+
+### View pipeline job using pipeline component
+
+After submitted pipeline job, you can go to pipeline job detail page to change pipeline component status, you can also drill down to child component in pipeline component to debug specific component.
+
+:::image type="content" source="./media/how-to-use-pipeline-component/pipeline-component-right-panel.png" alt-text="Screenshot of view pipeline component on the pipeline job detail page." lightbox= "./media/how-to-use-pipeline-component/pipeline-component-right-panel.png":::
+
+## Sample notebooks
+
+- [nyc_taxi_data_regression_with_pipeline_component](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component/nyc_taxi_data_regression_with_pipeline_component/nyc_taxi_data_regression_with_pipeline_component.ipynb)
+- [pipeline_with_train_eval_pipeline_component](https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/1j_pipeline_with_pipeline_component/pipeline_with_train_eval_pipeline_component/pipeline_with_train_eval_pipeline_component.ipynb)
+
+## Next steps
+- [YAML reference for pipeline component](reference-yaml-component-pipeline.md)
+- [Track an experiment](how-to-log-view-metrics.md)
+- [Deploy a trained model](how-to-deploy-managed-online-endpoints.md)

articles/machine-learning/media/how-to-use-pipeline-component/pipeline-component-authoring.png

@@ -0,0 +1,73 @@
+---
+title: 'CLI (v2) pipeline component YAML schema'
+titleSuffix: Azure Machine Learning
+description: Reference documentation for the CLI (v2) pipeline component YAML schema.
+services: machine-learning
+ms.service: machine-learning
+ms.subservice: core
+ms.topic: reference
+ms.custom: cliv2
+author: cloga
+ms.author: lochen
+ms.date: 04/12/2023
+ms.reviewer: lagayhar
+---
+
+# CLI (v2) pipeline component YAML schema (preview)
+
+[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
+
+The source JSON schema can be found at https://azuremlschemas.azureedge.net/latest/pipelineComponent.schema.json.
+
+[!INCLUDE [schema note](../../includes/machine-learning-preview-old-json-schema-note.md)]
+
+[!INCLUDE [machine-learning-preview-generic-disclaimer](../../includes/machine-learning-preview-generic-disclaimer.md)]
+
+## YAML syntax
+
+| Key | Type | Description | Allowed values | Default value |
+| --- | ---- | ----------- | -------------- | ------------- |
+| `$schema` | string | The YAML schema. If you use the Azure Machine Learning VS Code extension to author the YAML file, including `$schema` at the top of your file enables you to invoke schema and resource completions. | | |
+| `type` | const | The type of component. | `pipeline` | `pipeline` |
+| `name` | string | **Required.** Name of the component. Must start with lowercase letter. Allowed characters are lowercase letters, numbers, and underscore(_). Maximum length is 255 characters.| | |
+| `version` | string | Version of the component. If omitted, Azure Machine Learning will autogenerate a version. | | |
+| `display_name` | string | Display name of the component in the studio UI. It can be non-unique within the workspace. | | |
+| `description` | string | Description of the component. | | |
+| `tags` | object | Dictionary of tags for the component. | | |
+| `jobs` | object | **Required.** Dictionary of the set of individual jobs to run as steps within the pipeline. These jobs are considered child jobs of the parent pipeline job. <br><br> The key is the name of the step within the context of the pipeline job. This name is different from the unique job name of the child job. The value is the job specification, which can follow the [command job schema](reference-yaml-job-command.md#yaml-syntax) or [sweep job schema](reference-yaml-job-sweep.md#yaml-syntax). Currently only command jobs and sweep jobs can be run in a pipeline.  | | |
+| `inputs` | object | Dictionary of inputs to the pipeline job. The key is a name for the input within the context of the job and the value is the input value. <br><br> These pipeline inputs can be referenced by the inputs of an individual step job in the pipeline using the `${{ parent.inputs.<input_name> }}` expression. For more information on how to bind the inputs of a pipeline step to the inputs of the top-level pipeline job, see the [Expression syntax for binding inputs and outputs between steps in a pipeline job](reference-yaml-core-syntax.md#binding-inputs-and-outputs-between-steps-in-a-pipeline-job). | | |
+| `inputs.<input_name>` | number, integer, boolean, string or object | One of a literal value (of type number, integer, boolean, or string) or an object containing a [component input data specification](#component-input). | | |
+| `outputs` | object | Dictionary of output configurations of the pipeline job. The key is a name for the output within the context of the job and the value is the output configuration. <br><br> These pipeline outputs can be referenced by the outputs of an individual step job in the pipeline using the `${{ parents.outputs.<output_name> }}` expression. For more information on how to bind the inputs of a pipeline step to the inputs of the top-level pipeline job, see the [Expression syntax for binding inputs and outputs between steps in a pipeline job](reference-yaml-core-syntax.md#binding-inputs-and-outputs-between-steps-in-a-pipeline-job). | |
+| `outputs.<output_name>` | object | You can leave the object empty, in which case by default the output will be of type `uri_folder` and Azure Machine Learning will system-generate an output location for the output based on the following template path: `{settings.datastore}/azureml/{job-name}/{output-name}/`. File(s) to the output directory will be written via read-write mount. If you want to specify a different mode for the output, provide an object containing the [component output specification](#component-output). | |
+
+### Component input
+
+| Key | Type | Description | Allowed values | Default value |
+| --- | ---- | ----------- | -------------- | ------------- |
+| `type` | string | **Required.** The type of component input. [Learn more about data access](concept-data.md) | `number`, `integer`, `boolean`, `string`, `uri_file`, `uri_folder`, `mltable`, `mlflow_model`, `custom_model`| |
+| `description` | string | Description of the input. | | |
+| `default` | number, integer, boolean, or string | The default value for the input. | | |
+| `optional` | boolean | Whether the input is required. If set to `true`, you need use the command includes optional inputs with `$[[]]`| | `false` |
+| `min` | integer or number | The minimum accepted value for the input. This field can only be specified if `type` field is `number` or `integer`. | |
+| `max` | integer or number | The maximum accepted value for the input. This field can only be specified if `type` field is `number` or `integer`. | |
+| `enum` | array | The list of allowed values for the input. Only applicable if `type` field is `string`.| |
+
+### Component output
+
+| Key | Type | Description | Allowed values | Default value |
+| --- | ---- | ----------- | -------------- | ------------- |
+| `type` | string | **Required.** The type of component output. | `uri_file`, `uri_folder`, `mltable`, `mlflow_model`, `custom_model` | |
+| `description` | string | Description of the output. | | |
+
+## Remarks
+
+The `az ml component` commands can be used for managing Azure Machine Learning components.
+
+## Examples
+
+Examples are available in the [examples GitHub repository](https://github.com/Azure/azureml-examples/tree/lochen/pipeline-component-pup/cli/jobs/pipelines-with-components/pipeline_with_pipeline_component).
+
+## Next steps
+
+- [Install and use the CLI (v2)](how-to-configure-cli.md)
+- [Create ML pipelines using components](how-to-create-component-pipelines-cli.md)

articles/machine-learning/media/how-to-use-pipeline-component/pipeline-component-right-panel.png

@@ -1084,6 +1084,8 @@
               href: how-to-use-pipeline-ui.md
             - name: How to use parallel job in pipeline
               href: how-to-use-parallel-job-in-pipeline.md
+            - name: How to use pipeline component in pipeline
+              href: how-to-use-pipeline-component.md
             # v1
             - name: Create ML pipelines (Python)
               href: ./v1/how-to-create-machine-learning-pipelines.md
@@ -1231,6 +1233,8 @@
           href: reference-yaml-model.md
         - name: Schedule
           href: reference-yaml-schedule.md
+        - name: Pipeline component
+          href: reference-yaml-component-pipeline.md
         - name: Compute
           items:
             - name: Compute cluster (AmlCompute)