Sample Pipeline: StepSequence Migration SDK v1 to SDK v2 (#3667)

- Demonstrates a multi-step pipeline with dummy inputs and outputs
- Replicates the step sequence functionality available in AzureML SDK v1
- Serves as a reference for migrating pipelines from SDK v1 to SDK v2

Author: bhgtankita

.github/workflows/sdk-jobs-pipelines-stepsequence-pipeline_with_step_sequence_dummy_dependencies.yml

@@ -0,0 +1,86 @@
+name: sdk-jobs-pipelines-stepsequence-pipeline_with_step_sequence_dummy_dependencies
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "0 */8 * * *"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - sdk/python/jobs/pipelines/stepsequence/pipeline_with_step_sequence_dummy_dependencies.ipynb
+      - .github/workflows/sdk-jobs-pipelines-stepsequence-pipeline_with_step_sequence_dummy_dependencies.yml
+      - "!sdk/python/jobs/pipelines/stepsequence/**/*.md"
+      - sdk/python/dev-requirements.txt
+      - infra/bootstrapping/**
+      - sdk/python/setup.sh
+
+permissions:
+  id-token: write
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: check out repo
+      uses: actions/checkout@v3
+    - name: setup python
+      uses: actions/setup-python@v4
+      with:
+        python-version: "3.10"
+    - name: pip install notebook reqs
+      run: pip install -r sdk/python/dev-requirements.txt
+    - name: azure login
+      uses: azure/login@v1
+      with:
+        client-id: ${{ secrets.OIDC_AZURE_CLIENT_ID }}
+        tenant-id: ${{ secrets.OIDC_AZURE_TENANT_ID }}
+        subscription-id: ${{ secrets.OIDC_AZURE_SUBSCRIPTION_ID }}
+    - name: bootstrap resources
+      run: |
+          echo '${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}';
+          bash bootstrap.sh
+      working-directory: infra/bootstrapping
+      continue-on-error: false
+    - name: setup SDK
+      run: |
+          source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";
+          source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh";
+          bash setup.sh
+      working-directory: sdk/python
+      continue-on-error: true
+    - name: setup-cli
+      run: |
+          source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";
+          source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh";
+          bash setup.sh
+      working-directory: cli
+      continue-on-error: true
+    - name: Eagerly cache access tokens for required scopes
+      run: |
+          # Workaround for azure-cli's lack of support for ID token refresh
+          # Taken from: https://github.com/Azure/login/issues/372#issuecomment-2056289617
+
+          # Management
+          az account get-access-token --scope https://management.azure.com/.default --output none
+          # ML
+          az account get-access-token --scope https://ml.azure.com/.default --output none
+    - name: install ml extension
+      run: az extension add -n ml -y
+    - name: run jobs/pipelines/stepsequence/pipeline_with_step_sequence_dummy_dependencies.ipynb
+      run: |
+          source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh";
+          source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh";
+          bash "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh" generate_workspace_config "../../.azureml/config.json";
+          bash "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh" replace_template_values "pipeline_with_step_sequence_dummy_dependencies.ipynb";
+          [ -f "../../.azureml/config" ] && cat "../../.azureml/config";
+          papermill -k python pipeline_with_step_sequence_dummy_dependencies.ipynb pipeline_with_step_sequence_dummy_dependencies.output.ipynb
+      working-directory: sdk/python/jobs/pipelines/stepsequence
+    - name: upload notebook's working folder as an artifact
+      if: ${{ always() }}
+      uses: ./.github/actions/upload-artifact
+      with:
+        name: stepsequence
+        path: sdk/python/jobs/pipelines/stepsequence

sdk/python/README.md

@@ -265,6 +265,7 @@ Test Status is for branch - **_main_**
 |jobs|pipelines|[nyc_taxi_data_regression](jobs/pipelines/2c_nyc_taxi_data_regression/nyc_taxi_data_regression.ipynb)|Build pipeline with components for 5 jobs - prep data, transform data, train model, predict results and evaluate model performance|[![nyc_taxi_data_regression](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-pipelines-2c_nyc_taxi_data_regression-nyc_taxi_data_regression.yml/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-pipelines-2c_nyc_taxi_data_regression-nyc_taxi_data_regression.yml)|
 |jobs|pipelines|[image_classification_with_densenet](jobs/pipelines/2d_image_classification_with_densenet/image_classification_with_densenet.ipynb)|Create pipeline to train cnn image classification model|[![image_classification_with_densenet](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-pipelines-2d_image_classification_with_densenet-image_classification_with_densenet.yml/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-pipelines-2d_image_classification_with_densenet-image_classification_with_densenet.yml)|
 |jobs|pipelines|[image_classification_keras_minist_convnet](jobs/pipelines/2e_image_classification_keras_minist_convnet/image_classification_keras_minist_convnet.ipynb)|Create pipeline to train cnn image classification model with keras|[![image_classification_keras_minist_convnet](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-pipelines-2e_image_classification_keras_minist_convnet-image_classification_keras_minist_convnet.yml/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-pipelines-2e_image_classification_keras_minist_convnet-image_classification_keras_minist_convnet.yml)|
+|jobs|pipelines|[stepsequence](jobs/pipelines/stepsequence/pipeline_with_step_sequence_dummy_dependencies.ipynb)|Create pipeline demonstrating multi-step execution with dummy inputs/outputs, replicating AzureML SDK v1 step sequence behavior|[![stepsequence](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-pipelines-stepsequence-pipeline_with_step_sequence_dummy_dependencies.yml/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-pipelines-stepsequence-pipeline_with_step_sequence_dummy_dependencies.yml)|
 |jobs|single-step|[lightgbm-iris-sweep](jobs/single-step/lightgbm/iris/lightgbm-iris-sweep.ipynb)|Run **hyperparameter sweep** on a Command or CommandComponent|[![lightgbm-iris-sweep](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-single-step-lightgbm-iris-lightgbm-iris-sweep.yml/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-single-step-lightgbm-iris-lightgbm-iris-sweep.yml)|
 |jobs|single-step|[tutorial](jobs/single-step/pytorch/distributed-training-yolov5/yolov5/tutorial.ipynb)|*no description*|[![tutorial](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-single-step-pytorch-distributed-training-yolov5-yolov5-tutorial.yml/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-single-step-pytorch-distributed-training-yolov5-yolov5-tutorial.yml)|
 |jobs|single-step|[distributed-cifar10](jobs/single-step/pytorch/distributed-training/distributed-cifar10.ipynb)|*no description*|[![distributed-cifar10](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-single-step-pytorch-distributed-training-distributed-cifar10.yml/badge.svg?branch=main)](https://github.com/Azure/azureml-examples/actions/workflows/sdk-jobs-single-step-pytorch-distributed-training-distributed-cifar10.yml)|

sdk/python/jobs/pipelines/stepsequence/README.md

@@ -0,0 +1,13 @@
+---
+page_type: sample
+languages:
+- python
+products:
+- azure-machine-learning
+description: This sample shows how to run build pipeline with command_component decorator.
+---
+
+# Running a Pipeline job with step sequences and dummy dependencies
+This example shows how to create a pipeline with step sequences and dummy dependencies using Azure ML Python SDK v2..
+
+Please find the sample defined in [pipeline_with_step_sequence_dummy_dependencies.ipynb](pipeline_with_step_sequence_dummy_dependencies.ipynb).

sdk/python/jobs/pipelines/stepsequence/echo_component.yml

@@ -0,0 +1,14 @@
+$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
+name: echo_component
+display_name: Echo Hello
+version: 1
+type: command
+description: A component that just echoes hello
+
+outputs:
+  output_data:
+    type: uri_folder
+
+command: echo "hello" > ${{outputs.output_data}}/message.txt
+
+environment: azureml://registries/azureml/environments/sklearn-1.5/labels/latest