Merge pull request #221583 from lgayhardt/pipelines122022

Pipeline debug UI

Author: v-stsavell

articles/machine-learning/how-to-use-pipeline-ui.md

@@ -9,7 +9,7 @@ services: machine-learning
 ms.service: machine-learning
 ms.subservice: core
 ms.topic: how-to
-ms.date: 05/10/2022
+ms.date: 12/22/2022
 ms.custom: designer, event-tier1-build-2022
 ---
 
@@ -42,7 +42,7 @@ Then you can drag and drop either built-in components or custom components to th
 
 Now you've built your pipeline. Select **Submit** button above the canvas, and configure your pipeline job.
 
-:::image type="content" source="./media/how-to-use-pipeline-ui/submit-pipeline.png" alt-text="Screenshot showing set up pipeline job with the submit button highlighted." lightbox= "./media/how-to-use-pipeline-ui/submit-pipeline.png":::
+:::image type="content" source="./media/how-to-use-pipeline-ui/submit-pipeline.png" alt-text="Screenshot showing setup pipeline job with the submit button highlighted." lightbox= "./media/how-to-use-pipeline-ui/submit-pipeline.png":::
 
 After you submit your pipeline job, you'll see a submitted job list in the left pane, which shows all the pipeline job you create from the current pipeline draft in the same session. There's also notification popping up from the notification center. You can select through the pipeline job link in the submission list or the notification to check pipeline job status or debugging.
 
@@ -106,6 +106,170 @@ After cloning, you can also know which pipeline job it's cloned from by selectin
 
 You can edit your pipeline and then submit again. After submitting, you can see the lineage between the job you submit and the original job by selecting **Show lineage** in the job detail page.
 
+## Compare different pipelines to debug failure or other unexpected issues (preview)
+
+Pipeline comparison identifies the differences (including topology, component properties, and job properties) between multiple jobs. For example you can compare a successful pipeline and a failed pipeline, which helps you find what modifications make your pipeline fail.
+
+Two major scenarios where you can use pipeline comparison to help with debugging:
+
+- Debug your failed pipeline job by comparing it to a completed one.
+- Debug your failed node in a pipeline by comparing it to a similar completed one.
+
+To enable this feature:
+
+1. Navigate to Azure Machine Learning studio UI.
+2. Select **Manage preview features** (megaphone icon) among the icons on the top right side of the screen.
+3. In **Managed preview feature** panel, toggle on **Compare pipeline jobs to debug failures or unexpected issues** feature.
+
+:::image type="content" source="./media/how-to-use-pipeline-ui/enable-preview.png" alt-text="Screenshot of manage preview features toggled on." lightbox= "./media/how-to-use-pipeline-ui/enable-preview.png":::
+
+### How to debug your failed pipeline job by comparing it to a completed one
+
+During iterative model development, you may have a baseline pipeline, and then do some modifications such as changing a parameter, dataset or compute resource, etc. If your new pipeline failed, you can use pipeline comparison to identify what has changed by comparing it to the baseline pipeline, which could help with figuring out why it failed.
+
+#### Compare a pipeline with its parent
+
+The first thing you should check when debugging is to locate the failed node and check the logs.
+
+For example, you may get an error message showing that your pipeline failed due to out-of-memory. If your pipeline is cloned from a completed parent pipeline, you can use pipeline comparison to see what has changed.
+
+1. Select **Show lineage**.
+1. Select the link under "Cloned From". This will open a new browser tab with the parent pipeline.
+
+      :::image type="content" source="./media/how-to-use-pipeline-ui/cloned-from.png" alt-text="Screenshot showing the cloned from link, with the previous step, the lineage button highlighted." lightbox= "./media/how-to-use-pipeline-ui/cloned-from.png":::
+
+1. Select **Add to compare** on the failed pipeline and the parent pipeline. This will add them in the comparison candidate list.
+
+      :::image type="content" source="./media/how-to-use-pipeline-ui/comparison-list.png" alt-text="Screenshot showing the comparison list with a parent and child pipeline added." lightbox= "./media/how-to-use-pipeline-ui/comparison-list.png":::
+
+### Compare topology
+
+Once the two pipelines are added to the comparison list, you'll have two options: **Compare detail** and **Compare graph**. **Compare graph** allows you to compare pipeline topology.
+
+**Compare graph** shows you the graph topology changes between pipeline A and B. The special nodes in pipeline A are highlighted in red and marked with "A only". The special nodes in pipeline B are in green and marked with "B only". The shared nodes are in gray. If there are differences on the shared nodes, what has been changed is shown on the top of node.
+
+There are three categories of changes with summaries viewable in the detail page, parameter change, input source, pipeline component. When the pipeline component is changed this means that there's a topology change inside or an inner node parameter change, you can select the folder icon on the pipeline component node to dig down into the details. Other changes can be detected by viewing the colored nodes in the compare graph.
+
+   :::image type="content" source="./media/how-to-use-pipeline-ui/parameter-changed.png" alt-text="Screenshot showing the parameter changed and the component information tab." lightbox= "./media/how-to-use-pipeline-ui/parameter-changed.png":::
+
+### Compare pipeline meta info and properties
+
+If you investigate the dataset difference and find that data or topology doesn't seem to be the root cause of failure, you can also check the pipeline details like pipeline parameter, output or run settings.
+
+**Compare graph** is used to compare pipeline topology, **Compare detail** is used to compare pipeline properties link meta info or settings.
+
+To access the detail comparison, go to the comparison list, select **Compare details** or select **Show compare details** on the pipeline comparison page.
+
+You'll see *Pipeline properties* and *Run properties*.
+
+- Pipeline properties include pipeline parameters, run and output setting, etc.
+- Run properties include job status, submit time and duration, etc.
+
+The following screenshot shows an example of using the detail comparison, where the default compute setting might have been the reason for failure.
+
+:::image type="content" source="./media/how-to-use-pipeline-ui/compute.png" alt-text="Screenshot showing the comparison overview of the default compute." lightbox= "./media/how-to-use-pipeline-ui/compute.png":::
+
+To quickly check the topology comparison, select the pipeline name and select **Compare graph**.
+
+:::image type="content" source="./media/how-to-use-pipeline-ui/compare-graph.png" alt-text="Screenshot of detail comparison with compare graph highlighted." lightbox= "./media/how-to-use-pipeline-ui/compare-graph.png":::
+
+### How to debug your failed node in a pipeline by comparing to similar completed node
+
+If you only updated node properties and changed nothing in the pipeline, then you can debug the node by comparing it with the jobs that are submitted from the same component.
+
+#### Find the job to compare with
+
+1. Find a successful job to compare with by viewing all runs submitted from the same component.
+    1. Right select the failed node and select *View Jobs*. This will give you a list of all the jobs.
+  
+        :::image type="content" source="./media/how-to-use-pipeline-ui/view-jobs.png" alt-text="Screenshot that shows a failed node with view jobs highlighted." lightbox= "./media/how-to-use-pipeline-ui/view-jobs.png":::
+
+    1. Choose a completed job as a comparison target.
+1. After you found a failed and completed job to compare with, add the two jobs to the comparison candidate list.
+    1. For the failed node, right select and select *Add to compare*.
+    1. For the completed job, go to its parent pipeline and located the completed job. Then select *Add to compare*.
+1. Once the two jobs are in the comparison list, select **Compare detail** to show the differences.
+
+### Share the comparison results
+
+To share your comparison results select **Share** and copying the link. For example, you might find out that the dataset difference might of lead to the failure but you aren't a dataset specialist, you can share the comparison result with a data engineer on your team.
+
+:::image type="content" source="./media/how-to-use-pipeline-ui/share.png" alt-text="Screenshot showing the share button and the link you should copy." lightbox= "./media/how-to-use-pipeline-ui/share.png":::
+
+## View profiling to debug pipeline performance issues (preview)
+
+Profiling (preview) can help you debug pipeline performance issues such as hang, long pole etc. Profiling will list the duration information of each step in a pipeline and provide a Gantt chart for visualization.
+
+Profiling enables you to:
+
+- Quickly find which node takes longer time than expected.
+- Identify the time spent of job on each status
+
+To enable this feature:
+
+1. Navigate to Azure Machine Learning studio UI.
+2. Select **Manage preview features** (megaphone icon) among the icons on the top right side of the screen.
+3. In **Managed preview feature** panel, toggle on **View profiling to debug pipeline performance issues** feature.
+
+### How to find the node that runs totally the longest
+
+1. On the Jobs page, select the job name and enter the job detail page.
+1. In the action bar, select **View profiling**. Profiling only works for root level pipeline. It will take a few minutes to load the next page.
+
+    :::image type="content" source="./media/how-to-use-pipeline-ui/view-profiling.png" alt-text="Screenshot showing the pipeline at root level with the view profiling button highlighted." lightbox= "./media/how-to-use-pipeline-ui/view-profiling.png":::
+
+1. After the profiler loads, you'll see a Gantt chart. By Default the critical path of a pipeline is shown. A critical path is a subsequence of steps that determine a pipeline job's total duration.
+
+    :::image type="content" source="./media/how-to-use-pipeline-ui/critical-path.png" alt-text="Screenshot showing the Gantt chart and the critical path." lightbox= "./media/how-to-use-pipeline-ui/critical-path.png":::
+
+1. To find the step that takes the longest, you can either view the Gantt chart or the table below it.
+
+    In the Gantt chart, the length of each bar shows how long the step takes, steps with a longer bar length take more time. You can also filter the table below by "total duration". When you select a row in the table, it will show you the node in the Gantt chart too. When you select a bar on the Gantt chart it will also highlight it in the table.
+
+    In the table, reuse is denoted with the recycling icon.
+
+    If you select the log icon next the node name it will open the detail page, which shows parameter, code, outputs, logs etc.
+
+    :::image type="content" source="./media/how-to-use-pipeline-ui/detail-page-from-log-icon.png" alt-text="Screenshot highlighting the log icon and showing the detail page." lightbox= "./media/how-to-use-pipeline-ui/detail-page-from-log-icon.png":::
+
+    If you're trying to make the queue time shorter for a node, you can change the compute node number and modify job priority to get more compute resources on this one.
+
+### How to find the node that runs the longest in each status
+
+Besides the total duration, you can also sort by durations for each status. For example, you can sort by *Preparing* duration to see which step spends the most time on image building. Then you can open the detail page to find that image building fails because of timeout issue.
+
+#### What do I do if a duration issue identified
+
+Status and definitions:
+
+| Status | What does it mean? | Time estimation | Next step |
+|------|--------------|-------------|----------|
+| Not started | Job is submitted from client side and accepted in Azure ML services. Time spent in this stage is mainly in Azure ML service scheduling and preprocessing. | If there's no backend service issue, this time should be very short.| Open support case via Azure portal. |
+|Preparing | In this status, job is pending for some preparation on job dependencies, for example, environment image building.| If you're using curated or registered custom environment, this time should be very short. | Check image building log. |
+|Inqueue | Job is pending for compute resource allocation. Time spent in this stage is mainly depending on the status of your compute cluster or job yield policy for scope job.| If you're using a cluster with enough compute resource, this time should be short. | Check with workspace admin whether to increase the max nodes of the target compute or change the job to another less busy compute. |
+|Running | Job is executing on remote compute. Time spent in this stage is mainly in two parts: <br> Runtime preparation: image pulling, docker starting and data preparation (mount or download). <br> User script execution. | This status is expected to be most time consuming one.	| 1. Go to the source code check if there's any user error. <br>  2. View the monitoring tab of compute metrics (CPU, memory, networking etc.) to identify the bottleneck. <br> 3. Try online debug with [interactive endpoints](how-to-interactive-jobs.md) if the job is running or locally debug of your code. |
+| Finalizing | Job is in post processing after execution complete. Time spent in this stage is mainly for some post processes like: output uploading, metric/logs uploading and resources clean up.| It will be short for command job. However, might be very long for PRS/MPI job because for a distributed job, the finalizing status is from the first node starting finalizing to the last node done finalizing. | Change your step run output mode from upload to mount if you find unexpected long finalizing time, or open support case via Azure portal. |
+
+
+Along with the profiling, you can also use the *Output + logs* (on the details page), the Common Runtime enabled monitoring metric for PRS/MPI jobs.
+
+### Different view of Gantt chart
+
+- Critical path
+  - You'll see only the step jobs in the pipeline's critical path (jobs that have a dependency).
+  - By default the critical path of the pipeline job is shown.
+- Flatten view
+  - You'll see all step jobs.
+  - In this view, you'll see more nodes than in critical path.
+- Compact view
+  - You'll only see step jobs that are longer than 30 seconds.
+- Structured view.
+  - You'll see all jobs including pipeline component jobs and step jobs.
+
+### Download the duration table
+
+To export the table, select **Export CSV**.
+
 ## Next steps
 
 In this article, you learned the key features in how to create, explore, and debug a pipeline in UI. To learn more about how you can use the pipeline, see the following articles: