You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/machine-learning/how-to-debug-pipeline-failure.md
+10-10Lines changed: 10 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,11 +22,11 @@ After you submit a pipeline job, you can select a link to the job in your worksp
22
22
23
23
## Use outline to quickly find a node
24
24
25
-
On the pipeline job detail page, the **Outline** pane on the left shows the overall structure of your pipeline job. Hover on any row and select the **Locate in canvas** icon to highlight that node on the canvas and open an information pane for the node on the right.
25
+
On the pipeline job detail page, the **Outline** pane at left shows the overall structure of your pipeline job. Hover on any row and select the **Locate in canvas** icon to highlight that node on the canvas and open an information pane for the node on the right.
26
26
27
27
:::image type="content" source="./media/how-to-debug-pipeline-failure/outline-detail.png" alt-text="Screenshot showing outline and locate in the canvas." lightbox= "./media/how-to-debug-pipeline-failure/outline.png":::
28
28
29
-
In the **Outline** pane, you can select the **Filter** icon to quickly filter the view to **Completed nodes only**, **Component only**, or **Dataset only**. You can also filter the list by entering node names or component names in the Search box, or by selecting **Add filter** and choosing from a list of filters.
29
+
In the **Outline** pane, you can select the **Filter** icon to quickly filter the view to **Completed nodes only**, **Component only**, or **Data only**. You can also filter the list by entering node names or component names in the Search box, or by selecting **Add filter** and choosing from a list of filters.
30
30
31
31
:::image type="content" source="./media/how-to-debug-pipeline-failure/quick-filter-detail.png" alt-text="Screenshot showing quick filter and search in the Outline pane." lightbox= "./media/how-to-debug-pipeline-failure/quick-filter.png":::
32
32
@@ -40,9 +40,9 @@ If your pipeline fails or gets stuck on a node, first view the logs.
40
40
41
41

42
42
43
-
1. Select the node to open the information pane on the right.
43
+
1. Select the node to open the information pane at right.
44
44
45
-
1. Select **Outputs + logs** tab to view all the outputs and logs of this node.
45
+
1. Select the **Outputs + logs** tab to view all the outputs and logs from this node.
46
46
47
47
:::image type="content" source="./media/how-to-debug-pipeline-failure/log-detail.png" alt-text="Screenshot of the user_logs in the node information pane." lightbox= "./media/how-to-debug-pipeline-failure/log-detail.png":::
48
48
@@ -51,13 +51,13 @@ If your pipeline fails or gets stuck on a node, first view the logs.
51
51
- The *system_logs* folder contains logs generated by Azure Machine Learning. To learn more, see [View and download diagnostic logs](how-to-log-view-metrics.md#view-and-download-diagnostic-logs).
52
52
53
53
> [!NOTE]
54
-
> If you don't see those folders, the compute run time update might not be released to the compute cluster yet, and you can look at *70_driver_log.txt* in the *azureml-logs* folder first.
54
+
> If you don't see those folders, the compute run time update might not be released to the compute cluster yet. You can look at *70_driver_log.txt* in the *azureml-logs* folder first.
55
55
56
56
## Compare pipeline jobs (preview)
57
57
58
58
You can compare different pipeline jobs to debug failure or other unexpected issues (preview). Pipeline comparison identifies the differences, such as topology, component properties, and job properties, between pipeline jobs.
59
59
60
-
For example, you can compare successful and failed pipeline jobs to find differences that might have made one pipeline job fail. You can debug a failed pipeline job by comparing it to a completed job, or debug a failed node in a pipeline by comparing it to a similar completed node.
60
+
You can compare successful and failed pipeline jobs to find differences that might make one pipeline job fail. You can debug a failed pipeline job by comparing it to a completed job, or debug a failed node in a pipeline by comparing it to a similar completed node.
61
61
62
62
To enable this feature in Azure Machine Learning studio, select the megaphone icon at top right to manage preview features. In the **Managed preview feature** panel, make sure **Compare pipeline jobs to debug failures or unexpected issues** is set to **Enabled**.
63
63
@@ -67,14 +67,14 @@ To enable this feature in Azure Machine Learning studio, select the megaphone ic
67
67
68
68
During iterative model development, you might clone and modify a successful baseline pipeline by changing a parameter, dataset, compute resource, or other setting. If the new pipeline fails, you can use pipeline comparison to help figure out the failure by identifying the changes from the parent pipeline.
69
69
70
-
For example, if you get an error message that your new pipeline failed due to an out-of-memory issue, you can use pipeline comparison to see what changes from the parent pipeline might cause memory issues.
70
+
For example, if your new pipeline failed due to an out-of-memory issue, you can use pipeline comparison to see what changes from the parent pipeline might cause memory issues.
71
71
72
72
#### Compare the pipeline with its parent
73
73
74
74
1. On the failed pipeline job page, select **Show lineage**.
75
75
1. Select the link in the **Cloned from** popup to open the parent pipeline job page in a new browser tab.
76
76
77
-
:::image type="content" source="./media/how-to-debug-pipeline-failure/cloned-from.png" alt-text="Screenshot showing the cloned from link, with the previous step, the lineage button highlighted." lightbox= "./media/how-to-debug-pipeline-failure/cloned-from.png":::
77
+
:::image type="content" source="./media/how-to-debug-pipeline-failure/draft-show-lineage.png" alt-text="Screenshot showing the cloned from link, with the previous step, the lineage button highlighted." lightbox= "./media/how-to-debug-pipeline-failure/draft-show-lineage.png":::
78
78
79
79
1. On both pages, select **Add to compare** on the top menu bar to add both jobs to the **Compare** list.
80
80
@@ -86,7 +86,7 @@ Once you add both pipelines to the comparison list, you can select **Compare det
86
86
87
87
**Compare graph** shows the graph topology changes between pipelines **A** and **B**. On the canvas, nodes specific to pipeline A are marked **A** and highlighted in red, and nodes specific to pipeline B are marked **B** and highlighted in green. A description of changes appears at the tops of nodes that have differences.
88
88
89
-
You can select any node to open a **Component information** pane, where you can see **Dataset properties** or **Component properties** like **parameters**, **runSettings**, and **outputSettings**. You can choose to **Show only differences** and to **See differences inline**.
89
+
You can select any node to open a **Component information** pane, where you can see **Dataset properties** or **Component properties** like **parameters**, **runSettings**, and **outputSettings**. You can choose to **Show only differences** and to **Show differences inline**.
90
90
91
91
:::image type="content" source="./media/how-to-debug-pipeline-failure/parameter-changed.png" alt-text="Screenshot showing the parameter changed and the component information tab." lightbox= "./media/how-to-debug-pipeline-failure/parameter-changed.png":::
92
92
@@ -128,6 +128,6 @@ To share debug results with your teammates or other stakeholders, select **Share
0 commit comments