|
1 | 1 | --- |
2 | 2 | title: Iterative development and debugging in Azure Data Factory |
3 | | -description: Learn how to develop and debug Data Factory pipelines iteratively in the Azure portal. |
4 | | -ms.date: 09/26/2018 |
| 3 | +description: Learn how to develop and debug Data Factory pipelines iteratively in the ADF UX |
| 4 | +ms.date: 08/28/2020 |
5 | 5 | ms.topic: conceptual |
6 | 6 | ms.service: data-factory |
7 | 7 | services: data-factory |
8 | 8 | documentationcenter: '' |
9 | 9 | ms.workload: data-services |
10 | | - |
11 | 10 | author: djpmsft |
12 | 11 | ms.author: daperlov |
13 | | -manager: anandsub |
| 12 | + |
14 | 13 | --- |
15 | 14 | # Iterative development and debugging with Azure Data Factory |
16 | 15 | [!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)] |
17 | 16 |
|
18 | | -Azure Data Factory lets you iteratively develop and debug Data Factory pipelines. |
| 17 | +Azure Data Factory lets you iteratively develop and debug Data Factory pipelines as you are developing your data integration solutions. These features allow you to test your changes before creating a pull request or publishing them to the data factory service. |
19 | 18 |
|
20 | 19 | For an eight-minute introduction and demonstration of this feature, watch the following video: |
21 | 20 |
|
22 | 21 | > [!VIDEO https://channel9.msdn.com/Shows/Azure-Friday/Iterative-development-and-debugging-with-Azure-Data-Factory/player] |
23 | 22 |
|
24 | | -## Iterative debugging features |
25 | | -Create pipelines and do test runs using the **Debug** capability in the pipeline canvas without writing a single line of code. |
| 23 | +## Debugging a pipeline |
26 | 24 |
|
27 | | - |
| 25 | +As you author using the pipeline canvas, you can test your activities using the **Debug** capability. When you do test runs, you don't have to publish your changes to the data factory before you select **Debug**. This feature is helpful in scenarios where you want to make sure that the changes work as expected before you update the data factory workflow. |
28 | 26 |
|
29 | | -View the results of your test runs in the **Output** window of the pipeline canvas. |
| 27 | + |
30 | 28 |
|
31 | | - |
| 29 | +As the pipeline is running, you can see the results of each activity in the **Output** tab of the pipeline canvas. |
32 | 30 |
|
33 | | -After a test run succeeds, add more activities to your pipeline and continue debugging in an iterative manner. You can also **Cancel** a test run while it is in progress. |
| 31 | +View the results of your test runs in the **Output** window of the pipeline canvas. |
34 | 32 |
|
35 | | - |
| 33 | + |
36 | 34 |
|
37 | | -When you do test runs, you don't have to publish your changes to the data factory before you select **Debug**. This feature is helpful in scenarios where you want to make sure that the changes work as expected before you update the data factory workflow. |
| 35 | +After a test run succeeds, add more activities to your pipeline and continue debugging in an iterative manner. You can also **Cancel** a test run while it is in progress. |
38 | 36 |
|
39 | 37 | > [!IMPORTANT] |
40 | | -> Selecting **Debug** actually runs the pipeline. So, for example, if the pipeline contains copy activity, the test run copies data from source to destination. As a result, we recommend that you use test folders in your copy activities and other activities when debugging. After you've debugged the pipeline, switch to the actual folders that you want to use in normal operations. |
| 38 | +> Selecting **Debug** actually runs the pipeline. For example, if the pipeline contains copy activity, the test run copies data from source to destination. As a result, we recommend that you use test folders in your copy activities and other activities when debugging. After you've debugged the pipeline, switch to the actual folders that you want to use in normal operations. |
| 39 | +
|
| 40 | +### Setting breakpoints |
41 | 41 |
|
42 | | -## Visualizing debug runs |
| 42 | +Azure Data Factory allows for you to debug a pipeline until you reach a particular activity on the pipeline canvas. Put a breakpoint on the activity until which you want to test, and select **Debug**. Data Factory ensures that the test runs only until the breakpoint activity on the pipeline canvas. This *Debug Until* feature is useful when you don't want to test the entire pipeline, but only a subset of activities inside the pipeline. |
43 | 43 |
|
44 | | -You can visualize all the debug runs that are in progress for your data factory in one place. Select **View debug runs** in the upper right corner of the page. This feature is useful in scenarios where you have master pipelines kicking off debug runs for child pipelines, and you want a single view to see all the active debug runs. |
| 44 | + |
45 | 45 |
|
46 | | - |
| 46 | +To set a breakpoint, select an element on the pipeline canvas. A *Debug Until* option appears as an empty red circle at the upper right corner of the element. |
47 | 47 |
|
48 | | - |
| 48 | + |
49 | 49 |
|
50 | | -If you have active Data Flow debug sessions, those sessions will appear on the bottom part of the active debug window. You can select an active data flow session and stop the respective cluster. |
| 50 | +After you select the *Debug Until* option, it changes to a filled red circle to indicate the breakpoint is enabled. |
51 | 51 |
|
52 | | - |
| 52 | + |
53 | 53 |
|
54 | 54 | ## Monitoring debug runs |
55 | 55 |
|
56 | | -The test runs initiated with the **Debug** capability are not available in the list on the **Monitor** tab. You can only see runs triggered with **Trigger Now**, **Schedule**, or **Tumbling Window** triggers in the **Monitor** tab. You can see the last test run initiated with the **Debug** capability in the **Output** window of the pipeline canvas. |
| 56 | +When you run a pipeline debug run, the results will appear in the **Output** window of the pipeline canvas. The output tab will only contain the most recent run that occurred during the current browser session. |
57 | 57 |
|
58 | | -## Setting breakpoints for debugging |
| 58 | + |
59 | 59 |
|
60 | | -Data Factory also lets you debug until you reach a particular activity on the pipeline canvas. Just put a breakpoint on the activity until which you want to test, and select **Debug**. Data Factory ensures that the test runs only until the breakpoint activity on the pipeline canvas. This *Debug Until* feature is useful when you don't want to test the entire pipeline, but only a subset of activities inside the pipeline. |
| 60 | +To view a historical view of debug runs or see a list of all active debug runs, you can go into the **Monitor** experience. |
61 | 61 |
|
62 | | - |
| 62 | + |
63 | 63 |
|
64 | | -To set a breakpoint, select an element on the pipeline canvas. A *Debug Until* option appears as an empty red circle at the upper right corner of the element. |
| 64 | +> [!NOTE] |
| 65 | +> The Azure Data Factory service only persists debug run history for 15 days. |
65 | 66 |
|
66 | | - |
| 67 | +## Debugging mapping data flows |
67 | 68 |
|
68 | | -After you select the *Debug Until* option, it changes to a filled red circle to indicate the breakpoint is enabled. |
| 69 | +Mapping data flows allow you to build code-free data transformation logic that runs at scale. When building your logic, you can turn on a debug session to interactively work with your data using a live Spark cluster. To learn more, read about [mapping data flow debug mode](concepts-data-flow-debug-mode.md). |
| 70 | + |
| 71 | +You can monitor active data flow debug sessions across a factory in the **Monitor** experience. |
| 72 | + |
| 73 | + |
| 74 | + |
| 75 | +### Debugging a pipeline with a data flow activity |
69 | 76 |
|
70 | | - |
| 77 | +When running a debug run with a data flow, you have two options on which compute to use. You can either use an existing debug cluster or spin up a new just-in-time cluster for your data flows. |
| 78 | + |
| 79 | +Using an existing debug session will greatly reduce the data flow start up time as the cluster is already running, but is not recommended for complex or parallel workloads as it may fail when multiple jobs run at once. |
| 80 | + |
| 81 | +Using the activity runtime will create a new cluster using the settings specified in each data flow activity's integration runtime. This allows each job to be isolated and should be used for complex workloads or performance testing. |
| 82 | + |
| 83 | + |
71 | 84 |
|
72 | 85 | ## Next steps |
73 | | -[Continuous integration and deployment in Azure Data Factory](continuous-integration-deployment.md) |
| 86 | + |
| 87 | +After testing your changes, promote them to higher environments using [continuous integration and deployment in Azure Data Factory](continuous-integration-deployment.md). |
0 commit comments