You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-factory/solution-template-copy-new-files-last-modified-date.md
+27-19Lines changed: 27 additions & 19 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,72 +27,80 @@ The template defines six parameters:
27
27
-*Directory_Source* is the subfolder path where you can read the files from the source store. You need to replace the default value with your own subfolder path.
28
28
-*FolderPath_Destination* is the folder path where you want to copy files to the destination store. You need to replace the default value with your own folder path.
29
29
-*Directory_Destination* is the subfolder path where you want to copy files to the destination store. You need to replace the default value with your own subfolder path.
30
-
-*LastModified_From* is used to select the files whose LastModifiedDate attribute is after or equal to this datetime value. In order to select the new files only, which has not been copied last time, this datetime value can be the time when the pipeline was triggered last time. You can replace the default value '2019-02-01T00:00:00Z' to your expected LastModifiedDate in UTC timezone.
31
-
-*LastModified_To* is used to select the files whose LastModifiedDate attribute is before this datetime value. In order to select the new files only, which has not been copied last time, this datetime value can be the present time. You can replace the default value '2019-02-01T00:00:00Z' to your expected LastModifiedDate in UTC timezone.
30
+
-*LastModified_From* is used to select the files whose LastModifiedDate attribute is after or equal to this datetime value. In order to select the new files only, which hasn't been copied last time, this datetime value can be the time when the pipeline was triggered last time. You can replace the default value '2019-02-01T00:00:00Z' to your expected LastModifiedDate in UTC timezone.
31
+
-*LastModified_To* is used to select the files whose LastModifiedDate attribute is before this datetime value. In order to select the new files only, which weren't copied in prior runs, this datetime value can be the present time. You can replace the default value '2019-02-01T00:00:00Z' to your expected LastModifiedDate in UTC timezone.
32
32
33
33
## How to use this solution template
34
34
35
-
1. Go to template **Copy new files only by LastModifiedDate**. Create a **New** connection to your destination store. The destination store is where you want to copy files to.
35
+
1. Navigate to the **Template Gallery** from the **Author** tab in Azure Data Factory, then choose the **+** button, **Pipeline**, and finally **Template Gallery**.
36
+
37
+
:::image type="content" source="media/solution-template-copy-new-files-last-modified-date/open-template-gallery.png" alt-text="Screenshot showing how to open the Template gallery from the Azure Data Factory Studio's Author tab.":::
38
+
39
+
1. Search for the template **Copy new files only by LastModifiedDate**, select it, and then select **Continue**.
40
+
41
+
:::image type="content" source="media/solution-template-copy-new-files-last-modified-date/select-last-modified-date-template.png" alt-text="Screenshot showing how to find and select the Copy new files only by LastModifiedDate template.":::
42
+
43
+
1. Create a **New** connection to your destination store. The destination store is where you want to copy files to.
36
44
37
45
:::image type="content" source="media/solution-template-copy-new-files-last-modified-date/copy-new-files-last-modified-date-1.png" alt-text="Create a new connection to the source":::
38
46
39
-
2. Create a **New** connection to your source storage store. The source storage store is where you want to copy files from.
47
+
1. Create a **New** connection to your source storage store. The source storage store is where you want to copy files from.
40
48
41
49
:::image type="content" source="media/solution-template-copy-new-files-last-modified-date/copy-new-files-last-modified-date-3.png" alt-text="Create a new connection to the destination":::
42
50
43
-
3. Select **Use this template**.
51
+
1. Select **Use this template**.
44
52
45
53
:::image type="content" source="media/solution-template-copy-new-files-last-modified-date/copy-new-files-last-modified-date-4.png" alt-text="Use this template":::
46
54
47
-
4. You will see the pipeline available in the panel, as shown in the following example:
55
+
1. You see the pipeline available in the panel, as shown in the following example:
48
56
49
57
:::image type="content" source="media/solution-template-copy-new-files-last-modified-date/copy-new-files-last-modified-date-5.png" alt-text="Show the pipeline":::
50
58
51
-
5. Select **Debug**, write the value for the **Parameters** and select **Finish**. In the picture below, we set the parameters as following.
59
+
1. Select **Debug**, write the value for the **Parameters**, and select **Finish**. In the picture that follows, we set the parameters as following.
52
60
-**FolderPath_Source** = sourcefolder
53
61
-**Directory_Source** = subfolder
54
62
-**FolderPath_Destination** = destinationfolder
55
63
-**Directory_Destination** = subfolder
56
64
-**LastModified_From** = 2019-02-01T00:00:00Z
57
65
-**LastModified_To** = 2019-03-01T00:00:00Z
58
66
59
-
The example is indicating that the files, which have been last modified within the timespan (**2019-02-01T00:00:00Z** to **2019-03-01T00:00:00Z**) will be copied from the source path **sourcefolder/subfolder** to the destination path **destinationfolder/subfolder**. You can replace these with your own parameters.
67
+
The example is indicating that the files, which were last modified within the timespan (**2019-02-01T00:00:00Z** to **2019-03-01T00:00:00Z**) will be copied from the source path **sourcefolder/subfolder** to the destination path **destinationfolder/subfolder**. You can replace these times or folders with your own parameters.
60
68
61
69
:::image type="content" source="media/solution-template-copy-new-files-last-modified-date/copy-new-files-last-modified-date-6.png" alt-text="Run the pipeline":::
62
70
63
-
6. Review the result. You will see only the files last modified within the configured timespan has been copied to the destination store.
71
+
1. Review the result. You see only the files last modified within the configured timespan are copied to the destination store.
64
72
65
73
:::image type="content" source="media/solution-template-copy-new-files-last-modified-date/copy-new-files-last-modified-date-7.png" alt-text="Review the result":::
66
74
67
-
7. Now you can add a tumbling windows trigger to automate this pipeline, so that the pipeline can always copy new and changed files only by LastModifiedDate periodically. Select **Add trigger**, and select **New/Edit**.
75
+
1. Now you can add a tumbling windows trigger to automate this pipeline, so that the pipeline can always copy new and changed files only by LastModifiedDate periodically. Select **Add trigger**, and select **New/Edit**.
68
76
69
77
:::image type="content" source="media/solution-template-copy-new-files-last-modified-date/copy-new-files-last-modified-date-8.png" alt-text="Screenshot that highlights the New/Edit menu option that appears when you select Add trigger.":::
70
78
71
-
8. In the **Add Triggers** window, select **+ New**.
79
+
1. In the **Add Triggers** window, select **+ New**.
72
80
73
-
9. Select **Tumbling Window** for the trigger type, set **Every 15 minute(s)** as the recurrence (you can change to any interval time). Select **Yes** for Activated box, and then select **OK**.
81
+
1. Select **Tumbling Window** for the trigger type, set **Every 15 minute(s)** as the recurrence (you can change to any interval time). Select **Yes** for Activated box, and then select **OK**.
10. Set the value for the **Trigger Run Parameters** as following, and select **Finish**.
85
+
1. Set the value for the **Trigger Run Parameters** as following, and select **Finish**.
78
86
-**FolderPath_Source** = **sourcefolder**. You can replace with your folder in source data store.
79
87
-**Directory_Source** = **subfolder**. You can replace with your subfolder in source data store.
80
88
-**FolderPath_Destination** = **destinationfolder**. You can replace with your folder in destination data store.
81
89
-**Directory_Destination** = **subfolder**. You can replace with your subfolder in destination data store.
82
-
-**LastModified_From** = **\@trigger().outputs.windowStartTime**. It is a system variable from the trigger determining the time when the pipeline was triggered last time.
83
-
-**LastModified_To** = **\@trigger().outputs.windowEndTime**. It is a system variable from the trigger determining the time when the pipeline is triggered this time.
90
+
-**LastModified_From** = **\@trigger().outputs.windowStartTime**. It's a system variable from the trigger determining the time when the pipeline was triggered last time.
91
+
-**LastModified_To** = **\@trigger().outputs.windowEndTime**. It's a system variable from the trigger determining the time when the pipeline is triggered this time.
12. Create new files in your source folder of data source store. You are now waiting for the pipeline to be triggered automatically and only the new files will be copied to the destination store.
99
+
1. Create new files in your source folder of data source store. You're now waiting for the pipeline to be triggered automatically and only the new files are copied to the destination store.
92
100
93
-
13. Select **Monitor** tab in the left navigation panel, and wait for about 15 minutes if the recurrence of trigger has been set to every 15 minutes.
101
+
1. Select **Monitor** tab in the left navigation panel, and wait for about 15 minutes if the recurrence of trigger was set to every 15 minutes.
94
102
95
-
14. Review the result. You will see your pipeline will be triggered automatically every 15 minutes, and only the new or changed files from source store will be copied to the destination store in each pipeline run.
103
+
1. Review the result. You see your pipeline is triggered automatically every 15 minutes, and only the new or changed files from source store are copied to the destination store in each pipeline run.
96
104
97
105
:::image type="content" source="media/solution-template-copy-new-files-last-modified-date/copy-new-files-last-modified-date-15.png" alt-text="Screenshot that shows the results that return when the pipeline is triggered.":::
0 commit comments