You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-factory/how-to-create-event-trigger.md
+28-16Lines changed: 28 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,7 @@ manager: craigg
16
16
17
17
This article describes the event-based triggers that you can create in your Data Factory pipelines.
18
18
19
-
Event-driven architecture (EDA) is a common data integration pattern that involves production, detection, consumption, and reaction to events. Data integration scenarios often require Data Factory customers to trigger pipelines based on events. Data Factory is now integrated with [Azure Event Grid](https://azure.microsoft.com/services/event-grid/), which lets you trigger pipelines on an event.
19
+
Event-driven architecture (EDA) is a common data integration pattern that involves production, detection, consumption, and reaction to events. Data integration scenarios often require Data Factory customers to trigger pipelines based on events such as the arrival or deletion of a file in your Azure Storage account. Data Factory is now integrated with [Azure Event Grid](https://azure.microsoft.com/services/event-grid/), which lets you trigger pipelines on an event.
20
20
21
21
For a ten-minute introduction and demonstration of this feature, watch the following video:
22
22
@@ -28,34 +28,46 @@ For a ten-minute introduction and demonstration of this feature, watch the follo
28
28
29
29
## Data Factory UI
30
30
31
-
### Create a new event trigger
31
+
This section shows you how to create an event trigger within the Azure Data Factory User Interface.
32
32
33
-
A typical event is the arrival of a file, or the deletion of a file, in your Azure Storage account. You can create a trigger that responds to this event in your Data Factory pipeline.
33
+
1. Go to the **Authoring Canvas**
34
34
35
-
> [!NOTE]
36
-
> This integration supports only version 2 Storage accounts (General purpose).
35
+
2. In the bottom-left corner, click on the **Triggers** button
36
+
37
+
3. Click **+ New** which will open up the create trigger side nav
38
+
39
+
4. Select trigger type **Event**
37
40
38
41

39
42
40
-
### Configure the event trigger
43
+
5. Select your storage account from the Azure subscription dropdown or manually using its Storage account resource ID. Choose which container you wish the events to occur on. Container selection is optional, but be mindful that selecting all containers can lead to a large number of events.
44
+
45
+
> [!NOTE]
46
+
> The Event Trigger currently supports only version 2 Storage accounts (General purpose).
47
+
48
+
> [!NOTE]
49
+
> Due to an Azure Event Grid limitation, Azure Data Factory only supports a maximum of 500 event triggers per storage account.
50
+
51
+
6. The **Blob path begins with** and **Blob path ends with** properties allow you to specify the containers, folders, and blob names for which you want to receive events. Your event trigger requires at least one of these properties to be defined. You can use variety of patterns for both **Blob path begins with** and **Blob path ends with** properties, as shown in the examples later in this article.
41
52
42
-
With the **Blob path begins with** and **Blob path ends with** properties, you can specify the containers, folders, and blob names for which you want to receive events. You can use variety of patterns for both **Blob path begins with** and **Blob path ends with** properties, as shown in the examples later in this article. At least one of these properties is required.
53
+
***Blob path begins with:** The blob path must start with a folder path. Valid values include `2018/` and `2018/april/shoes.csv`. This field can't be selected if a container isn't selected.
54
+
***Blob path ends with:** The blob path must end with a file name or extension. Valid values include `shoes.csv` and `.csv`. Container and folder name are optional but, when specified, they must be separated by a `/blobs/` segment. For example, a container named 'orders' can have a value of `/orders/blobs/2018/april/shoes.csv`. To specify a folder in any container, omit the leading '/' character. For example, `april/shoes.csv` will trigger an event on any file named `shoes.csv` in folder a called 'april' in any container.
43
55
44
-

56
+
7. Select whether your trigger will respond to a **Blob created**event, **Blob deleted**event, or both. In your specified storage location, each event will trigger the Data Factory pipelines associated with the trigger.
45
57
46
-
### Select the event trigger type
58
+

47
59
48
-
As soon as the file arrives in your storage location and the corresponding blob is created, this event triggers and runs your Data Factory pipeline. You can create a trigger that responds to a blob creation event, a blob deletion event, or both events, in your Data Factory pipelines.
60
+
8. Once you've configured you trigger, click on **Next: Data preview**. This screen shows the existing blobs matched by your event trigger configuration. Make sure you've specific filters. Configuring filters that are too broad can match a large number of files created/deleted and may significantly impact your cost. Once your filter conditions have been verified, click **Finish**.
49
61
50
-

62
+

51
63
52
-
### Map trigger properties to pipeline parameters
64
+
9. To attach a pipeline to this trigger, go to the pipeline canvas and click **Add trigger** and select **New/Edit**. When the side nav appears, click on the **Choose trigger...** dropdown and select the trigger you created. Click **Next: Data preview** to confirm the configuration is correct and then **Next** to validate the Data preview is correct.
53
65
54
-
When an event trigger fires for a specific blob, the event captures the folder path and file name of the blob into the properties `@triggerBody().folderPath` and `@triggerBody().fileName`. To use the values of these properties in a pipeline, you must map the properties to pipeline parameters. After mapping the properties to parameters, you can access the values captured by the trigger through the `@pipeline().parameters.parameterName` expression throughout the pipeline.
66
+
10. If your pipeline has parameters, you can specify them on the trigger runs parameter side nav. The event trigger captures the folder path and file name of the blob into the properties `@triggerBody().folderPath` and `@triggerBody().fileName`. To use the values of these properties in a pipeline, you must map the properties to pipeline parameters. After mapping the properties to parameters, you can access the values captured by the trigger through the `@pipeline().parameters.parameterName` expression throughout the pipeline. Click **Finish** once you are done.
55
67
56
-

68
+

57
69
58
-
For example, in the preceding screenshot. the trigger is configured to fire when a blob path ending in `.csv` is created in the Storage Account. As a result, when a blob with the `.csv` extension is created anywhere in the Storage Account, the `folderPath` and `fileName` properties capture the location of the new blob. For example, `@triggerBody().folderPath` has a value like `/containername/foldername/nestedfoldername` and `@triggerBody().fileName` has a value like `filename.csv`. These values are mapped in the example to the pipeline parameters `sourceFolder` and `sourceFile`. You can use them throughout the pipeline as `@pipeline().parameters.sourceFolder` and `@pipeline().parameters.sourceFile` respectively.
70
+
In the preceding example, the trigger is configured to fire when a blob path ending in .csv is created in the folder event-testing in the container sample-data. The **folderPath** and **fileName** properties capture the location of the new blob. For example, when MoviesDB.csv is added to the path sample-data/event-testing, `@triggerBody().folderPath` has a value of `sample-data/event-testing` and `@triggerBody().fileName` has a value of `moviesDB.csv`. These values are mapped in the example to the pipeline parameters `sourceFolder` and `sourceFile` which can be used throughout the pipeline as `@pipeline().parameters.sourceFolder` and `@pipeline().parameters.sourceFile` respectively.
59
71
60
72
## JSON schema
61
73
@@ -73,7 +85,7 @@ The following table provides an overview of the schema elements that are related
73
85
This section provides examples of event-based trigger settings.
74
86
75
87
> [!IMPORTANT]
76
-
> You have to include the `/blobs/` segment of the path, as shown in the following examples, whenever you specify container and folder, container and file, or container, folder, and file.
88
+
> You have to include the `/blobs/` segment of the path, as shown in the following examples, whenever you specify container and folder, container and file, or container, folder, and file. For **blobPathBeginsWith**, the Data Factory UI will automatically add `/blobs/` between the folder and container name in the trigger JSON.
0 commit comments