Skip to content

Commit 3691b44

Browse files
ShawnJacksonShawnJackson
authored
Merge pull request #279955 from paulth1/data-factory-batch3
[AQ] edit pass: Data factory batch3
2 parents 113624d + 6c108cb commit 3691b44

File tree

3 files changed

+155
-157
lines changed

3 files changed

+155
-157
lines changed

‎articles/data-factory/how-to-create-custom-event-trigger.md

Lines changed: 58 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Create custom event triggers in Azure Data Factory
3-
description: Learn how to create a trigger in Azure Data Factory that runs a pipeline in response to a custom event published to Event Grid.
3+
description: Learn how to create a trigger in Azure Data Factory that runs a pipeline in response to a custom event published to Azure Event Grid.
44
ms.service: data-factory
55
ms.subservice: orchestration
66
author: kromerm
@@ -14,31 +14,28 @@ ms.date: 01/05/2024
1414

1515
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
1616

17-
Event-driven architecture (EDA) is a common data integration pattern that involves production, detection, consumption, and reaction to events. Data integration scenarios often require Azure Data Factory customers to trigger pipelines when certain events occur. Data Factory native integration with [Azure Event Grid](https://azure.microsoft.com/services/event-grid/) now covers [custom topics](../event-grid/custom-topics.md). You send events to an event grid topic. Data Factory subscribes to the topic, listens, and then triggers pipelines accordingly.
18-
19-
> [!NOTE]
20-
> The integration described in this article depends on [Azure Event Grid](https://azure.microsoft.com/services/event-grid/). Make sure that your subscription is registered with the Event Grid resource provider. For more information, see [Resource providers and types](../azure-resource-manager/management/resource-providers-and-types.md#azure-portal). You must be able to do the `Microsoft.EventGrid/eventSubscriptions/` action. This action is part of the [EventGrid EventSubscription Contributor](../role-based-access-control/built-in-roles.md#eventgrid-eventsubscription-contributor) built-in role.
17+
Event-driven architecture is a common data integration pattern that involves production, detection, consumption, and reaction to events. Data integration scenarios often require Azure Data Factory customers to trigger pipelines when certain events occur. Data Factory native integration with [Azure Event Grid](https://azure.microsoft.com/services/event-grid/) now covers [custom topics](../event-grid/custom-topics.md). You send events to an Event Grid topic. Data Factory subscribes to the topic, listens, and then triggers pipelines accordingly.
2118

19+
The integration described in this article depends on [Azure Event Grid](https://azure.microsoft.com/services/event-grid/). Make sure that your subscription is registered with the Event Grid resource provider. For more information, see [Resource providers and types](../azure-resource-manager/management/resource-providers-and-types.md#azure-portal). You must be able to do the `Microsoft.EventGrid/eventSubscriptions/` action. This action is part of the [EventGrid EventSubscription Contributor](../role-based-access-control/built-in-roles.md#eventgrid-eventsubscription-contributor) built-in role.
2220

2321
> [!IMPORTANT]
24-
> If you are using this feature in Azure Synapse Analytics, please ensure that your subscription is also registered with Data Factory resource provider, or otherwise you will get an error stating that _the creation of an "Event Subscription" failed_.
25-
22+
> If you're using this feature in Azure Synapse Analytics, ensure that your subscription is also registered with a Data Factory resource provider. Otherwise, you get a message stating that "the creation of an Event Subscription failed."
2623
27-
If you combine pipeline parameters and a custom event trigger, you can parse and reference custom `data` payloads in pipeline runs. Because the `data` field in a custom event payload is a free-form, JSON key-value structure, you can control event-driven pipeline runs.
24+
If you combine pipeline parameters and a custom event trigger, you can parse and reference custom `data` payloads in pipeline runs. Because the `data` field in a custom event payload is a freeform, JSON key-value structure, you can control event-driven pipeline runs.
2825

2926
> [!IMPORTANT]
30-
> If a key referenced in parameterization is missing in the custom event payload, `trigger run` will fail. You'll get an error that states the expression cannot be evaluated because property `keyName` doesn't exist. In this case, **no** `pipeline run` will be triggered by the event.
27+
> If a key referenced in parameterization is missing in the custom event payload, `trigger run` fails. You get a message that states the expression can't be evaluated because the `keyName` property doesn't exist. In this case, **no** `pipeline run` is triggered by the event.
3128
3229
## Set up a custom topic in Event Grid
3330

3431
To use the custom event trigger in Data Factory, you need to *first* set up a [custom topic in Event Grid](../event-grid/custom-topics.md).
3532

36-
Go to Azure Event Grid and create the topic yourself. For more information on how to create the custom topic, see Azure Event Grid [portal tutorials](../event-grid/custom-topics.md#azure-portal-tutorials) and [CLI tutorials](../event-grid/custom-topics.md#azure-cli-tutorials).
33+
Go to Event Grid and create the topic yourself. For more information on how to create the custom topic, see Event Grid [portal tutorials](../event-grid/custom-topics.md#azure-portal-tutorials) and [Azure CLI tutorials](../event-grid/custom-topics.md#azure-cli-tutorials).
3734

3835
> [!NOTE]
39-
> The workflow is different from Storage Event Trigger. Here, Data Factory doesn't set up the topic for you.
36+
> The workflow is different from a storage event trigger. Here, Data Factory doesn't set up the topic for you.
4037
41-
Data Factory expects events to follow the [Event Grid event schema](../event-grid/event-schema.md). Make sure event payloads have the following fields:
38+
Data Factory expects events to follow the [Event Grid event schema](../event-grid/event-schema.md). Make sure that event payloads have the following fields:
4239

4340
```json
4441
[
@@ -59,101 +56,103 @@ Data Factory expects events to follow the [Event Grid event schema](../event-gri
5956

6057
## Use Data Factory to create a custom event trigger
6158

62-
1. Go to Azure Data Factory and sign in.
59+
1. Go to Data Factory and sign in.
6360

6461
1. Switch to the **Edit** tab. Look for the pencil icon.
6562

6663
1. Select **Trigger** on the menu and then select **New/Edit**.
6764

68-
1. On the **Add Triggers** page, select **Choose trigger**, and then select **+New**.
65+
1. On the **Add Triggers** page, select **Choose trigger**, and then select **+ New**.
6966

70-
1. Select **Custom events** for **Type**.
67+
1. Under **Type**, select **Custom events**.
7168

72-
:::image type="content" source="media/how-to-create-custom-event-trigger/custom-event-1-creation.png" alt-text="Screenshot of Author page to create a new custom event trigger in Data Factory UI." lightbox="media/how-to-create-custom-event-trigger/custom-event-1-creation-expanded.png":::
69+
:::image type="content" source="media/how-to-create-custom-event-trigger/custom-event-1-creation.png" alt-text="Screenshot that shows creating a new custom event trigger in the Data Factory UI." lightbox="media/how-to-create-custom-event-trigger/custom-event-1-creation-expanded.png":::
7370

74-
1. Select your custom topic from the Azure subscription dropdown or manually enter the event topic scope.
71+
1. Select your custom topic from the Azure subscription dropdown list or manually enter the event topic scope.
7572

7673
> [!NOTE]
77-
> To create or modify a custom event trigger in Data Factory, you need to use an Azure account with appropriate role-based access control (Azure RBAC). No additional permission is required. The Data Factory service principal does *not* require special permission to your Event Grid. For more information about access control, see the [Role-based access control](#role-based-access-control) section.
74+
> To create or modify a custom event trigger in Data Factory, you need to use an Azure account with appropriate Azure role-based access control (Azure RBAC). No other permission is required. The Data Factory service principal does *not* require special permission to your Event Grid. For more information about access control, see the [Role-based access control](#role-based-access-control) section.
7875
79-
1. The **Subject begins with** and **Subject ends with** properties allow you to filter for trigger events. Both properties are optional.
76+
1. The `Subject begins with` and `Subject ends with` properties allow you to filter for trigger events. Both properties are optional.
8077

81-
1. Use **+ New** to add **Event Types** to filter on. The list of custom event triggers uses an OR relationship. When a custom event with an `eventType` property that matches one on the list, a pipeline run is triggered. The event type is case insensitive. For example, in the following screenshot, the trigger matches all `copycompleted` or `copysucceeded` events that have a subject that begins with *factories*.
78+
1. Use **+ New** to add **Event types** to filter on. The list of custom event triggers uses an OR relationship. When a custom event with an `eventType` property matches one on the list, a pipeline run is triggered. The event type is case insensitive. For example, in the following screenshot, the trigger matches all `copycompleted` or `copysucceeded` events that have a subject that begins with *factories*.
8279

83-
:::image type="content" source="media/how-to-create-custom-event-trigger/custom-event-2-properties.png" alt-text="Screenshot of Edit Trigger page to explain Event Types and Subject filtering in Data Factory UI.":::
80+
:::image type="content" source="media/how-to-create-custom-event-trigger/custom-event-2-properties.png" alt-text="Screenshot that shows the Edit trigger page to explain Event types and Subject filtering in the Data Factory UI.":::
81+
82+
1. A custom event trigger can parse and send a custom `data` payload to your pipeline. You create the pipeline parameters and then fill in the values on the **Parameters** page. Use the format `@triggerBody().event.data._keyName_` to parse the data payload and pass values to the pipeline parameters.
83+
84+
For a detailed explanation, see:
8485

85-
1. A custom event trigger can parse and send a custom `data` payload to your pipeline. You create the pipeline parameters, and then fill in the values on the **Parameters** page. Use the format `@triggerBody().event.data._keyName_` to parse the data payload and pass values to the pipeline parameters.
86-
87-
For a detailed explanation, see the following articles:
8886
- [Reference trigger metadata in pipelines](how-to-use-trigger-parameterization.md)
8987
- [System variables in custom event trigger](control-flow-system-variables.md#custom-event-trigger-scope)
9088

91-
:::image type="content" source="media/how-to-create-custom-event-trigger/custom-event-4-trigger-values.png" alt-text="Screenshot of pipeline parameters settings.":::
89+
:::image type="content" source="media/how-to-create-custom-event-trigger/custom-event-4-trigger-values.png" alt-text="Screenshot that shows pipeline parameters settings.":::
9290

93-
:::image type="content" source="media/how-to-create-custom-event-trigger/custom-event-3-parameters.png" alt-text="Screenshot of the parameters page to reference data payload in custom event.":::
91+
:::image type="content" source="media/how-to-create-custom-event-trigger/custom-event-3-parameters.png" alt-text="Screenshot that shows the parameters page to reference data payload in a custom event.":::
9492

95-
1. After you've entered the parameters, select **OK**.
93+
1. After you enter the parameters, select **OK**.
9694

9795
## Advanced filtering
9896

99-
Custom event trigger supports advanced filtering capabilities, similar to [Event Grid Advanced Filtering](../event-grid/event-filtering.md#advanced-filtering). These conditional filters allow pipelines to trigger based upon the _values_ of event payload. For instance, you may have a field in the event payload, named _Department_, and pipeline should only trigger if _Department_ equals to _Finance_. You may also specify complex logic, such as _date_ field in list [1, 2, 3, 4, 5], _month_ field __not__ in list [11, 12], _tag_ field contains any of ['Fiscal Year 2021', 'FiscalYear2021', 'FY2021'].
97+
Custom event triggers support advanced filtering capabilities, similar to [Event Grid advanced filtering](../event-grid/event-filtering.md#advanced-filtering). These conditional filters allow pipelines to trigger based on the _values_ of the event payload. For instance, you might have a field in the event payload named _Department_, and the pipeline should only trigger if _Department_ equals _Finance_. You might also specify complex logic, such as the _date_ field in list [1, 2, 3, 4, 5], the _month_ field *not* in the list [11, 12], and if the _tag_ field contains [Fiscal Year 2021, FiscalYear2021, or FY2021].
10098

101-
:::image type="content" source="media/how-to-create-custom-event-trigger/custom-event-5-advanced-filters.png" alt-text="Screenshot of setting advanced filters for customer event trigger":::
99+
:::image type="content" source="media/how-to-create-custom-event-trigger/custom-event-5-advanced-filters.png" alt-text="Screenshot that shows setting advanced filters for a customer event trigger.":::
102100

103-
As of today custom event trigger supports a __subset__ of [advanced filtering operators](../event-grid/event-filtering.md#advanced-filtering) in Event Grid. Following filter conditions are supported:
101+
As of today, custom event triggers support a *subset* of [advanced filtering operators](../event-grid/event-filtering.md#advanced-filtering) in Event Grid. The following filter conditions are supported:
104102

105-
* NumberIn
106-
* NumberNotIn
107-
* NumberLessThan
108-
* NumberGreaterThan
109-
* NumberLessThanOrEquals
110-
* NumberGreaterThanOrEquals
111-
* BoolEquals
112-
* StringContains
113-
* StringBeginsWith
114-
* StringEndsWith
115-
* StringIn
116-
* StringNotIn
103+
* `NumberIn`
104+
* `NumberNotIn`
105+
* `NumberLessThan`
106+
* `NumberGreaterThan`
107+
* `NumberLessThanOrEquals`
108+
* `NumberGreaterThanOrEquals`
109+
* `BoolEquals`
110+
* `StringContains`
111+
* `StringBeginsWith`
112+
* `StringEndsWith`
113+
* `StringIn`
114+
* `StringNotIn`
117115

118-
Select **+New** to add new filter conditions.
116+
Select **+ New** to add new filter conditions.
119117

120-
Additionally, custom event triggers obey the [same limitations as Event Grid](../event-grid/event-filtering.md#limitations), including:
118+
Custom event triggers also obey the [same limitations as Event Grid](../event-grid/event-filtering.md#limitations), such as:
121119

122-
* 5 advanced filters and 25 filter values across all the filters per custom event trigger
123-
* 512 characters per string value
124-
* 5 values for in and not in operators
125-
* keys cannot have `.` (dot) character in them, for example, `[email protected]`. Currently, there's no support for escape characters in keys.
120+
* 5 advanced filters and 25 filter values across all the filters per custom event trigger.
121+
* 512 characters per string value.
122+
* 5 values for `in` and `not in` operators.
123+
* Keys can't have the `.` (dot) character in them, for example, `[email protected]`. Currently, there's no support for escape characters in keys.
126124
* The same key can be used in more than one filter.
127125

128-
Data Factory relies upon the latest _GA_ version of [Event Grid API](../event-grid/whats-new.md). As new API versions get to GA stage, Data Factory will expand its support for more advanced filtering operators.
126+
Data Factory relies on the latest general availability (GA) version of the [Event Grid API](../event-grid/whats-new.md). As new API versions get to the GA stage, Data Factory expands its support for more advanced filtering operators.
129127

130128
## JSON schema
131129

132-
The following table provides an overview of the schema elements that are related to custom event triggers:
130+
The following table provides an overview of the schema elements that are related to custom event triggers.
133131

134132
| JSON element | Description | Type | Allowed values | Required |
135133
|---|----------------------------|---|---|---|
136-
| `scope` | The Azure Resource Manager resource ID of the Event Grid topic. | String | Azure Resource Manager ID | Yes |
134+
| `scope` | The Azure Resource Manager resource ID of the Event Grid topic. | String | Azure Resource Manager ID | Yes. |
137135
| `events` | The type of events that cause this trigger to fire. | Array of strings | | Yes, at least one value is expected. |
138-
| `subjectBeginsWith` | The `subject` field must begin with the provided pattern for the trigger to fire. For example, _factories_ only fire the trigger for event subjects that start with *factories*. | String | | No |
139-
| `subjectEndsWith` | The `subject` field must end with the provided pattern for the trigger to fire. | String | | No |
140-
| `advancedFilters` | List of JSON blobs, each specifying a filter condition. Each blob specifies `key`, `operatorType`, and `values`. | List of JSON blob | | No |
136+
| `subjectBeginsWith` | The `subject` field must begin with the provided pattern for the trigger to fire. For example, *factories* only fire the trigger for event subjects that start with *factories*. | String | | No. |
137+
| `subjectEndsWith` | The `subject` field must end with the provided pattern for the trigger to fire. | String | | No. |
138+
| `advancedFilters` | List of JSON blobs, each specifying a filter condition. Each blob specifies `key`, `operatorType`, and `values`. | List of JSON blobs | | No. |
141139

142140
## Role-based access control
143141

144-
Azure Data Factory uses Azure role-based access control (RBAC) to prohibit unauthorized access. To function properly, Data Factory requires access to:
142+
Data Factory uses Azure RBAC to prohibit unauthorized access. To function properly, Data Factory requires access to:
143+
145144
- Listen to events.
146145
- Subscribe to updates from events.
147146
- Trigger pipelines linked to custom events.
148147

149-
To successfully create or update a custom event trigger, you need to sign in to Data Factory with an Azure account that has appropriate access. Otherwise, the operation will fail with an _Access Denied_ error.
148+
To successfully create or update a custom event trigger, you need to sign in to Data Factory with an Azure account that has appropriate access. Otherwise, the operation fails with the message "Access Denied."
150149

151-
Data Factory doesn't require special permission to your Event Grid. You also do *not* need to assign special Azure RBAC role permission to the Data Factory service principal for the operation.
150+
Data Factory doesn't require special permission to your instance of Event Grid. You also do *not* need to assign special Azure RBAC role permission to the Data Factory service principal for the operation.
152151

153152
Specifically, you need `Microsoft.EventGrid/EventSubscriptions/Write` permission on `/subscriptions/####/resourceGroups//####/providers/Microsoft.EventGrid/topics/someTopics`.
154153

155-
- When authoring in the data factory (in the development environment for instance), the Azure account signed in needs to have the above permission
156-
- When publishing through [CI/CD](continuous-integration-delivery.md), the account used to publish the ARM template into the testing or production factory needs to have the above permission.
154+
- When you author in the data factory (in the development environment, for instance), the Azure account signed in needs to have the preceding permission.
155+
- When you publish through [continuous integration and continuous delivery](continuous-integration-delivery.md), the account used to publish the Azure Resource Manager template into the testing or production factory needs to have the preceding permission.
157156

158157
## Related content
159158

0 commit comments

Comments
 (0)