You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-factory/airflow-sync-github-repository.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -37,15 +37,15 @@ To sync your GitHub repository by using the Workflow Orchestration Manager UI:
37
37
38
38
1. When you create a Workflow Orchestration Manager integration runtime, select **Enable git sync** in the **Airflow environment setup** dialog.
39
39
40
-
:::image type="content" source="media/airflow-git-sync-repository/enable-git-sync.png" alt-text="Screenshot that shows the Enable git sync checkbox in the Airflow environment setup dialog that appears during creation of an Airflow integration runtime.":::
40
+
:::image type="content" source="media/airflow-git-sync-repository/enable-git-sync.png" alt-text="Screenshot that shows the Enable git sync checkbox in the Airflow environment setup dialog that appears during creation of a Airflow integration runtime.":::
41
41
42
42
1. Select one of the following supported Git service types:
43
43
-**GitHub**
44
44
-**ADO**
45
45
-**GitLab**
46
46
-**Bitbucket**
47
47
48
-
:::image type="content" source="media/airflow-git-sync-repository/git-service-type.png" alt-text="Screenshot that shows the Git service type selection dropdown in the environment setup dialog that appears during creation of an Workflow Orchestration Manager integration runtime.":::
48
+
:::image type="content" source="media/airflow-git-sync-repository/git-service-type.png" alt-text="Screenshot that shows the Git service type selection dropdown in the environment setup dialog that appears during creation of a Workflow Orchestration Manager integration runtime.":::
49
49
50
50
1. Select a credential type:
51
51
@@ -59,7 +59,7 @@ To sync your GitHub repository by using the Workflow Orchestration Manager UI:
59
59
- GitLab personal access token
60
60
- Bitbucket personal access token
61
61
62
-
:::image type="content" source="media/airflow-git-sync-repository/git-pat-credentials.png" alt-text="Screenshot that shows the Git PAT credential options in the Airflow environment setup dialog that appears during creation of an AWorkflow Orchestration Manager integration runtime.":::
62
+
:::image type="content" source="media/airflow-git-sync-repository/git-pat-credentials.png" alt-text="Screenshot that shows the Git PAT credential options in the Airflow environment setup dialog that appears during creation of a Workflow Orchestration Manager integration runtime.":::
63
63
-**SPN** ([service principal name](https://devblogs.microsoft.com/devops/introducing-service-principal-and-managed-identity-support-on-azure-devops/)): Only ADO supports this credential type.
64
64
After you select this option, fill out the remaining fields based on the selected **Git service type**:
65
65
-**Git repo url** (required): The clone URL to the Git repository to sync.
@@ -68,7 +68,7 @@ To sync your GitHub repository by using the Workflow Orchestration Manager UI:
68
68
-**Service principal secret** (required): A manually generated secret in the service principal whose value is used to authenticate and access the ADO repo.
69
69
-**Service principal tenant id** (required): The service principal tenant ID.
70
70
71
-
:::image type="content" source="media/airflow-git-sync-repository/git-spn-credentials.png" alt-text="Screenshot that shows the Git SPN credential options in the Airflow environment setup dialog that appears during creation of an Workflow Orchestration Manager integration runtime.":::
71
+
:::image type="content" source="media/airflow-git-sync-repository/git-spn-credentials.png" alt-text="Screenshot that shows the Git SPN credential options in the Airflow environment setup dialog that appears during creation of a Workflow Orchestration Manager integration runtime.":::
72
72
73
73
1. Fill in the rest of the fields with the required information.
74
74
1. Select **Create**.
@@ -295,7 +295,7 @@ This process assumes that your private package was autosynced via Git sync. You
295
295
296
296
For example, if your private package is in `/dags/test/private.whl` in a GitHub repo, you should add the requirement `/opt/airflow/git/\<repoName\>.git/dags/test/private.whl` in the Workflow Orchestration Manager environment.
297
297
298
-
:::image type="content" source="media/airflow-git-sync-repository/airflow-private-package.png" alt-text="Screenshot that shows the Airflow requirements section in the Airflow environment setup dialog that appears during creation of an Workflow Orchestration Manager integration runtime.":::
298
+
:::image type="content" source="media/airflow-git-sync-repository/airflow-private-package.png" alt-text="Screenshot that shows the Airflow requirements section in the Airflow environment setup dialog that appears during creation of a Workflow Orchestration Manager integration runtime.":::
Copy file name to clipboardExpand all lines: articles/data-factory/azure-integration-runtime-ip-addresses.md
+1-3Lines changed: 1 addition & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,9 +15,7 @@ ms.date: 02/13/2025
15
15
The IP addresses that Azure Integration Runtime uses depends on the region where your Azure integration runtime is located. *All* Azure integration runtimes that are in the same region use the same IP address ranges.
16
16
17
17
> [!IMPORTANT]
18
-
> Azure Integration Runtime which enable Managed Virtual Network and all data flows don't support the use of fixed IP ranges.
19
-
>
20
-
> You can use these IP ranges for Data Movement, Pipeline and External activities executions. These IP ranges can be used for filtering in data stores/ Network Security Group (NSG)/ Firewalls for inbound access from Azure Integration runtime.
18
+
> The Azure integration runtime lets you used a managed virtual network. Some data flows require the use of fixed IP ranges. You can use these IP ranges for data movement, pipeline and external activity executions, as well as for filtering in data stores, network security groups (NSGs), and firewalls for inbound access from the Azure integration runtime.
21
19
22
20
## Azure Integration Runtime IP addresses: Specific regions
Copy file name to clipboardExpand all lines: articles/data-factory/concepts-change-data-capture.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,7 +25,7 @@ When you perform data integration and ETL processes in the cloud, your jobs can
25
25
26
26
### Change Data Capture factory resource
27
27
28
-
The easiest and quickest way to get started in data factory with CDC is through the factory level Change Data Capture resource. From the main pipeline designer, select on **New** under Factory Resources to create a new Change Data Capture. The CDC factory resource provides a configuration walk-through experience where you can select your sources and destinations, apply optional transformations, and then select start to begin your data capture. With the CDC resource, you don't need to design pipelines or data flow activities. You're also only billed for four cores of General Purpose data flows while your data in being processed. You can set a preferred latency, which ADF uses to wake up and look for changed data. That initial check is the only time you are billed. The top-level CDC resource is also the ADF method of running your processes continuously. Pipelines in ADF are batch only, but the CDC resource can run continuously.
28
+
The easiest and quickest way to get started in data factory with CDC is through the factory level Change Data Capture resource. From the main pipeline designer, select **New** under Factory Resources to create a new Change Data Capture. The CDC factory resource provides a configuration walk-through experience where you can select your sources and destinations, apply optional transformations, and then select start to begin your data capture. With the CDC resource, you don't need to design pipelines or data flow activities. You're also only billed for four cores of General Purpose data flows while your data in being processed. You can set a preferred latency, which ADF uses to wake up and look for changed data. That initial check is the only time you are billed. The top-level CDC resource is also the ADF method of running your processes continuously. Pipelines in ADF are batch only, but the CDC resource can run continuously.
29
29
30
30
### Native change data capture in mapping data flow
Copy file name to clipboardExpand all lines: articles/data-factory/concepts-pipelines-activities.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -102,7 +102,7 @@ Data factory displays the pipeline editor where you can find:
102
102
1. The pipeline configurations pane, including parameters, variables, general settings, and output.
103
103
1. The pipeline properties pane, where the pipeline name, optional description, and annotations can be configured. This pane also shows any related items to the pipeline within the data factory.
104
104
105
-
:::image type="content" source="media/concepts-pipelines-activities/pipeline-configuration-with-ui.png" alt-text="Shows the pipeline editor pane in Azure Data Factory studio with each of the sections described highlighted.":::
105
+
:::image type="content" source="media/concepts-pipelines-activities/pipeline-configuration-with-ui.png" alt-text="Screenshot that shows the pipeline editor pane in Azure Data Factory studio with each of the sections described highlighted.":::
106
106
107
107
# [Synapse Analytics](#tab/synapse-analytics)
108
108
To create a new pipeline, navigate to the Integrate tab in Synapse Studio (represented by the pipeline icon), then select the plus sign and choose Pipeline from the menu.
Copy file name to clipboardExpand all lines: articles/data-factory/connector-amazon-simple-storage-service.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -372,7 +372,7 @@ To learn details about the properties, check [Delete activity](delete-activity.m
372
372
|:--- |:--- |:--- |
373
373
| type | The **type** property of the dataset must be set to **AmazonS3Object**. |Yes |
374
374
| bucketName | The S3 bucket name. The wildcard filter is not supported. |Yes for the Copy or Lookup activity, no for the GetMetadata activity |
375
-
| key | The name or wildcard filter of the S3 object key under the specified bucket. Applies only when the **prefix** property is not specified. <br/><br/>The wildcard filter is supported for both the folder part and the file name part. Allowed wildcards are: `*` (matches zero or more characters) and `?` (matches zero or single character).<br/>- Example 1: `"key": "rootfolder/subfolder/*.csv"`<br/>- Example 2: `"key": "rootfolder/subfolder/???20180427.txt"`<br/>See more example in [Folder and file filter examples](#folder-and-file-filter-examples). Use `^` to escape if your actual folder or file name has a wildcard or this escape character inside. |No |
375
+
| key | The name or wildcard filter of the S3 object key under the specified bucket. Applies only when the **prefix** property is not specified. <br/><br/>The wildcard filter is supported for both the folder part and the file name part. Allowed wildcards are: `*` (matches zero or more characters) and `?` (matches zero or single character).<br/>- Example 1: `"key": "rootfolder/subfolder/*.csv"`<br/>- Example 2: `"key": "rootfolder/subfolder/???20180427.txt"`<br/>See more examples in [Folder and file filter examples](#folder-and-file-filter-examples). Use `^` to escape if your actual folder or file name has a wildcard or this escape character inside. |No |
376
376
| prefix | Prefix for the S3 object key. Objects whose keys start with this prefix are selected. Applies only when the **key** property is not specified. |No |
377
377
| version | The version of the S3 object, if S3 versioning is enabled. If a version is not specified, the latest version will be fetched. |No |
378
378
| modifiedDatetimeStart | Files are filtered based on the attribute: last modified. The files will be selected if their last modified time is greater than or equal to `modifiedDatetimeStart` and less than `modifiedDatetimeEnd`. The time is applied to the UTC time zone in the format of "2018-12-01T05:00:00Z". <br/><br/> Be aware that enabling this setting will affect the overall performance of data movement when you want to filter huge amounts of files. <br/><br/> The properties can be **NULL**, which means no file attribute filter will be applied to the dataset. When `modifiedDatetimeStart` has a datetime value but `modifiedDatetimeEnd` is **NULL**, the files whose last modified attribute is greater than or equal to the datetime value will be selected. When `modifiedDatetimeEnd` has a datetime value but `modifiedDatetimeStart` is NULL, the files whose last modified attribute is less than the datetime value will be selected.| No |
Copy file name to clipboardExpand all lines: articles/data-factory/connector-azure-sql-data-warehouse.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -543,11 +543,11 @@ To copy data to Azure Synapse Analytics, set the sink type in Copy Activity to *
543
543
| allowCopyCommand | Indicates whether to use [COPY statement](/sql/t-sql/statements/copy-into-transact-sql) to load data into Azure Synapse Analytics. `allowCopyCommand`and`allowPolyBase` cannot be both true. <br/><br/>See [Use COPY statement to load data into Azure Synapse Analytics](#use-copy-statement) section for constraints and details.<br/><br/>Allowed values are **True** and **False** (default). | No.<br>Apply when using COPY. |
544
544
| copyCommandSettings | A group of properties that can be specified when `allowCopyCommand` property is set to TRUE. | No.<br/>Apply when using COPY. |
545
545
| writeBatchSize | Number of rows to inserts into the SQL table **per batch**.<br/><br/>The allowed value is **integer** (number of rows). By default, the service dynamically determines the appropriate batch size based on the row size. | No.<br/>Apply when using bulk insert. |
546
-
| writeBatchTimeout | The wait time for the insert, upsert and stored procedure operation to complete before it times out. <br/>Allowed values are for the timespan. An example is "00:30:00" for 30 minutes. If no value is specified, the timeout defaults to "00:30:00". | No.<br/>Apply when using bulk insert. |
546
+
| writeBatchTimeout | The wait time for the insert, upsert and stored procedure operation to complete before it times out. <br/>Allowed values are for the timespan. An example is "00:30:00" for 30 minutes. If no value is specified, the time-out defaults to "00:30:00". | No.<br/>Apply when using bulk insert. |
547
547
| preCopyScript | Specify a SQL query for Copy Activity to run before writing data into Azure Synapse Analytics in each run. Use this property to clean up the preloaded data. | No |
548
548
| tableOption | Specifies whether to [automatically create the sink table](copy-activity-overview.md#auto-create-sink-tables), if it does not exist, based on the source schema. Allowed values are: `none` (default), `autoCreate`. |No |
549
549
| disableMetricsCollection | The service collects metrics such as Azure Synapse Analytics DWUs for copy performance optimization and recommendations, which introduce additional master DB access. If you are concerned with this behavior, specify `true` to turn it off. | No (default is `false`) |
550
-
|maxConcurrentConnections|Theupperlimit of concurrentconnectionsestablished to thedatastoreduringtheactivityrun.Specify a valueonlywhenyouwant to limitconcurrentconnections.| No |
550
+
|maxConcurrentConnections|Theupperlimit of concurrentconnectionsestablished to thedatastoreduringtheactivityrun.Specify a valueonlywhenyouwant to limitconcurrentconnections.| No |
551
551
| WriteBehavior | Specify the write behavior for copy activity to load data into Azure Synapse Analytics. <br/> The allowed value is **Insert**and**Upsert**. By default, the service uses insert to load data. | No |
552
552
| upsertSettings | Specify the group of the settings for write behavior. <br/> Apply when the WriteBehavior option is `Upsert`. | No |
553
553
| ***Under `upsertSettings`:*** | | |
@@ -1140,7 +1140,7 @@ To upgrade the Azure Synapse Analytics version, in **Edit linked service** page,
1140
1140
1141
1141
The table below shows the differences between Azure Synapse Analytics using the recommended and the legacy version.
1142
1142
1143
-
| Recommended version| Legacy version |
1143
+
| Recommended version| Legacy version |
1144
1144
|:--- |:--- |
1145
1145
| Support TLS 1.3 via `encrypt` as `strict`. | TLS 1.3 is not supported.|
Copy file name to clipboardExpand all lines: articles/data-factory/connector-azure-sql-database.md
+8-8Lines changed: 8 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -154,12 +154,12 @@ To use SQL authentication, in addition to the generic properties that are descri
154
154
"authenticationType": "SQL",
155
155
"userName": "<user name>",
156
156
"password": {
157
-
"type":"AzureKeyVaultSecret",
157
+
"type":"AzureKeyVaultSecret",
158
158
"store": {
159
-
"referenceName":"<Azure Key Vault linked service name>",
160
-
"type":"LinkedServiceReference"
159
+
"referenceName":"<Azure Key Vault linked service name>",
160
+
"type":"LinkedServiceReference"
161
161
},
162
-
"secretName":"<secretName>"
162
+
"secretName":"<secretName>"
163
163
}
164
164
},
165
165
"connectVia": {
@@ -579,9 +579,9 @@ To copy data to Azure SQL Database, the following properties are supported in th
579
579
| sqlWriterTableType |The table type name to be used in the stored procedure. The copy activity makes the data being moved available in a temp table with this table type. Stored procedure code can then merge the data that's being copied with existing data. |No |
580
580
| storedProcedureParameters |Parameters for the stored procedure.<br/>Allowed values are name and value pairs. Names and casing of parameters must match the names and casing of the stored procedure parameters. | No |
581
581
| writeBatchSize | Number of rows to insert into the SQL table *per batch*.<br/> The allowed value is **integer** (number of rows). By default, the service dynamically determines the appropriate batch size based on the row size. | No |
582
-
| writeBatchTimeout | The wait time for the insert, upsert and stored procedure operation to complete before it times out. <br/>Allowed values are for the timespan. An example is "00:30:00" for 30 minutes. If no value is specified, the timeout defaults to "00:30:00". | No |
582
+
| writeBatchTimeout | The wait time for the insert, upsert and stored procedure operation to complete before it times out. <br/>Allowed values are for the timespan. An example is "00:30:00" for 30 minutes. If no value is specified, the time-out defaults to "00:30:00". | No |
583
583
| disableMetricsCollection | The service collects metrics such as Azure SQL Database DTUs for copy performance optimization and recommendations, which introduces additional master DB access. If you are concerned with this behavior, specify `true` to turn it off. | No (default is `false`) |
584
-
|maxConcurrentConnections|Theupperlimit of concurrentconnectionsestablished to thedatastoreduringtheactivityrun.Specify a valueonlywhenyouwant to limitconcurrentconnections.| No |
584
+
|maxConcurrentConnections|Theupperlimit of concurrentconnectionsestablished to thedatastoreduringtheactivityrun.Specify a valueonlywhenyouwant to limitconcurrentconnections.| No |
585
585
| WriteBehavior | Specify the write behavior for copy activity to load data into Azure SQL Database. <br/> The allowed value is **Insert** and **Upsert**. By default, the service uses insert to load data. | No |
586
586
| upsertSettings | Specify the group of the settings for write behavior. <br/> Apply when the WriteBehavior option is `Upsert`. | No |
587
587
| ***Under `upsertSettings`:*** | | |
@@ -847,7 +847,7 @@ The following sample shows how to use a stored procedure to do an upsert into a
847
847
}
848
848
```
849
849
850
-
When writing data tointo AzureSQL Database using stored procedure, the sink splits the source data into mini batches then do the insert, so the extra query in stored procedure can be executed multiple times.If you have the query for the copy activity to run before writing data into Azure SQL Database, it's not recommended to add it to the stored procedure, add it in the **Pre-copy script**box.
850
+
When writing data tointo AzureSQL Database using stored procedure, the sink splits the source data into mini batches then do the insert, so the extra query in stored procedure can be executed multiple times.If you have the query for the copy activity to run before writing data into Azure SQL Database, it's not recommended to add it to the stored procedure, add it in the **Pre-copy script**box.
851
851
852
852
## Mapping data flow properties
853
853
@@ -1094,7 +1094,7 @@ To upgrade the Azure SQL Database version, in **Edit linked service** page, sele
1094
1094
1095
1095
The table below shows the differences between Azure SQL Database using the recommended and the legacy version.
1096
1096
1097
-
| Recommended version| Legacy version |
1097
+
| Recommended version| Legacy version |
1098
1098
|:--- |:--- |
1099
1099
| Support TLS 1.3 via `encrypt` as `strict`. | TLS 1.3 is not supported.|
0 commit comments