Skip to content

Commit 99e069a

Browse files
PRMerger9PRMerger9
authored
Merge pull request #96645 from linda33wj/master
Update ADF connector articles
2 parents 7b93b90 + b74da31 commit 99e069a

7 files changed

+79
-19
lines changed

‎articles/data-factory/connector-azure-cosmos-db-mongodb-api.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ ms.service: multiple
1010
ms.workload: data-services
1111
ms.tgt_pltfrm: na
1212
ms.topic: conceptual
13-
ms.date: 08/01/2019
13+
ms.date: 11/20/2019
1414
ms.author: jingwang
1515

1616
---
@@ -166,7 +166,7 @@ The following properties are supported in the Copy Activity **sink** section:
166166
| Property | Description | Required |
167167
|:--- |:--- |:--- |
168168
| type | The **type** property of the Copy Activity sink must be set to **CosmosDbMongoDbApiSink**. |Yes |
169-
| writeBehavior |Describes how to write data to Azure Cosmos DB. Allowed values: **insert** and **upsert**.<br/><br/>The behavior of **upsert** is to replace the document if a document with the same ID already exists; otherwise, insert the document.<br /><br />**Note**: Data Factory automatically generates an ID for a document if an ID isn't specified either in the original document or by column mapping. This means that you must ensure that, for **upsert** to work as expected, your document has an ID. |No<br />(the default is **insert**) |
169+
| writeBehavior |Describes how to write data to Azure Cosmos DB. Allowed values: **insert** and **upsert**.<br/><br/>The behavior of **upsert** is to replace the document if a document with the same `_id` already exists; otherwise, insert the document.<br /><br />**Note**: Data Factory automatically generates an `_id` for a document if an `_id` isn't specified either in the original document or by column mapping. This means that you must ensure that, for **upsert** to work as expected, your document has an ID. |No<br />(the default is **insert**) |
170170
| writeBatchSize | The **writeBatchSize** property controls the size of documents to write in each batch. You can try increasing the value for **writeBatchSize** to improve performance and decreasing the value if your document size being large. |No<br />(the default is **10,000**) |
171171
| writeBatchTimeout | The wait time for the batch insert operation to finish before it times out. The allowed value is timespan. | No<br/>(the default is **00:30:00** - 30 minutes) |
172172

‎articles/data-factory/connector-db2.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ ms.workload: data-services
1212
ms.tgt_pltfrm: na
1313

1414
ms.topic: conceptual
15-
ms.date: 09/04/2019
15+
ms.date: 11/20/2019
1616

1717
ms.author: jingwang
1818

@@ -74,6 +74,8 @@ The following properties are supported for DB2 linked service:
7474
| authenticationType |Type of authentication used to connect to the DB2 database.<br/>Allowed value is: **Basic**. |Yes |
7575
| username |Specify user name to connect to the DB2 database. |Yes |
7676
| password |Specify password for the user account you specified for the username. Mark this field as a SecureString to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). |Yes |
77+
| packageCollection | Specify under where the needed packages are auto created by ADF when querying the database | No |
78+
| certificateCommonName | When you use Secure Sockets Layer (SSL) or Transport Layer Security (TLS) encryption, you must enter a value for Certificate common name. | No |
7779
| connectVia | The [Integration Runtime](concepts-integration-runtime.md) to be used to connect to the data store. Learn more from [Prerequisites](#prerequisites) section. If not specified, it uses the default Azure Integration Runtime. |No |
7880

7981
**Example:**

‎articles/data-factory/connector-dynamics-crm-office-365.md

Lines changed: 60 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ author: linda33wj
1212
manager: craigg
1313
ms.reviewer: douglasl
1414
ms.custom: seo-lt-2019
15-
ms.date: 10/25/2019
15+
ms.date: 11/20/2019
1616
---
1717

1818
# Copy data from and to Dynamics 365 (Common Data Service) or Dynamics CRM by using Azure Data Factory
@@ -38,7 +38,7 @@ Refer to the following table on the supported authentication types and configura
3838

3939
| Dynamics versions | Authentication types | Linked service samples |
4040
|:--- |:--- |:--- |
41-
| Dynamics 365 online <br> Dynamics CRM Online | Office365 | [Dynamics online + Office365 auth](#dynamics-365-and-dynamics-crm-online) |
41+
| Common Data Service <br> Dynamics 365 online <br> Dynamics CRM Online | AAD service principal <br> Office365 | [Dynamics online + AAD service principal or Office365 auth](#dynamics-365-and-dynamics-crm-online) |
4242
| Dynamics 365 on-premises with IFD <br> Dynamics CRM 2016 on-premises with IFD <br> Dynamics CRM 2015 on-premises with IFD | IFD | [Dynamics on-premises with IFD + IFD auth](#dynamics-365-and-dynamics-crm-on-premises-with-ifd) |
4343

4444
For Dynamics 365 specifically, the following application types are supported:
@@ -73,22 +73,76 @@ The following properties are supported for the Dynamics linked service.
7373
| type | The type property must be set to **Dynamics**, **DynamicsCrm**, or **CommonDataServiceForApps**. | Yes |
7474
| deploymentType | The deployment type of the Dynamics instance. It must be **"Online"** for Dynamics online. | Yes |
7575
| serviceUri | The service URL of your Dynamics instance, e.g. `https://adfdynamics.crm.dynamics.com`. | Yes |
76-
| authenticationType | The authentication type to connect to a Dynamics server. Specify **"Office365"** for Dynamics online. | Yes |
77-
| username | Specify the user name to connect to Dynamics. | Yes |
78-
| password | Specify the password for the user account you specified for username. Mark this field as a SecureString to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
76+
| authenticationType | The authentication type to connect to a Dynamics server. Allowed values are: **AADServicePrincipal** or **"Office365"**. | Yes |
77+
| servicePrincipalId | Specify the Azure Active Directory application's client ID. | Yes when using `AADServicePrincipal` authentication |
78+
| servicePrincipalCredentialType | Specify the credential type to use for service principal authentication. Allowed values are: **ServicePrincipalKey** or **ServicePrincipalCert**. | Yes when using `AADServicePrincipal` authentication |
79+
| servicePrincipalCredential | Specify the service principal credential. <br>When using `ServicePrincipalKey` as credential type, `servicePrincipalCredential` can be a string (ADF will encrypt it upon linked service deployment) or a reference to a secret in AKV. <br>When using `ServicePrincipalCert` as credential, `servicePrincipalCredential` should be a reference to a certificate in AKV. | Yes when using `AADServicePrincipal` authentication |
80+
| username | Specify the user name to connect to Dynamics. | Yes when using `Office365` authentication |
81+
| password | Specify the password for the user account you specified for username. Mark this field as a SecureString to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes when using `Office365` authentication |
7982
| connectVia | The [integration runtime](concepts-integration-runtime.md) to be used to connect to the data store. If not specified, it uses the default Azure Integration Runtime. | No for source, Yes for sink if the source linked service doesn't have an integration runtime |
8083

8184
>[!NOTE]
8285
>The Dynamics connector used to use optional "organizationName" property to identify your Dynamics CRM/365 Online instance. While it keeps working, you are suggested to specify the new "serviceUri" property instead to gain better performance for instance discovery.
8386
87+
**Example: Dynamics online using AAD service principal + key authentication**
88+
89+
```json
90+
{
91+
"name": "DynamicsLinkedService",
92+
"properties": {
93+
"type": "Dynamics",
94+
"typeProperties": {
95+
"deploymentType": "Online",
96+
"serviceUri": "https://adfdynamics.crm.dynamics.com",
97+
"authenticationType": "AADServicePrincipal",
98+
"servicePrincipalId": "<service principal id>",
99+
"servicePrincipalCredentialType": "ServicePrincipalKey",
100+
"servicePrincipalCredential": "<service principal key>"
101+
},
102+
"connectVia": {
103+
"referenceName": "<name of Integration Runtime>",
104+
"type": "IntegrationRuntimeReference"
105+
}
106+
}
107+
}
108+
```
109+
**Example: Dynamics online using AAD service principal + certificate authentication**
110+
111+
```json
112+
{
113+
"name": "DynamicsLinkedService",
114+
"properties": {
115+
"type": "Dynamics",
116+
"typeProperties": {
117+
"deploymentType": "Online",
118+
"serviceUri": "https://adfdynamics.crm.dynamics.com",
119+
"authenticationType": "AADServicePrincipal",
120+
"servicePrincipalId": "<service principal id>",
121+
"servicePrincipalCredentialType": "ServicePrincipalCert",
122+
"servicePrincipalCredential": {
123+
"type": "AzureKeyVaultSecret",
124+
"store": {
125+
"referenceName": "<AKV reference>",
126+
"type": "LinkedServiceReference"
127+
},
128+
"secretName": "<certificate name in AKV>"
129+
}
130+
},
131+
"connectVia": {
132+
"referenceName": "<name of Integration Runtime>",
133+
"type": "IntegrationRuntimeReference"
134+
}
135+
}
136+
}
137+
```
138+
84139
**Example: Dynamics online using Office365 authentication**
85140

86141
```json
87142
{
88143
"name": "DynamicsLinkedService",
89144
"properties": {
90145
"type": "Dynamics",
91-
"description": "Dynamics online linked service using Office365 authentication",
92146
"typeProperties": {
93147
"deploymentType": "Online",
94148
"serviceUri": "https://adfdynamics.crm.dynamics.com",

‎articles/data-factory/connector-http.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ ms.workload: data-services
1212
ms.tgt_pltfrm: na
1313

1414
ms.topic: conceptual
15-
ms.date: 10/24/2019
15+
ms.date: 11/20/2019
1616
ms.author: jingwang
1717

1818
---
@@ -175,7 +175,7 @@ The following properties are supported for HTTP under `location` settings in for
175175
| Property | Description | Required |
176176
| ----------- | ------------------------------------------------------------ | -------- |
177177
| type | The type property under `location` in dataset must be set to **HttpServerLocation**. | Yes |
178-
| relativeUrl | A relative URL to the resource that contains the data. | No |
178+
| relativeUrl | A relative URL to the resource that contains the data. The HTTP connector copies data from the combined URL: `[URL specified in linked service]/[relative URL specified in dataset]`. | No |
179179

180180
> [!NOTE]
181181
> The supported HTTP request payload size is around 500 KB. If the payload size you want to pass to your web endpoint is larger than 500 KB, consider batching the payload in smaller chunks.
@@ -281,7 +281,7 @@ The following properties are supported for HTTP under `storeSettings` settings i
281281
| requestMethod | The HTTP method. <br>Allowed values are **Get** (default) and **Post**. | No |
282282
| addtionalHeaders | Additional HTTP request headers. | No |
283283
| requestBody | The body for the HTTP request. | No |
284-
| requestTimeout | The timeout (the **TimeSpan** value) for the HTTP request to get a response. This value is the timeout to get a response, not the timeout to read response data. The default value is **00:01:40**. | No |
284+
| httpRequestTimeout | The timeout (the **TimeSpan** value) for the HTTP request to get a response. This value is the timeout to get a response, not the timeout to read response data. The default value is **00:01:40**. | No |
285285
| maxConcurrentConnections | The number of the connections to connect to storage store concurrently. Specify only when you want to limit the concurrent connection to the data store. | No |
286286

287287
**Example:**

‎articles/data-factory/connector-rest.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ ms.service: data-factory
1010
ms.workload: data-services
1111
ms.tgt_pltfrm: na
1212
ms.topic: conceptual
13-
ms.date: 09/04/2019
13+
ms.date: 11/20/2019
1414
ms.author: jingwang
1515
---
1616
# Copy data from a REST endpoint by using Azure Data Factory
@@ -168,7 +168,7 @@ To copy data from REST, the following properties are supported:
168168
| Property | Description | Required |
169169
|:--- |:--- |:--- |
170170
| type | The **type** property of the dataset must be set to **RestResource**. | Yes |
171-
| relativeUrl | A relative URL to the resource that contains the data. When this property isn't specified, only the URL that's specified in the linked service definition is used. | No |
171+
| relativeUrl | A relative URL to the resource that contains the data. When this property isn't specified, only the URL that's specified in the linked service definition is used. The HTTP connector copies data from the combined URL: `[URL specified in linked service]/[relative URL specified in dataset]`. | No |
172172

173173
If you were setting `requestMethod`, `additionalHeaders`, `requestBody` and `paginationRules` in dataset, it is still supported as-is, while you are suggested to use the new model in activity source going forward.
174174

@@ -211,6 +211,9 @@ The following properties are supported in the copy activity **source** section:
211211
| httpRequestTimeout | The timeout (the **TimeSpan** value) for the HTTP request to get a response. This value is the timeout to get a response, not the timeout to read response data. The default value is **00:01:40**. | No |
212212
| requestInterval | The time to wait before sending the request for next page. The default value is **00:00:01** | No |
213213

214+
>[!NOTE]
215+
>REST connector ignores any "Accept" header specified in `additionalHeaders`. As REST connector only support response in JSON, tt will auto generate a header of `Accept: application/json`.
216+
214217
**Example 1: Using the Get method with pagination**
215218

216219
```json

‎articles/data-factory/control-flow-get-metadata-activity.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ ms.workload: data-services
1313
ms.tgt_pltfrm: na
1414

1515
ms.topic: conceptual
16-
ms.date: 08/12/2019
16+
ms.date: 11/20/2019
1717
ms.author: jingwang
1818

1919
---
@@ -54,6 +54,7 @@ The Get Metadata activity takes a dataset as an input and returns metadata infor
5454

5555
- For Amazon S3 and Google Cloud Storage, `lastModified` applies to the bucket and the key but not to the virtual folder, and `exists` applies to the bucket and the key but not to the prefix or virtual folder.
5656
- For Azure Blob storage, `lastModified` applies to the container and the blob but not to the virtual folder.
57+
- Wildcard filter on folders/files is not supported for Get Metadata activity.
5758

5859
**Relational database**
5960

0 commit comments

Comments
 (0)