Skip to content

Commit 8076a54

Browse files
garycentricgarycentric
authored
Merge pull request #115624 from linda33wj/spo-list-support
Add SPO List connector in ADF
2 parents d413949 + 7367d31 commit 8076a54

7 files changed

+253
-8
lines changed

‎articles/data-factory/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -368,6 +368,8 @@
368368
href: connector-servicenow.md
369369
- name: SFTP
370370
href: connector-sftp.md
371+
- name: SharePoint Online List
372+
href: connector-sharepoint-online-list.md
371373
- name: Shopify
372374
href: connector-shopify.md
373375
- name: Spark

‎articles/data-factory/connector-sharepoint-online-list.md

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
---
2+
title: Copy data from SharePoint Online List by using Azure Data Factory
3+
description: Learn how to copy data from SharePoint Online List to supported sink data stores by using a copy activity in an Azure Data Factory pipeline.
4+
services: data-factory
5+
documentationcenter: ''
6+
author: linda33wj
7+
manager: shwang
8+
ms.reviewer: douglasl
9+
10+
ms.service: data-factory
11+
ms.workload: data-services
12+
ms.topic: conceptual
13+
ms.date: 05/19/2020
14+
ms.author: jingwang
15+
16+
---
17+
# Copy data from SharePoint Online List by using Azure Data Factory
18+
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
19+
20+
This article outlines how to use Copy Activity in Azure Data Factory to copy data from SharePoint Online List. The article builds on [Copy Activity in Azure Data Factory](copy-activity-overview.md), which presents a general overview of Copy Activity.
21+
22+
## Supported capabilities
23+
24+
This SharePoint Online List connector is supported for the following activities:
25+
26+
- [Copy activity](copy-activity-overview.md) with [supported source/sink matrix](copy-activity-overview.md)
27+
- [Lookup activity](control-flow-lookup-activity.md)
28+
29+
You can copy data from SharePoint Online List to any supported sink data store. For a list of data stores that Copy Activity supports as sources and sinks, see [Supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).
30+
31+
Specifically, this SharePoint List Online connector uses service principal authentication and retrieves data via OData protocol.
32+
33+
> [!TIP]
34+
> This connector supports copying data from SharePoint Online **List** but not file. Learn how to copy file from [Copy file from SharePoint Online](#copy-file-from-sharepoint-online) section.
35+
36+
## Prerequisites
37+
38+
The SharePoint List Online connector uses service principal authentication to connect to SharePoint. Follow these steps to set it up:
39+
40+
1. Register an application entity in Azure Active Directory (Azure AD) by following [Register your application with an Azure AD tenant](../storage/common/storage-auth-aad-app.md#register-your-application-with-an-azure-ad-tenant). Make note of the following values, which you use to define the linked service:
41+
42+
- Application ID
43+
- Application key
44+
- Tenant ID
45+
46+
2. Grant SharePoint Online site permission to your registered application:
47+
48+
> [!NOTE]
49+
> This operation requires SharePoint Online site owner permission. You can find the owner by going to the site home page -> click the "X members" in the right corner -> check who has the "Owner" role.
50+
51+
1. Open SharePoint Online site link e.g. `https://[your_site_url]/_layouts/15/appinv.aspx` (replace tenant and site name).
52+
2. Search the application ID you registered, fill the empty fields, and click "Create".
53+
54+
- App Domain: localhost.com
55+
- Redirect URL: https://www.localhost.com
56+
- Permission Request XML:
57+
58+
```xml
59+
<AppPermissionRequests AllowAppOnlyPolicy="true">
60+
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="Read"/>
61+
</AppPermissionRequests>
62+
```
63+
64+
![sharepoint grant permission](media/connector-sharepoint-online-list/sharepoint-online-grant-permission.png)
65+
66+
3. Click "Trust It" for this app.
67+
68+
## Get started
69+
70+
[!INCLUDE [data-factory-v2-connector-get-started](../../includes/data-factory-v2-connector-get-started.md)]
71+
72+
The following sections provide details about properties you can use to define Data Factory entities that are specific to SharePoint Online List connector.
73+
74+
## Linked service properties
75+
76+
The following properties are supported for an SharePoint Online List linked service:
77+
78+
| **Property** | **Description** | **Required** |
79+
| ------------------- | ------------------------------------------------------------ | ------------ |
80+
| type | The type property must be set to: **SharePointOnlineList**. | Yes |
81+
| siteUrl | The SharePoint Online site url, e.g. `https://contoso.sharepoint.com/sites/siteName`. | Yes |
82+
| servicePrincipalId | The Application (client) ID of the application registered in Azure Active Directory. | Yes |
83+
| servicePrincipalKey | The application's key. Mark this field as a **SecureString** to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
84+
| tenantId | The tenant ID under which your application resides. | Yes |
85+
| connectVia | The [Integration Runtime](concepts-integration-runtime.md) to use to connect to the data store. Learn more from [Prerequisites](#prerequisites), earlier in this article. If not specified, the default Azure Integration Runtime is used. | No |
86+
87+
**Example:**
88+
89+
```json
90+
{
91+
"name": "SharePointOnlineList",
92+
"properties": {
93+
"type": "SharePointOnlineList",
94+
"typeProperties": {
95+
"siteUrl": "<site URL>",
96+
"servicePrincipalId": "<service principal id>",
97+
"servicePrincipalKey": {
98+
"type": "SecureString",
99+
"value": "<service principal key>"
100+
},
101+
"tenantId": "<tenant ID>"
102+
}
103+
}
104+
}
105+
```
106+
107+
## Dataset properties
108+
109+
For a full list of sections and properties that are available for defining datasets, see [Datasets and linked services](concepts-datasets-linked-services.md). The following section provides a list of the properties supported by the SAP table dataset.
110+
111+
| Property | Description | Required |
112+
|:--- |:--- |:--- |
113+
| type | The **type** property of the dataset must be set to **SharePointOnlineLResource**. | Yes |
114+
| listName | The name of the SharePoint Online List. | Yes |
115+
116+
**Example**
117+
118+
```json
119+
{
120+
"name": "SharePointOnlineListDataset",
121+
"properties":
122+
{
123+
"type": "SharePointOnlineListResource",
124+
"linkedServiceName": {
125+
"referenceName": "<SharePoint Online List linked service name>",
126+
"type": "LinkedServiceReference"
127+
},
128+
"typeProperties":
129+
{
130+
"listName": "<name of the list>"
131+
}
132+
}
133+
}
134+
```
135+
136+
## Copy Activity properties
137+
138+
For a full list of sections and properties that are available for defining activities, see [Pipelines](concepts-pipelines-activities.md). The following section provides a list of the properties supported by the SharePoint Online List source.
139+
140+
### SharePoint Online List as source
141+
142+
To copy data from SharePoint Online List, the following properties are supported in the Copy Activity **source** section:
143+
144+
| Property | Description | Required |
145+
|:--- |:--- |:--- |
146+
| type | The **type** property of the Copy Activity source must be set to **SharePointOnlineListSource**. | Yes |
147+
| query | Custom OData query options for filtering data. Example: `"$top=10&$select=Title,Number"`. | No |
148+
| httpRequestTimeout | The timeout (in second) for the HTTP request to get a response. Default is 300 (5 minutes). | No |
149+
150+
**Example**
151+
152+
```json
153+
"activities":[
154+
{
155+
"name": "CopyFromSharePointOnlineList",
156+
"type": "Copy",
157+
"inputs": [
158+
{
159+
"referenceName": "<SharePoint Online List input dataset name>",
160+
"type": "DatasetReference"
161+
}
162+
],
163+
"outputs": [
164+
{
165+
"referenceName": "<output dataset name>",
166+
"type": "DatasetReference"
167+
}
168+
],
169+
"typeProperties": {
170+
"source": {
171+
"type": "SharePointOnlineListSource",
172+
"query": "<OData query e.g. $top=10&$select=Title,Number>"
173+
},
174+
"sink": {
175+
"type": "<sink type>"
176+
}
177+
}
178+
}
179+
]
180+
```
181+
182+
## Data type mapping for SharePoint Online List
183+
184+
When you copy data from SharePoint Online List, the following mappings are used between SharePoint Online List data types and Azure Data Factory interim data types.
185+
186+
| **SharePoint Online data type** | **OData data type** | **Azure Data Factory interim data type** |
187+
| ----------------------------------------------- | ---------------------------------------------------- | ---------------------------------------- |
188+
| Single line of text | Edm.String | String |
189+
| Multiple lines of text | Edm.String | String |
190+
| Choice (menu to choose from) | Edm.String | String |
191+
| Number (1, 1.0, 100) | Edm.Double | Double |
192+
| Currency ($, ¥, €) | Edm.Double | Double |
193+
| Date and Time | Edm.DateTime | DateTime |
194+
| Lookup (information already on this site) | Edm.Int32 | Int32 |
195+
| Yes/No (check box) | Edm.Boolean | Boolean |
196+
| Person or Group | Edm.Int32 | Int32 |
197+
| Hyperlink or Picture | Edm.String | String |
198+
| Calculated (calculation based on other columns) | Edm.String / Edm.Double / Edm.DateTime / Edm.Boolean | String / Double / DateTime / Boolean |
199+
| Attachment | Not supported | |
200+
| Task Outcome | Not supported | |
201+
| External Data | Not supported | |
202+
| Managed Metadata | Not supported | |
203+
204+
## Copy file from SharePoint Online
205+
206+
You can copy file from SharePoint Online by using **Web activity** to authenticate and grab access token from SPO, then passing to subsequent **Copy activity** to copy data with **HTTP connector as source**.
207+
208+
![sharepoint copy file flow](media/connector-sharepoint-online-list/sharepoint-online-copy-file-flow.png)
209+
210+
1. Follow the [Prerequisites](#prerequisites) section to create AAD application and grant permission to SharePoint Online.
211+
212+
2. Create a **Web Activity** to get the access token from SharePoint Online:
213+
214+
- **URL**: `https://accounts.accesscontrol.windows.net/[Tenant-ID]/tokens/OAuth/2`. Replace the tenant ID.
215+
- **Method**: POST
216+
- **Headers**:
217+
- Content-Type: application/x-www-form-urlencoded
218+
- **Body**: `grant_type=client_credentials&client_id=[Client-ID]@[Tenant-ID]&client_secret=[Client-Secret]&resource=00000003-0000-0ff1-ce00-000000000000/[Tenant-Name].sharepoint.com@[Tenant-ID]`. Replace the client ID, client secret, tenant ID and tenant name.
219+
220+
> [!CAUTION]
221+
> Set the Secure Output option to true in Web activity to prevent the token value from being logged in plain text. Any further activities that consume this value should have their Secure Input option set to true.
222+
223+
3. Chain with a **Copy activity** with HTTP connector as source to copy SharePoint Online file content:
224+
225+
- HTTP linked service:
226+
- **Base URL**: `https://[site-url]/_api/web/GetFileByServerRelativeUrl('[relative-path-to-file]')/$value`. Replace the site URL and relative path to file. Sample relative path to file as `/sites/site2/Shared Documents/TestBook.xlsx`.
227+
- **Authentication type:** Anonymous *(to use the Bearer token configured in copy activity source later)*
228+
- Dataset: choose the format you want. To copy file as-is, select "Binary" type.
229+
- Copy activity source:
230+
- **Request method**: GET
231+
- **Additional header**: use the following expression`@{concat('Authorization: Bearer ', activity('<Web-activity-name>').output.access_token)}`, which uses the Bearer token generated by the upstream Web activity as authorization header. Replace the Web activity name.
232+
- Configure the copy activity sink as usual.
233+
234+
## Lookup activity properties
235+
236+
To learn details about the properties, check [Lookup activity](control-flow-lookup-activity.md).
237+
238+
## Next steps
239+
240+
For a list of data stores that Copy Activity supports as sources and sinks in Azure Data Factory, see [Supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).

‎articles/data-factory/media/connector-sharepoint-online-list/sharepoint-online-copy-file-flow.png

4.79 KB
Loading

‎articles/data-factory/media/connector-sharepoint-online-list/sharepoint-online-grant-permission.png

14.6 KB
Loading

0 commit comments

Comments
 (0)