Skip to content

Commit 9485b33

Browse files
linda33wjlinda33wj
committed
Add SPO List connector
1 parent ee6f8fb commit 9485b33

File tree

6 files changed

+220
-8
lines changed

6 files changed

+220
-8
lines changed

‎articles/data-factory/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -370,6 +370,8 @@
370370
href: connector-servicenow.md
371371
- name: SFTP
372372
href: connector-sftp.md
373+
- name: SharePoint Online List
374+
href: connector-sharepoint-online-list.md
373375
- name: Shopify
374376
href: connector-shopify.md
375377
- name: Spark

‎articles/data-factory/connector-sharepoint-online-list.md

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
---
2+
title: Copy data from SharePoint Online List by using Azure Data Factory
3+
description: Learn how to copy data from SharePoint Online List to supported sink data stores by using a copy activity in an Azure Data Factory pipeline.
4+
services: data-factory
5+
documentationcenter: ''
6+
author: linda33wj
7+
manager: shwang
8+
ms.reviewer: douglasl
9+
10+
ms.service: data-factory
11+
ms.workload: data-services
12+
13+
14+
ms.topic: conceptual
15+
ms.date: 05/19/2020
16+
ms.author: jingwang
17+
18+
---
19+
# Copy data from SharePoint Online List by using Azure Data Factory
20+
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
21+
22+
This article outlines how to use Copy Activity in Azure Data Factory to copy data from SharePoint Online List. The article builds on [Copy Activity in Azure Data Factory](copy-activity-overview.md), which presents a general overview of Copy Activity.
23+
24+
## Supported capabilities
25+
26+
This SharePoint Online List connector is supported for the following activities:
27+
28+
- [Lookup activity](control-flow-lookup-activity.md)
29+
30+
You can copy data from SharePoint Online List to any supported sink data store. For a list of data stores that Copy Activity supports as sources and sinks, see [Supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).
31+
32+
Specifically, this SharePoint List Online connector uses service principal authentication and retrieves data via OData protocol.
33+
34+
## Prerequisites
35+
36+
The SharePoint List Online connector uses service principal authentication to connect to SharePoint. Follow these steps to set it up:
37+
38+
1. Register an application entity in Azure Active Directory (Azure AD) by following [Register your application with an Azure AD tenant](../storage/common/storage-auth-aad-app.md#register-your-application-with-an-azure-ad-tenant). Make note of the following values, which you use to define the linked service:
39+
40+
- Application ID
41+
- Application key
42+
- Tenant ID
43+
44+
2. Grant SharePoint Online site permission to your registered application:
45+
46+
> [!NOTE]
47+
>
48+
> This operation requires SharePoint Online site owner permission. You can find the owner by going to the site home page -> click the "X members" in the right corner -> check who has the "Owner" role.
49+
50+
1. 1. Open SharePoint site link e.g. `https://[tenant-name].sharepoint.com/sites/[site-name]/_layouts/15/appinv.aspx` (replace tenant and site name).
51+
2. Search the application ID you just registered in Step 1, fill the empty fields, and click "Create".
52+
1. App Domain: localhost.com
53+
2. Redirect URL: https://www.localhost.com
54+
3. Permission Request XML:
55+
3. ```xml
56+
<AppPermissionRequests AllowAppOnlyPolicy="true">
57+
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="Read"/>
58+
</AppPermissionRequests>
59+
```
60+
61+
2. ![sharepoint grant permission](media/connector-sharepoint-online-list/sharepoint-online-grant-permission.png)
62+
63+
3. 3. Click "Trust It" for this app.
64+
65+
## Get started
66+
67+
[!INCLUDE [data-factory-v2-connector-get-started](../../includes/data-factory-v2-connector-get-started.md)]
68+
69+
The following sections provide details about properties you can use to define Data Factory entities that are specific to SharePoint Online List connector.
70+
71+
## Linked service properties
72+
73+
The following properties are supported for an SharePoint Online List linked service:
74+
75+
| **Property** | **Description** | **Required** |
76+
| ------------------- | ------------------------------------------------------------ | ------------ |
77+
| type | The type property must be set to: **SharePointOnlineList**. | Yes |
78+
| siteUrl | The SharePoint Online site url. Eg. `https://[tenant-name].sharepoint.com/sites/[site-name]`. | Yes |
79+
| servicePrincipalId | The Application (client) ID of the application registered in Azure Active Directory. | Yes |
80+
| servicePrincipalKey | The application's key. Mark this field as a **SecureString** to store it securely in Data Factory, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | Yes |
81+
| tenantId | The tenant ID under which your application resides. | Yes |
82+
| connectVia | The [Integration Runtime](concepts-integration-runtime.md) to use to connect to the data store. Learn more from [Prerequisites](#prerequisites) section. If not specified, the default Azure Integration Runtime is used. | No |
83+
84+
**Example:**
85+
86+
```json
87+
{
88+
"name": "SharePointOnlineList",
89+
"properties": {
90+
"type": "SharePointOnlineList",
91+
"typeProperties": {
92+
"siteUrl": "https://[tenant-name].sharepoint.com/sites/[site-name]",
93+
"servicePrincipalId": "<service principal id>",
94+
"servicePrincipalKey": {
95+
"type": "SecureString",
96+
"value": "<service principal key>"
97+
},
98+
"tenantId": "<tenant GUID>"
99+
}
100+
}
101+
}
102+
```
103+
104+
## Dataset properties
105+
106+
For a full list of sections and properties that are available for defining datasets, see [Datasets and linked services](concepts-datasets-linked-services.md). The following section provides a list of the properties supported by the SAP table dataset.
107+
108+
| Property | Description | Required |
109+
|:--- |:--- |:--- |
110+
| type | The **type** property of the dataset must be set to **SharePointOnlineLResource**. | Yes |
111+
| listName | The name of the SharePoint Online List. | Yes |
112+
113+
**Example**
114+
115+
```json
116+
{
117+
"name": "SharePointOnlineListDataset",
118+
"properties":
119+
{
120+
"type": "SharePointOnlineListResource",
121+
"linkedServiceName": {
122+
"referenceName": "<SharePoint Online List linked service name>",
123+
"type": "LinkedServiceReference"
124+
},
125+
"typeProperties":
126+
{
127+
"listName": "MyList"
128+
}
129+
}
130+
}
131+
```
132+
133+
## Copy Activity properties
134+
135+
For a full list of sections and properties that are available for defining activities, see [Pipelines](concepts-pipelines-activities.md). The following section provides a list of the properties supported by the SharePoint Online List source.
136+
137+
### SharePoint Online List as source
138+
139+
To copy data from SharePoint Online List, the following properties are supported in the Copy Activity **source** section:
140+
141+
| Property | Description | Required |
142+
|:--- |:--- |:--- |
143+
| type | The **type** property of the Copy Activity source must be set to **SharePointOnlineListSource**. | Yes |
144+
| query | Custom OData query options for filtering data. Example: `"$top=10&$select=Title,Number"`. | No |
145+
| httpRequestTimeout | The timeout (in second) for the HTTP request to get a response. Default is 300 (5 minutes). | No |
146+
147+
**Example**
148+
149+
```json
150+
"activities":[
151+
{
152+
"name": "CopyFromSharePointOnlineList",
153+
"type": "Copy",
154+
"inputs": [
155+
{
156+
"referenceName": "<SharePoint Online List input dataset name>",
157+
"type": "DatasetReference"
158+
}
159+
],
160+
"outputs": [
161+
{
162+
"referenceName": "<output dataset name>",
163+
"type": "DatasetReference"
164+
}
165+
],
166+
"typeProperties": {
167+
"source": {
168+
"type": "SharePointOnlineListSource",
169+
"query": "<OData query e.g. $top=10&$select=Title,Number>"
170+
},
171+
"sink": {
172+
"type": "<sink type>"
173+
}
174+
}
175+
}
176+
]
177+
```
178+
179+
## Data type mapping for SharePoint Online List
180+
181+
When you copy data from SharePoint Online List, the following mappings are used between SharePoint Online List data types and Azure Data Factory interim data types.
182+
183+
| **SharePoint Online data type** | **OData data type** | **Azure Data Factory interim data type** |
184+
| ----------------------------------------------- | ---------------------------------------------------- | ---------------------------------------- |
185+
| Single line of text | Edm.String | String |
186+
| Multiple lines of text | Edm.String | String |
187+
| Choice (menu to choose from) | Edm.String | String |
188+
| Number (1, 1.0, 100) | Edm.Double | Double |
189+
| Currency ($, ¥, €) | Edm.Double | Double |
190+
| Date and Time | Edm.DateTime | DateTime |
191+
| Lookup (information already on this site) | Edm.Int32 | Int32 |
192+
| Yes/No (check box) | Edm.Boolean | Boolean |
193+
| Person or Group | Edm.Int32 | Int32 |
194+
| Hyperlink or Picture | Edm.String | String |
195+
| Calculated (calculation based on other columns) | Edm.String / Edm.Double / Edm.DateTime / Edm.Boolean | String / Double / DateTime / Boolean |
196+
| Attachment | Not supported | |
197+
| Task Outcome | Not supported | |
198+
| External Data | Not supported | |
199+
| Managed Metadata | Not supported | |
200+
201+
## Lookup activity properties
202+
203+
To learn details about the properties, check [Lookup activity](control-flow-lookup-activity.md).
204+
205+
## Next steps
206+
207+
For a list of data stores that Copy Activity supports as sources and sinks in Azure Data Factory, see [Supported data stores and formats](copy-activity-overview.md#supported-data-stores-and-formats).

‎articles/data-factory/media/connector-sharepoint-online-list/sharepoint-online-grant-permission.png

10.4 KB
Loading

0 commit comments

Comments
 (0)