Skip to content

Commit 602a1cd

Browse files
authored
Merge pull request #194821 from Clare-Zheng82/0412-Add_dataworld_Twilio_connector_docs
Add data.world and Twilio connector docs
2 parents a58eb86 + 50a7701 commit 602a1cd

File tree

7 files changed

+229
-0
lines changed

7 files changed

+229
-0
lines changed

articles/data-factory/TOC.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -394,6 +394,8 @@ items:
394394
href: connector-concur.md
395395
- name: Couchbase
396396
href: connector-couchbase.md
397+
- name: data.world
398+
href: connector-dataworld.md
397399
- name: DB2
398400
href: connector-db2.md
399401
- name: Dataverse
@@ -552,6 +554,8 @@ items:
552554
href: connector-teamdesk.md
553555
- name: Teradata
554556
href: connector-teradata.md
557+
- name: Twilio
558+
href: connector-twilio.md
555559
- name: Vertica
556560
href: connector-vertica.md
557561
- name: Web Table
Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
title: Transform data in data.world (Preview)
3+
titleSuffix: Azure Data Factory & Azure Synapse
4+
description: Learn how to transform data in data.world (Preview) by using Data Factory or Azure Synapse Analytics.
5+
ms.author: jianleishen
6+
author: jianleishen
7+
ms.service: data-factory
8+
ms.subservice: data-movement
9+
ms.topic: conceptual
10+
ms.custom: synapse
11+
ms.date: 04/12/2022
12+
---
13+
14+
# Transform data in data.world (Preview) using Azure Data Factory or Synapse Analytics
15+
16+
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
17+
18+
This article outlines how to use Data Flow to transform data in data.world (Preview). To learn more, read the introductory article for [Azure Data Factory](introduction.md) or [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md).
19+
20+
> [!IMPORTANT]
21+
> This connector is currently in preview. You can try it out and give us feedback. If you want to take a dependency on preview connectors in your solution, please contact [Azure support](https://azure.microsoft.com/support/).
22+
23+
## Supported capabilities
24+
25+
This data.world connector is supported for the following activities:
26+
27+
- [Mapping data flow](concepts-data-flow-overview.md)
28+
29+
## Create a data.world linked service using UI
30+
31+
Use the following steps to create a data.world linked service in the Azure portal UI.
32+
33+
1. Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services, then select New:
34+
35+
# [Azure Data Factory](#tab/data-factory)
36+
37+
:::image type="content" source="media/doc-common-process/new-linked-service.png" alt-text="Screenshot of creating a new linked service with Azure Data Factory U I.":::
38+
39+
# [Azure Synapse](#tab/synapse-analytics)
40+
41+
:::image type="content" source="media/doc-common-process/new-linked-service-synapse.png" alt-text="Screenshot of creating a new linked service with Azure Synapse U I.":::
42+
43+
2. Search for data.world (Preview) and select the data.world (Preview) connector.
44+
45+
:::image type="content" source="media/connector-dataworld/dataworld-connector.png" alt-text="Screenshot showing selecting data.world connector.":::
46+
47+
3. Configure the service details, test the connection, and create the new linked service.
48+
49+
:::image type="content" source="media/connector-dataworld/configure-dataworld-linked-service.png" alt-text="Screenshot of configuration for data.world linked service.":::
50+
51+
## Connector configuration details
52+
53+
The following sections provide information about properties that are used to define Data Factory and Synapse pipeline entities specific to data.world.
54+
55+
## Linked service properties
56+
57+
The following properties are supported for the data.world linked service:
58+
59+
| Property | Description | Required |
60+
|:--- |:--- |:--- |
61+
| type | The type property must be set to **Dataworld**. |Yes |
62+
| apiToken | Specify an API token for the data.world. Mark this field as **SecureString** to store it securely. Or, you can [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). |Yes |
63+
64+
**Example:**
65+
66+
```json
67+
{
68+
"name": "DataworldLinkedService",
69+
"properties": {
70+
"type": "Dataworld",
71+
"typeProperties": {
72+
"apiToken": {
73+
"type": "SecureString",
74+
"value": "<API token>"
75+
}
76+
}
77+
}
78+
}
79+
```
80+
81+
## Mapping data flow properties
82+
83+
When transforming data in mapping data flow, you can read tables from data.world. For more information, see the [source transformation](data-flow-source.md) in mapping data flows. You can only use an [inline dataset](data-flow-source.md#inline-datasets) as source type.
84+
85+
86+
### Source transformation
87+
88+
The below table lists the properties supported by data.world source. You can edit these properties in the **Source options** tab.
89+
90+
| Name | Description | Required | Allowed values | Data flow script property |
91+
| ---- | ----------- | -------- | -------------- | ---------------- |
92+
| Dataset name| The ID of the dataset in data.world.| Yes | String | datasetId |
93+
| Table name | The ID of the table within the dataset in data.world. | No (if `query` is specified) | String | tableId |
94+
| Query | Enter a SQL query to fetch data from data.world. An example is `select * from MyTable`.| No (if `tableId` is specified)| String | query |
95+
| Owner | The owner of the dataset in data.world. | Yes | String | owner |
96+
97+
#### data.world source script example
98+
99+
When you use data.world as source type, the associated data flow script is:
100+
101+
```
102+
source(allowSchemaDrift: true,
103+
validateSchema: false,
104+
store: 'dataworld',
105+
format: 'rest',
106+
owner: 'owner1',
107+
datasetId: 'dataset1',
108+
tableId: 'MyTable') ~> DataworldSource
109+
```
110+
111+
## Next steps
112+
113+
For a list of data stores supported as sources and sinks by the copy activity, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
---
2+
title: Transform data in Twilio (Preview)
3+
titleSuffix: Azure Data Factory & Azure Synapse
4+
description: Learn how to transform data in Twilio (Preview) by using Data Factory or Azure Synapse Analytics.
5+
ms.author: jianleishen
6+
author: jianleishen
7+
ms.service: data-factory
8+
ms.subservice: data-movement
9+
ms.topic: conceptual
10+
ms.custom: synapse
11+
ms.date: 04/12/2022
12+
---
13+
14+
# Transform data in Twilio (Preview) using Azure Data Factory or Synapse Analytics
15+
16+
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
17+
18+
This article outlines how to use Data Flow to transform data in Twilio (Preview). To learn more, read the introductory article for [Azure Data Factory](introduction.md) or [Azure Synapse Analytics](../synapse-analytics/overview-what-is.md).
19+
20+
> [!IMPORTANT]
21+
> This connector is currently in preview. You can try it out and give us feedback. If you want to take a dependency on preview connectors in your solution, please contact [Azure support](https://azure.microsoft.com/support/).
22+
23+
## Supported capabilities
24+
25+
This Twilio connector is supported for the following activities:
26+
27+
- [Mapping data flow](concepts-data-flow-overview.md)
28+
29+
## Create a Twilio linked service using UI
30+
31+
Use the following steps to create a Twilio linked service in the Azure portal UI.
32+
33+
1. Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services, then select New:
34+
35+
# [Azure Data Factory](#tab/data-factory)
36+
37+
:::image type="content" source="media/doc-common-process/new-linked-service.png" alt-text="Screenshot of creating a new linked service with Azure Data Factory U I.":::
38+
39+
# [Azure Synapse](#tab/synapse-analytics)
40+
41+
:::image type="content" source="media/doc-common-process/new-linked-service-synapse.png" alt-text="Screenshot of creating a new linked service with Azure Synapse U I.":::
42+
43+
2. Search for Twilio (Preview) and select the Twilio (Preview) connector.
44+
45+
:::image type="content" source="media/connector-twilio/twilio-connector.png" alt-text="Screenshot showing selecting Twilio connector.":::
46+
47+
3. Configure the service details, test the connection, and create the new linked service.
48+
49+
:::image type="content" source="media/connector-twilio/configure-twilio-linked-service.png" alt-text="Screenshot of configuration for Twilio linked service.":::
50+
51+
## Connector configuration details
52+
53+
The following sections provide information about properties that are used to define Data Factory and Synapse pipeline entities specific to Twilio.
54+
55+
## Linked service properties
56+
57+
The following properties are supported for the Twilio linked service:
58+
59+
| Property | Description | Required |
60+
|:--- |:--- |:--- |
61+
| type | The type property must be set to **Twilio**. | Yes |
62+
| userName | The account SID of your Twilio account. | No |
63+
| password | The auth token of your Twilio account. Mark this field as **SecureString** to store it securely. Or, you can [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). |Yes |
64+
65+
**Example:**
66+
67+
```json
68+
{
69+
"name": "TwilioLinkedService",
70+
"properties": {
71+
"type": "Twilio",
72+
"typeProperties": {
73+
"userName": "<account SID>",
74+
"password": {
75+
"type": "SecureString",
76+
"value": "<auth token>"
77+
}
78+
}
79+
}
80+
}
81+
```
82+
83+
84+
85+
### Source transformation
86+
87+
When transforming data in mapping data flow, you can read resources from Twilio. For more information, see the [source transformation](data-flow-source.md) in mapping data flows. You can only use an [inline dataset](data-flow-source.md#inline-datasets) as source type.
88+
89+
The below table lists the properties supported by Twilio source. You can edit these properties in the **Source options** tab.
90+
91+
| Name | Description | Required | Allowed values | Data flow script property |
92+
| ---- | ----------- | -------- | -------------- | ---------------- |
93+
| Resource | The type of resources that data flow fetch from Twilio. | Yes | `Messages`<br>`Calls` | resource |
94+
| From | The phone number with country code, for example `+17755425856`. | No | String | from |
95+
| To | The phone number with country code, for example `+17755425856`. | No | String | to |
96+
97+
#### Twilio source script example
98+
99+
When you use Twilio as source type, the associated data flow script is:
100+
101+
```
102+
source(allowSchemaDrift: true,
103+
validateSchema: false,
104+
store: 'twilio',
105+
format: 'rest',
106+
resource: 'Messages',
107+
from: '+17755425856') ~> TwilioSource
108+
```
109+
110+
## Next steps
111+
112+
For a list of data stores supported as sources and sinks by the copy activity, see [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).
28.6 KB
Loading
18.5 KB
Loading
29.7 KB
Loading
16.8 KB
Loading

0 commit comments

Comments
 (0)