Skip to content

Commit ed4d870

Browse files
PMEds28PMEds28
authored
Merge pull request #268352 from v-luckywang/0305-Update_postgresql_v2
 [New feature] - Update PostgreSQL v2 connector doc in ADF
2 parents 26675e6 + 5f43cb1 commit ed4d870

File tree

6 files changed

+379
-51
lines changed

6 files changed

+379
-51
lines changed

‎articles/data-factory/TOC.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -554,7 +554,11 @@ items:
554554
- name: Phoenix
555555
href: connector-phoenix.md
556556
- name: PostgreSQL
557-
href: connector-postgresql.md
557+
items:
558+
- name: PostgreSQL
559+
href: connector-postgresql.md
560+
- name: PostgreSQL (legacy)
561+
href: connector-postgresql-legacy.md
558562
- name: Presto
559563
href: connector-presto.md
560564
- name: Quickbase

‎articles/data-factory/connector-postgresql-legacy.md

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
---
2+
title: Copy data from PostgreSQL using legacy
3+
titleSuffix: Azure Data Factory & Azure Synapse
4+
description: Learn how to copy data from PostgreSQL to supported sink data stores by using a copy activity in a legacy Azure Data Factory or Synapse Analytics pipeline.
5+
author: jianleishen
6+
ms.service: data-factory
7+
ms.subservice: data-movement
8+
ms.custom: synapse
9+
ms.topic: conceptual
10+
ms.date: 03/07/2024
11+
ms.author: jianleishen
12+
---
13+
# Copy data from PostgreSQL using Azure Data Factory or Synapse Analytics (legacy)
14+
[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
15+
16+
This article outlines how to use the Copy Activity in Azure Data Factory and Synapse Analytics pipelines to copy data from a PostgreSQL database. It builds on the [copy activity overview](copy-activity-overview.md) article that presents a general overview of copy activity.
17+
18+
>[!IMPORTANT]
19+
>The service has released a new PostgreSQL connector which provides better native PostgreSQL support, refer to [PostgreSQL connector](connector-postgresql.md) article on details.
20+
21+
## Supported capabilities
22+
23+
This PostgreSQL connector is supported for the following capabilities:
24+
25+
| Supported capabilities|IR |
26+
|---------| --------|
27+
|[Copy activity](copy-activity-overview.md) (source/-)|① ②|
28+
|[Lookup activity](control-flow-lookup-activity.md)|① ②|
29+
30+
*① Azure integration runtime ② Self-hosted integration runtime*
31+
32+
For a list of data stores that are supported as sources/sinks by the copy activity, see the [Supported data stores](copy-activity-overview.md#supported-data-stores-and-formats) table.
33+
34+
Specifically, this PostgreSQL connector supports PostgreSQL **version 7.4 and above**.
35+
36+
## Prerequisites
37+
38+
[!INCLUDE [data-factory-v2-integration-runtime-requirements](includes/data-factory-v2-integration-runtime-requirements.md)]
39+
40+
The Integration Runtime provides a built-in PostgreSQL driver starting from version 3.7, therefore you don't need to manually install any driver.
41+
42+
## Getting started
43+
44+
[!INCLUDE [data-factory-v2-connector-get-started](includes/data-factory-v2-connector-get-started.md)]
45+
46+
## Create a linked service to PostgreSQL using UI
47+
48+
Use the following steps to create a linked service to PostgreSQL in the Azure portal UI.
49+
50+
1. Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services, then click New:
51+
52+
# [Azure Data Factory](#tab/data-factory)
53+
54+
:::image type="content" source="media/doc-common-process/new-linked-service.png" alt-text="Screenshot showing Create a new linked service with Azure Data Factory UI.":::
55+
56+
# [Azure Synapse](#tab/synapse-analytics)
57+
58+
:::image type="content" source="media/doc-common-process/new-linked-service-synapse.png" alt-text="Screenshot showing Create a new linked service with Azure Synapse UI.":::
59+
60+
2. Search for Postgre and select the PostgreSQL connector.
61+
62+
:::image type="content" source="media/connector-postgresql-legacy/postgresql-legacy-connector.png" alt-text="Screenshot showing Select the PostgreSQL connector.":::
63+
64+
1. Configure the service details, test the connection, and create the new linked service.
65+
66+
:::image type="content" source="media/connector-postgresql-legacy/configure-postgresql-legacy-linked-service.png" alt-text="Screenshot showing Configure a linked service to PostgreSQL.":::
67+
68+
## Connector configuration details
69+
70+
The following sections provide details about properties that are used to define Data Factory entities specific to PostgreSQL connector.
71+
72+
## Linked service properties
73+
74+
The following properties are supported for PostgreSQL linked service:
75+
76+
| Property | Description | Required |
77+
|:--- |:--- |:--- |
78+
| type | The type property must be set to: **PostgreSql** | Yes |
79+
| connectionString | An ODBC connection string to connect to Azure Database for PostgreSQL. <br/>You can also put password in Azure Key Vault and pull the `password` configuration out of the connection string. Refer to the following samples and [Store credentials in Azure Key Vault](store-credentials-in-key-vault.md) article with more details. | Yes |
80+
| connectVia | The [Integration Runtime](concepts-integration-runtime.md) to be used to connect to the data store. Learn more from [Prerequisites](#prerequisites) section. If not specified, it uses the default Azure Integration Runtime. |No |
81+
82+
A typical connection string is `Server=<server>;Database=<database>;Port=<port>;UID=<username>;Password=<Password>`. More properties you can set per your case:
83+
84+
| Property | Description | Options | Required |
85+
|:--- |:--- |:--- |:--- |
86+
| EncryptionMethod (EM)| The method the driver uses to encrypt data sent between the driver and the database server. E.g., `EncryptionMethod=<0/1/6>;`| 0 (No Encryption) **(Default)** / 1 (SSL) / 6 (RequestSSL) | No |
87+
| ValidateServerCertificate (VSC) | Determines whether the driver validates the certificate that is sent by the database server when SSL encryption is enabled (Encryption Method=1). E.g., `ValidateServerCertificate=<0/1>;`| 0 (Disabled) **(Default)** / 1 (Enabled) | No |
88+
89+
> [!NOTE]
90+
> In order to have full SSL verification via the ODBC connection when using the Self Hosted Integration Runtime you must use an ODBC type connection instead of the PostgreSQL connector explicitly, and complete the following configuration:
91+
>
92+
> 1. Set up the DSN on any SHIR servers.
93+
> 1. Put the proper certificate for PostgreSQL in C:\Windows\ServiceProfiles\DIAHostService\AppData\Roaming\postgresql\root.crt on the SHIR servers. This is where the ODBC driver looks > for the SSL cert to verify when it connects to the database.
94+
> 1. In your data factory connection, use an ODBC type connection, with your connection string pointing to the DSN you created on your SHIR servers.
95+
96+
**Example:**
97+
98+
```json
99+
{
100+
"name": "PostgreSqlLinkedService",
101+
"properties": {
102+
"type": "PostgreSql",
103+
"typeProperties": {
104+
"connectionString": "Server=<server>;Database=<database>;Port=<port>;UID=<username>;Password=<Password>"
105+
},
106+
"connectVia": {
107+
"referenceName": "<name of Integration Runtime>",
108+
"type": "IntegrationRuntimeReference"
109+
}
110+
}
111+
}
112+
```
113+
114+
**Example: store password in Azure Key Vault**
115+
116+
```json
117+
{
118+
"name": "PostgreSqlLinkedService",
119+
"properties": {
120+
"type": "PostgreSql",
121+
"typeProperties": {
122+
"connectionString": "Server=<server>;Database=<database>;Port=<port>;UID=<username>;",
123+
"password": { 
124+
"type": "AzureKeyVaultSecret", 
125+
"store": { 
126+
"referenceName": "<Azure Key Vault linked service name>", 
127+
"type": "LinkedServiceReference" 
128+
}, 
129+
"secretName": "<secretName>" 
130+
}
131+
},
132+
"connectVia": {
133+
"referenceName": "<name of Integration Runtime>",
134+
"type": "IntegrationRuntimeReference"
135+
}
136+
}
137+
}
138+
```
139+
140+
If you were using PostgreSQL linked service with the following payload, it is still supported as-is, while you are suggested to use the new one going forward.
141+
142+
**Previous payload:**
143+
144+
```json
145+
{
146+
"name": "PostgreSqlLinkedService",
147+
"properties": {
148+
"type": "PostgreSql",
149+
"typeProperties": {
150+
"server": "<server>",
151+
"database": "<database>",
152+
"username": "<username>",
153+
"password": {
154+
"type": "SecureString",
155+
"value": "<password>"
156+
}
157+
},
158+
"connectVia": {
159+
"referenceName": "<name of Integration Runtime>",
160+
"type": "IntegrationRuntimeReference"
161+
}
162+
}
163+
}
164+
```
165+
166+
## Dataset properties
167+
168+
For a full list of sections and properties available for defining datasets, see the [datasets](concepts-datasets-linked-services.md) article. This section provides a list of properties supported by PostgreSQL dataset.
169+
170+
To copy data from PostgreSQL, the following properties are supported:
171+
172+
| Property | Description | Required |
173+
|:--- |:--- |:--- |
174+
| type | The type property of the dataset must be set to: **PostgreSqlTable** | Yes |
175+
| schema | Name of the schema. |No (if "query" in activity source is specified) |
176+
| table | Name of the table. |No (if "query" in activity source is specified) |
177+
| tableName | Name of the table with schema. This property is supported for backward compatibility. Use `schema` and `table` for new workload. | No (if "query" in activity source is specified) |
178+
179+
**Example**
180+
181+
```json
182+
{
183+
"name": "PostgreSQLDataset",
184+
"properties":
185+
{
186+
"type": "PostgreSqlTable",
187+
"typeProperties": {},
188+
"schema": [],
189+
"linkedServiceName": {
190+
"referenceName": "<PostgreSQL linked service name>",
191+
"type": "LinkedServiceReference"
192+
}
193+
}
194+
}
195+
```
196+
197+
If you were using `RelationalTable` typed dataset, it's still supported as-is, while you are suggested to use the new one going forward.
198+
199+
## Copy activity properties
200+
201+
For a full list of sections and properties available for defining activities, see the [Pipelines](concepts-pipelines-activities.md) article. This section provides a list of properties supported by PostgreSQL source.
202+
203+
### PostgreSQL as source
204+
205+
To copy data from PostgreSQL, the following properties are supported in the copy activity **source** section:
206+
207+
| Property | Description | Required |
208+
|:--- |:--- |:--- |
209+
| type | The type property of the copy activity source must be set to: **PostgreSqlSource** | Yes |
210+
| query | Use the custom SQL query to read data. For example: `"query": "SELECT * FROM \"MySchema\".\"MyTable\""`. | No (if "tableName" in dataset is specified) |
211+
212+
> [!NOTE]
213+
> Schema and table names are case-sensitive. Enclose them in `""` (double quotes) in the query.
214+
215+
**Example:**
216+
217+
```json
218+
"activities":[
219+
{
220+
"name": "CopyFromPostgreSQL",
221+
"type": "Copy",
222+
"inputs": [
223+
{
224+
"referenceName": "<PostgreSQL input dataset name>",
225+
"type": "DatasetReference"
226+
}
227+
],
228+
"outputs": [
229+
{
230+
"referenceName": "<output dataset name>",
231+
"type": "DatasetReference"
232+
}
233+
],
234+
"typeProperties": {
235+
"source": {
236+
"type": "PostgreSqlSource",
237+
"query": "SELECT * FROM \"MySchema\".\"MyTable\""
238+
},
239+
"sink": {
240+
"type": "<sink type>"
241+
}
242+
}
243+
}
244+
]
245+
```
246+
247+
If you were using `RelationalSource` typed source, it is still supported as-is, while you are suggested to use the new one going forward.
248+
249+
## Lookup activity properties
250+
251+
To learn details about the properties, check [Lookup activity](control-flow-lookup-activity.md).
252+
253+
254+
## Related content
255+
For a list of data stores supported as sources and sinks by the copy activity, see [supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).

0 commit comments

Comments
 (0)