Skip to content

Commit c9cca68

Browse files
WilliamDAssafMSFTWilliamDAssafMSFT
committed
Merge branch 'master' of https://github.com/periclesrocha/azure-docs-pr into 20241227-external-tables-tutorials
2 parents 849af37 + d4cf873 commit c9cca68

File tree

3 files changed

+373
-0
lines changed

3 files changed

+373
-0
lines changed

‎articles/synapse-analytics/sql/tutorial-external-tables-using-managed-identity.md

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
---
2+
title: 'Tutorial: Loading external data using a managed identity'
3+
description: This tutorial shows how to connect to external data for queries or ingestion using a managed identity.
4+
author: periclesrocha
5+
ms.service: azure-synapse-analytics
6+
ms.topic: tutorial
7+
ms.subservice: sql
8+
ms.date: 01/04/2025
9+
ms.custom:
10+
ms.author: procha
11+
ms.reviewer: WilliamDAssafMSFT
12+
---
13+
14+
# Tutorial: Loading external data using a managed identity
15+
16+
This article explains how to create external tables or ingest data from Azure Data Lake Storage (ADLS) Gen2 accounts using a managed identity.
17+
18+
## Prerequisites:
19+
20+
The following resources are required to complete this tutorial:
21+
22+
* An Azure Data Lake Storage Gen2 (ADLS Gen2) account
23+
* An Azure Synapse Analytics workspace and a dedicated SQL Pool
24+
25+
## Give the workspace identity access to the storage account
26+
27+
Each Azure Synapse Analytics workspace automatically creates a managed identity that helps you configure secure access to external data from your workspace. To learn more about managed identities for Azure Synapse Analytics, visit [Managed service identity for Azure Synapse Analytics - Azure Synapse | Microsoft Learn](https://learn.microsoft.com/azure/synapse-analytics/synapse-service-identity).
28+
29+
To enable your managed identity to access data on ADLS Gen2 accounts, you need to give your identity access to the source account. To grant proper permissions, follow these steps:
30+
31+
1. In the Azure portal, find your storage account.
32+
2. Select **Data storage -> Containers**, and navigate to the folder where the source data the external table needs access to is.
33+
3. Select **Access control (IAM)**.
34+
4. Select **Add -> Add role assignment**.
35+
5. In the list of job function roles, select **Storage Blob Data Contributor** and select **Next**.
36+
6. In the Add role assignment page, select **+ Select members**. The Select members pane opens in the right-hand corner.
37+
7. Type the name of your workspace identity. The workspace identity is the same as your workspace name. When displayed, pick your workspace identity and chose **Select**.
38+
8. In the **Add role assignment** page, make sure the list of Members include your desired Entra ID account. Once verified, select **Review + assign**.
39+
9. In the confirmation page, review the changes and select **Review + assign**.
40+
41+
Your workspace identity is now a member of the Storage Blob Data Contributor role and has access to the source folder.
42+
43+
Note: these steps also apply to secure ADLS Gen2 accounts that are configured to restrict public access. To learn more about securing your ADLS Gen2 account, visit [Configure Azure Storage firewalls and virtual networks | Microsoft Learn](https://learn.microsoft.com/azure/storage/common/storage-network-security?tabs=azure-portal).
44+
45+
## Ingest data using COPY INTO
46+
47+
The COPY INTO statement provides flexible, high-throughput data ingestion into your tables, and is the primary strategy to ingest data into your dedicated SQL Pool tables. It allows users to ingest data from external locations without having to create any of the extra database objects that are required for external tables.
48+
49+
To run the COPY INTO statement using a workspace managed identity for authentication, use the following command:
50+
51+
```sql
52+
COPY INTO <TableName>
53+
FROM 'https://<AccountName>.dfs.core.windows.net/<Container>/<Folder>/ '
54+
WITH
55+
(
56+
CREDENTIAL = (IDENTITY = 'Managed Identity'),
57+
[<CopyIntoOptions>]
58+
)
59+
```
60+
61+
Where:
62+
63+
* \<TableName> is the name of the table you'll ingest data into
64+
* \<AccountName> is your ADLS Gen2 account name
65+
* \<Container> is the name of the container within your storage account where the source data is stored
66+
* \<Folder> is the folder (or path with subfolders) where the source data is stored within your container. You can also provide a file name if pointing directly to a single file.
67+
* \<CopyIntoOptions> is the list of any other options you wish to provide to the COPY INTO statement.
68+
69+
To learn more and explore the full syntax of COPY INTO, refer to <https://learn.microsoft.com/sql/t-sql/statements/copy-into-transact-sql?view=azure-sqldw-latest>.
70+
71+
## Query data on ADLS Gen2 using external tables
72+
73+
External tables allow users to query data from ADLS Gen2 accounts without the need to ingest data first. Users can create an external table which points to files on an ADLS Gen2 container, and query it just like a regular user table.
74+
75+
The following steps describe the process to create a new external table pointing to data on ADLS Gen2, using a managed identity for authentication.
76+
77+
### Create the required database objects
78+
79+
External tables require the following objects to be created:
80+
81+
1. A database master key that encrypts the database scoped credential’s secret
82+
2. A database scoped credential that uses your workspace identity.
83+
3. An external data source that points to the source folder.
84+
4. An external file format that defines the format of the source files.
85+
5. An external table definition that is used for queries.
86+
87+
To follow these steps, you'll need to use the SQL editor in the Azure Synapse Workspace, or your preferred SQL client connected to your dedicated SQL Pool. Let’s look at these steps in detail.
88+
89+
#### Create the database master key
90+
91+
The database master key is a symmetric key used to protect the private keys of certificates and asymmetric keys that are present in the database and secrets in database scoped credentials. If there's already a master key in the database, you don't need to create a new one.
92+
93+
To create a master key, use the following command:
94+
95+
```sql
96+
-- Replace <Secure Password Here> with a secure password
97+
CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<Secure Password Here>'
98+
```
99+
100+
Where:
101+
102+
* \<Secure Password Here> should be replaced with a strong password. This password is used to encrypt the master key in the database
103+
104+
To learn more about the database master key, refer to <https://learn.microsoft.com/sql/t-sql/statements/create-master-key-transact-sql?view=azure-sqldw-latest>.
105+
106+
#### Create the database scoped credential
107+
108+
A database scoped credential uses your workspace identity and is needed to access to the external location anytime the external table requires access to the source data.
109+
110+
To create the database scoped credential, use the following command:
111+
112+
```sql
113+
CREATE DATABASE SCOPED CREDENTIAL <CredentialName> WITH IDENTITY = 'Managed Service Identity'
114+
```
115+
116+
Where:
117+
118+
* \<CredentialName> should be replaced with the name you would like to use for your database scoped credential
119+
120+
To learn more about database scoped credentials, refer to <https://learn.microsoft.com/sql/t-sql/statements/create-database-scoped-credential-transact-sql?view=azure-sqldw-latest>.
121+
122+
#### Create the external data source
123+
124+
The next step is to create an external data source that specifies where the source data used by the external table resides.
125+
126+
To create the external data source, use the following command:
127+
128+
```sql
129+
CREATE EXTERNAL DATA SOURCE <ExternalDataSourceName>
130+
WITH (
131+
TYPE = hadoop,
132+
LOCATION = 'abfss://<Container>@<AccountName>.dfs.core.windows.net/<Folder>/,
133+
CREDENTIAL = <CredentialName>
134+
)
135+
```
136+
137+
Where:
138+
139+
* \<ExternalDataSourceName> is the name you want to use for your external data source
140+
* \<AccountName> is your ADLS Gen2 account name
141+
* \<Container> is the name of the container within your storage account where the source data is stored
142+
* \<Folder> is the folder (or path with subfolders) where the source data is stored within your container. You can also provide a file name if pointing directly to a single file.
143+
* \<Credential> is the name of the database scoped credential you created in step b)
144+
145+
To learn more about external data sources, refer to <https://learn.microsoft.com/sql/t-sql/statements/create-external-data-source-transact-sql?view=azure-sqldw-latest&tabs=dedicated>.
146+
147+
#### Create the external file format
148+
149+
The next step is to create the external file format. It specifies the actual layout of the data referenced by the external table.
150+
151+
To create the external file format, use the following command:
152+
153+
```sql
154+
CREATE EXTERNAL FILE FORMAT <FileFormatName>
155+
WITH (
156+
FORMAT_TYPE = DELIMITEDTEXT,
157+
FORMAT_OPTIONS (
158+
FIELD_TERMINATOR = ',',
159+
STRING_DELIMITER = '"',
160+
FIRST_ROW = 2,
161+
USE_TYPE_DEFAULT = True
162+
)
163+
)
164+
```
165+
166+
Where:
167+
168+
* \<FileFormatName> is the name you want to use for your external file format
169+
170+
In this example, adjust parameters such as FIELD_TERMINATOR, STRING_DELIMITER, FIRST_ROW and others as needed in accordance with your source data. For more formatting options and to learn more about EXTERNAL FILE FORMAT, visit <https://learn.microsoft.com/sql/t-sql/statements/create-external-file-format-transact-sql?view=azure-sqldw-latest&tabs=delimited>.
171+
172+
#### Create the external table
173+
174+
Now that all the necessary objects that hold the metadata to securely access external data are created, it's time to create the external table. To create the external table, use the following command:
175+
176+
```sql
177+
-- Adjust the table name and columns to your desired name and external table schema
178+
CREATE EXTERNAL TABLE <ExternalTableName> (
179+
Col1 INT,
180+
Col2 NVARCHAR(100),
181+
Col4 INT
182+
)
183+
WITH
184+
(
185+
LOCATION = '<Path>',
186+
DATA_SOURCE = <ExternalDataSourceName>,
187+
FILE_FORMAT = <FileFormatName>
188+
)
189+
```
190+
191+
Where:
192+
193+
* \<ExternalTableName> is the name you want to use for your external table
194+
* \<Path> is the relative path of the source data from the location specified in the external data source on step c)
195+
* \<ExternalDataSourceName> is the name of the external data source you created previously c)
196+
* \<FileFormatName> is the name of the external file format you created in step d)
197+
198+
Make sure to adjust the table name and schema to the desired name and the schema of the data in your source files.
199+
200+
At this point, all the metadata required to access the external table are created. To test your external table, use a query such as the following one to validate your work:
201+
202+
```sql
203+
SELECT TOP 10 Col1, Col2 FROM <ExternalTableName>
204+
```
205+
206+
If everything was configured properly, you should see the data from your source data as a result of this query.
207+
208+
To learn more and explore the full syntax of EXTERNAL TABLE, refer to <https://learn.microsoft.com/sql/t-sql/statements/create-external-table-transact-sql?view=azure-sqldw-latest&tabs=dedicated>.

‎articles/synapse-analytics/sql/tutorial-load-data-using-entra-id.md

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
---
2+
title: 'Tutorial: load data using Entra ID'
3+
description: This tutorial shows how to connect to external data for queries or ingestion using Entra ID passthrough
4+
author: periclesrocha
5+
ms.service: azure-synapse-analytics
6+
ms.topic: tutorial
7+
ms.subservice: sql
8+
ms.date: 01/04/2025
9+
ms.custom:
10+
ms.author: periclesrocha
11+
ms.reviewer: WilliamDAssafMSFT
12+
---
13+
14+
# Tutorial: Loading external data using Entra ID
15+
16+
This article explains how to create external tables using Entra ID passthrough.
17+
18+
## Prerequisites:
19+
20+
The following resources are required to complete this tutorial:
21+
22+
* An Azure Synapse Analytics workspace and a dedicated SQL Pool
23+
24+
## Give the Entra ID account access to the storage account
25+
26+
This example uses an Entra ID account (or group) to authenticate to the source data.
27+
28+
To enable access to data on Azure Data Lake Storage (ADLS) Gen2 accounts, you need to give your Entra ID account (or group) access to the source account. To grant the proper permissions, follow these steps:
29+
30+
1. In the Azure portal, find your storage account.
31+
2. Select **Data storage -> Containers**, and navigate to the folder where the source data the external table needs access to is.
32+
3. Select **Access control (IAM)**.
33+
4. Select **Add -> Add role assignment**.
34+
5. In the list of job function roles, select **Storage Blob Data Reader** and select **Next**.
35+
6. In the Add role assignment page, select **+ Select members**. The **Select members** pane opens in the right-hand corner.
36+
7. Type the name of the desired Entra ID account. When displayed, pick your desired Entra ID account and chose **Select**.
37+
8. In the the **Add role assignment** page, make sure the list of Members include your desired Entra ID account. Once verified, select **Review + assign**.
38+
9. In the confirmation page, review the changes and select **Review + assign**.
39+
40+
The Entra ID account or group is now a member of the Storage Blob Data Reader role and has access to the source folder.
41+
42+
## Ingest data using COPY INTO
43+
44+
The COPY INTO statement provides flexible, high-throughput data ingestion into your tables, and is the primary strategy to ingest data into your dedicated SQL Pool tables. It allows users to ingest data from external locations without having to create any of the extra database objects that are required for external tables.
45+
46+
The COPY INTO statement uses the CREDENTIAL argument to specify the authentication mechanism used to connect to the source account. However, when authenticating using Microsoft Entra ID or to a public storage account, CREDENTIAL doesn't need to be specified. Therefore, to run the COPY INTO statement using a workspace managed identity for authentication, use the following command:
47+
48+
```sql
49+
COPY INTO <TableName>
50+
FROM 'https://<AccountName>.dfs.core.windows.net/<Container>/<Folder>/ '
51+
WITH
52+
(
53+
[<CopyIntoOptions>]
54+
)
55+
```
56+
57+
Where:
58+
59+
* \<TableName> is the name of the table to ingest data into
60+
* \<AccountName> is your ADLS Gen2 account name
61+
* \<Container> is the name of the container within your storage account where the source data is stored
62+
* \<Folder> is the folder (or path with subfolders) where the source data is stored within your container. You can also provide a file name if pointing directly to a single file.
63+
* \<CopyIntoOptions> is the list of any other options you wish to provide to the COPY INTO statement.
64+
65+
To learn more and explore the full syntax of COPY INTO, refer to <https://learn.microsoft.com/sql/t-sql/statements/copy-into-transact-sql?view=azure-sqldw-latest>.
66+
67+
## Create the required database objects
68+
69+
External tables require the following objects to be created:
70+
71+
1. An external data source that points to the source folder.
72+
2. An external file format that defines the format of the source files.
73+
3. An external table definition that is used for queries.
74+
75+
To follow these steps, you need to use the SQL editor in the Azure Synapse Workspace, or your preferred SQL client connected to your dedicated SQL Pool. Let’s look at these steps in detail.
76+
77+
### Create the external data source
78+
79+
The next step is to create an external data source that specifies where the source data used by the external table resides.
80+
81+
To create the external data source, use the following command:
82+
83+
```sql
84+
CREATE EXTERNAL DATA SOURCE <ExternalDataSourceName>
85+
WITH (
86+
TYPE = hadoop,
87+
LOCATION = 'abfss://<Container>@<AccountName>.dfs.core.windows.net/<Folder>/
88+
)
89+
```
90+
91+
Where:
92+
93+
* \<ExternalDataSourceName> is the name you want to use for your external data source
94+
* \<AccountName> is your ADLS Gen2 account name
95+
* \<Container> is the name of the container within your storage account where the source data is stored
96+
* \<Folder> is the folder (or path with subfolders) where the source data is stored within your container
97+
98+
To learn more about external data sources, refer to <https://learn.microsoft.com/sql/t-sql/statements/create-external-data-source-transact-sql?view=azure-sqldw-latest&tabs=dedicated>.
99+
100+
### Create the external file format
101+
102+
The next step is to create the external file format. It specifies the actual layout of the data referenced by the external table.
103+
104+
To create the external file format, use the following command:
105+
106+
```sql
107+
CREATE EXTERNAL FILE FORMAT <FileFormatName>
108+
WITH (
109+
FORMAT_TYPE = DELIMITEDTEXT,
110+
FORMAT_OPTIONS (
111+
FIELD_TERMINATOR = ',',
112+
STRING_DELIMITER = '"',
113+
FIRST_ROW = 2,
114+
USE_TYPE_DEFAULT = True
115+
)
116+
)
117+
```
118+
119+
Where:
120+
121+
* \<FileFormatName> is the name you want to use for your external file format
122+
123+
In this example, adjust parameters such as FIELD_TERMINATOR, STRING_DELIMITER, FIRST_ROW and others as needed in accordance with your source data. For more formatting options and to learn more about EXTERNAL FILE FORMAT, visit <https://learn.microsoft.com/sql/t-sql/statements/create-external-file-format-transact-sql?view=azure-sqldw-latest&tabs=delimited>.
124+
125+
### Create the external table
126+
127+
Now that the necessary objects that hold the metadata to securely access external data are created, it's time to create the external table. To create the external table, use the following command:
128+
129+
```sql
130+
-- Adjust the table name and columns to your desired name and external table schema
131+
CREATE EXTERNAL TABLE <ExternalTableName> (
132+
Col1 INT,
133+
Col2 NVARCHAR(100),
134+
Col4 INT
135+
)
136+
WITH
137+
(
138+
LOCATION = '<Path>',
139+
DATA_SOURCE = <ExternalDataSourceName>,
140+
FILE_FORMAT = <FileFormatName>
141+
)
142+
```
143+
144+
Where:
145+
146+
* \<ExternalTableName> is the name you want to use for your external table
147+
* \<Path> is the relative path of the source data from the location specified in the external data source on step c)
148+
* \<ExternalDataSourceName> is the name of the external data source you created previously c)
149+
* \<FileFormatName> is the name of the external file format you created in step d)
150+
151+
Make sure to adjust the table name and schema to the desired name and the schema of the data in your source files.
152+
153+
At this point, all the metadata required to access the external table are created. To test your external table, use a query such as the following one to validate your work:
154+
155+
```sql
156+
SELECT TOP 10 Col1, Col2 FROM <ExternalTableName>
157+
```
158+
159+
If everything was configured properly, you should see the data from your source data as a result of this query.
160+
161+
To learn more and explore the full syntax of EXTERNAL TABLE, refer to <https://learn.microsoft.com/sql/t-sql/statements/create-external-table-transact-sql?view=azure-sqldw-latest&tabs=dedicated>.

‎articles/synapse-analytics/toc.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -320,6 +320,10 @@ items:
320320
href: ./sql/tutorial-logical-data-warehouse.md
321321
- name: Power BI Desktop
322322
href: ./sql/tutorial-connect-power-bi-desktop.md
323+
- name: Loading external data using Entra ID
324+
href: ./sql/tutorial-load-data-using-entra-id.md
325+
- name: Loading external data using a Managed Identity
326+
href: ./sql/tutorial-external-tables-using-managed-identity.md
323327
- name: Concepts
324328
items:
325329
- name: Back up and restore

0 commit comments

Comments
 (0)