You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In order to have access to scan the data source, an authentication method in the ADLS Gen2 Storage account needs to be configured.
82
-
The following options are supported:
81
+
Your Azure network may allow for communications between your Azure resources, but if you've set up firewalls, private endpoints, or virtual networks within Azure, you'll need to follow one of these configurations below.
83
82
84
-
> [!Note]
85
-
> If you have firewall enabled for the storage account, you must use managed identity authentication method when setting up a scan.
83
+
|Networking constraints |Integration runtime type |Available credential types |
84
+
|---------|---------|---------|
85
+
|No private endpoints or firewalls | Azure IR | Managed identity (Recommended), service principal, or account key|
86
+
|Firewall enabled but no private endpoints| Azure IR | Managed identity |
87
+
|Private endpoints enabled |*Self-Hosted IR | Service principal, account key|
86
88
87
-
***System-assigned managed identity (Recommended)** - As soon as the Microsoft Purview Account is created, a system-assigned managed identity (SAMI) is created automatically in Azure AD tenant. Depending on the type of resource, specific RBAC role assignments are required for the Microsoft Purview system-assigned managed identity (SAMI) to perform the scans.
88
-
89
-
***User-assigned managed identity** (preview) - Similar to a system managed identity, a user-assigned managed identity (UAMI) is a credential resource that can be used to allow Microsoft Purview to authenticate against Azure Active Directory. For more information, you can see our [User-assigned managed identity guide](manage-credentials.md#create-a-user-assigned-managed-identity).
89
+
*To use a self-hosted integration runtime, you'll first need to [create one](manage-integration-runtimes.md) and confirm your [network settings for Microsoft Purview](catalog-private-link.md)
90
90
91
-
***Account Key** - Secrets can be created inside an Azure Key Vault to store credentials in order to enable access for Microsoft Purview to scan data sources securely using the secrets. A secret can be a storage account key, SQL login password, or a password.
92
-
93
-
> [!Note]
94
-
> If you use this option, you need to deploy an _Azure key vault_ resource in your subscription and assign _Microsoft Purview account’s_ SAMI with required access permission to secrets inside _Azure key vault_.
91
+
# [System or user assigned managed identity](#tab/MI)
95
92
96
-
***Service Principal** - In this method, you can create a new or use an existing service principal in your Azure Active Directory tenant.
93
+
#### Using a system or user assigned managed identity for scanning
97
94
98
-
### Authentication for a scan
95
+
There are two types of managed identity you can use:
99
96
100
-
# [System or user assigned managed identity](#tab/MI)
97
+
***System-assigned managed identity (Recommended)** - As soon as the Microsoft Purview Account is created, a system-assigned managed identity (SAMI) is created automatically in Azure AD tenant. Depending on the type of resource, specific RBAC role assignments are required for the Microsoft Purview system-assigned managed identity (SAMI) to perform the scans.
101
98
102
-
#### Using a system or user assigned managed identity for scanning
99
+
***User-assigned managed identity** (preview) - Similar to a system managed identity, a user-assigned managed identity (UAMI) is a credential resource that can be used to allow Microsoft Purview to authenticate against Azure Active Directory. For more information, you can see our [User-assigned managed identity guide](manage-credentials.md#create-a-user-assigned-managed-identity).
103
100
104
101
It's important to give your Microsoft Purview account or user-assigned managed identity (UAMI) the permission to scan the ADLS Gen2 data source. You can add your Microsoft Purview account's system-assigned managed identity (which has the same name as your Microsoft Purview account) or UAMI at the Subscription, Resource Group, or Resource level, depending on what level scan permissions are needed.
105
102
@@ -141,6 +138,9 @@ It's important to give your Microsoft Purview account or user-assigned managed i
141
138
142
139
#### Using Account Key for scanning
143
140
141
+
> [!Note]
142
+
> If you use this option, you need to deploy an _Azure key vault_ resource in your subscription and [assign _Microsoft Purview account’s_ System Assigned Managed Identity (SAMI) required access permission to secrets inside _Azure key vault_.](manage-credentials.md#microsoft-purview-permissions-on-the-azure-key-vault)
143
+
144
144
When authentication method selected is **Account Key**, you need to get your access key and store in the key vault:
145
145
146
146
1. Navigate to your ADLS Gen2 storage account
@@ -225,15 +225,19 @@ It's important to give your service principal the permission to scan the ADLS Ge
225
225
226
226
#### If using Account Key
227
227
228
-
1. Provide a **Name** for the scan, choose the appropriate collection for the scan, and select **Authentication method** as _Account Key_
228
+
1. Provide a **Name** for the scan, select the Azure IR or your Self-Hosted IR depending on your configuration, choose the appropriate collection for the scan, and select **+ New** under credential.
229
+
230
+
1. Select **Account Key** as the authentication method, then select the appropriate **Key vault connection**, and provide the name of the secret you used to store the account key. Then select **Create**
229
231
230
232
:::image type="content" source="media/register-scan-adls-gen2/register-adls-gen2-acct-key.png" alt-text="Screenshot that shows the Account Key option for scanning":::
231
233
234
+
1. Select **Test connection**. On a successful connection, select **Continue**
235
+
232
236
# [Service Principal](#tab/SP)
233
237
234
238
#### If using Service Principal
235
239
236
-
1. Provide a **Name** for the scan, choose the appropriate collection for the scan, and select the **+ New** under **Credential**
240
+
1. Provide a **Name** for the scan, select the Azure IR or your Self-Hosted IR depending on your configuration, choose the appropriate collection for the scan, and select the **+ New** under **Credential**
237
241
238
242
:::image type="content" source="media/register-scan-adls-gen2/register-adls-gen2-sp-option.png" alt-text="Screenshot that shows the option for service principal to enable scanning":::
Copy file name to clipboardExpand all lines: articles/purview/register-scan-azure-blob-storage-source.md
+23-26Lines changed: 23 additions & 26 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,23 +25,23 @@ For file types such as csv, tsv, psv, ssv, the schema is extracted when the foll
25
25
26
26
* First row values are non-empty
27
27
* First row values are unique
28
-
* First row values are not a date or a number
28
+
* First row values aren't a date or a number
29
29
30
30
## Prerequisites
31
31
32
32
* An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).
33
33
34
34
* An active [Microsoft Purview account](create-catalog-portal.md).
35
35
36
-
* You will need to be a Data Source Administrator and Data Reader to register a source and manage it in the Microsoft Purview governance portal. See our [Microsoft Purview Permissions page](catalog-permissions.md) for details.
36
+
* You'll need to be a Data Source Administrator and Data Reader to register a source and manage it in the Microsoft Purview governance portal. See our [Microsoft Purview Permissions page](catalog-permissions.md) for details.
37
37
38
38
\** Lineage is supported if dataset is used as a source/sink in [Data Factory Copy activity](how-to-link-azure-data-factory.md)
39
39
40
40
## Register
41
41
This section will enable you to register the Azure Blob storage account for scan and data share in Purview.
42
42
43
43
### Prerequisites for register
44
-
* You will need to be a Data Source Admin and one of the other Purview roles (e.g. Data Reader or Data Share Contributor) to register a source and manage it in the Microsoft Purview governance portal. See our [Microsoft Purview Permissions page](catalog-permissions.md) for details.
44
+
* You'll need to be a Data Source Admin and one of the other Purview roles (for example, Data Reader or Data Share Contributor) to register a source and manage it in the Microsoft Purview governance portal. See our [Microsoft Purview Permissions page](catalog-permissions.md) for details.
45
45
46
46
### Steps to register
47
47
@@ -81,31 +81,28 @@ For file types such as csv, tsv, psv, ssv, the schema is extracted when the foll
81
81
82
82
* First row values are non-empty
83
83
* First row values are unique
84
-
* First row values are neither a date nor a number
84
+
* First row values are not a date or a number
85
85
86
86
### Authentication for a scan
87
87
88
-
In order to have access to scan the data source, an authentication method in the Azure Blob Storage account needs to be configured.
89
-
90
-
The following options are supported:
91
-
92
-
> [!Note]
93
-
> If you have firewall enabled for the storage account, you must use managed identity authentication method when setting up a scan.
94
-
95
-
-**System-assigned managed identity (Recommended)** - As soon as the Microsoft Purview Account is created, a system-assigned managed identity (SAMI) is created automatically in Azure AD tenant. Depending on the type of resource, specific RBAC role assignments are required for the Microsoft Purview SAMI to perform the scans.
88
+
Your Azure network may allow for communications between your Azure resources, but if you've set up firewalls, private endpoints, or virtual networks within Azure, you'll need to follow one of these configurations below.
96
89
97
-
-**User-assigned managed identity** (preview) - Similar to a system-managed identity, a user-assigned managed identity (UAMI) is a credential resource that can be used to allow Microsoft Purview to authenticate against Azure Active Directory. For more information, you can see our [User-assigned managed identity guide](manage-credentials.md#create-a-user-assigned-managed-identity).
90
+
|Networking constraints |Integration runtime type |Available credential types |
91
+
|---------|---------|---------|
92
+
|No private endpoints or firewalls | Azure IR | Managed identity (Recommended), service principal, or account key|
93
+
|Firewall enabled but no private endpoints| Azure IR | Managed identity |
94
+
|Private endpoints enabled |*Self-Hosted IR | Service principal, account key|
98
95
99
-
-**Account Key** - Secrets can be created inside an Azure Key Vault to store credentials in order to enable access for Microsoft Purview to scan data sources securely using the secrets. A secret can be a storage account key, SQL login password, or a password.
96
+
*To use a self-hosted integration runtime, you'll first need to [create one](manage-integration-runtimes.md) and confirm your [network settings for Microsoft Purview](catalog-private-link.md)
100
97
101
-
> [!Note]
102
-
> If you use this option, you need to deploy an _Azure key vault_ resource in your subscription and assign _Microsoft Purview account’s_ SAMI with required access permission to secrets inside _Azure key vault_.
98
+
#### Using a system or user assigned managed identity for scanning
103
99
104
-
-**Service Principal** - In this method, you can create a new or use an existing service principal in your Azure Active Directory tenant.
100
+
There are two types of managed identity you can use:
105
101
106
-
#### Using a system or user assigned managed identity for scanning
102
+
***System-assigned managed identity (Recommended)** - As soon as the Microsoft Purview Account is created, a system-assigned managed identity (SAMI) is created automatically in Azure AD tenant. Depending on the type of resource, specific RBAC role assignments are required for the Microsoft Purview system-assigned managed identity (SAMI) to perform the scans.
107
103
108
-
It is important to give your Microsoft Purview account the permission to scan the Azure Blob data source. You can add access for the SAMI or UAMI at the Subscription, Resource Group, or Resource level, depending on what level scan permission is needed.
104
+
***User-assigned managed identity** (preview) - Similar to a system managed identity, a user-assigned managed identity (UAMI) is a credential resource that can be used to allow Microsoft Purview to authenticate against Azure Active Directory. For more information, you can see our [User-assigned managed identity guide](manage-credentials.md#create-a-user-assigned-managed-identity).
105
+
It's important to give your Microsoft Purview account the permission to scan the Azure Blob data source. You can add access for the SAMI or UAMI at the Subscription, Resource Group, or Resource level, depending on what level scan permission is needed.
109
106
110
107
> [!NOTE]
111
108
> If you have firewall enabled for the storage account, you must use **managed identity** authentication method when setting up a scan.
@@ -164,14 +161,14 @@ When authentication method selected is **Account Key**, you need to get your acc
164
161
165
162
1. Select **Create** to complete
166
163
167
-
1. If your key vault is not connected to Microsoft Purview yet, you will need to [create a new key vault connection](manage-credentials.md#create-azure-key-vaults-connections-in-your-microsoft-purview-account)
164
+
1. If your key vault isn't connected to Microsoft Purview yet, you'll need to [create a new key vault connection](manage-credentials.md#create-azure-key-vaults-connections-in-your-microsoft-purview-account)
168
165
1. Finally, [create a new credential](manage-credentials.md#create-a-new-credential) using the key to set up your scan
169
166
170
167
#### Using Service Principal for scanning
171
168
172
169
##### Creating a new service principal
173
170
174
-
If you need to [Create a new service principal](./create-service-principal-azure.md), it is required to register an application in your Azure AD tenant and provide access to Service Principal in your data sources. Your Azure AD Global Administrator or other roles such as Application Administrator can perform this operation.
171
+
If you need to [Create a new service principal](./create-service-principal-azure.md), it's required to register an application in your Azure AD tenant and provide access to Service Principal in your data sources. Your Azure AD Global Administrator or other roles such as Application Administrator can perform this operation.
175
172
176
173
##### Getting the Service Principal's Application ID
177
174
@@ -181,7 +178,7 @@ If you need to [Create a new service principal](./create-service-principal-azure
181
178
182
179
##### Granting the Service Principal access to your Azure Blob account
183
180
184
-
It is important to give your service principal the permission to scan the Azure Blob data source. You can add access for the service principal at the Subscription, Resource Group, or Resource level, depending on what level scan access is needed.
181
+
It's important to give your service principal the permission to scan the Azure Blob data source. You can add access for the service principal at the Subscription, Resource Group, or Resource level, depending on what level scan access is needed.
185
182
186
183
> [!Note]
187
184
> You need to be an owner of the subscription to be able to add a service principal on an Azure resource.
@@ -214,13 +211,13 @@ Provide a **Name** for the scan, select the Microsoft Purview accounts SAMI or U
214
211
215
212
#### If using Account Key
216
213
217
-
Provide a **Name** for the scan, choose the appropriate collection for the scan, and select **Authentication method** as _Account Key_ and select **Create**
214
+
Provide a **Name** for the scan, select the Azure IR or your Self-Hosted IR depending on your configuration, choose the appropriate collection for the scan, and select **Authentication method** as _Account Key_ and select **Create**
218
215
219
216
:::image type="content" source="media/register-scan-azure-blob-storage-source/register-blob-acct-key.png" alt-text="Screenshot that shows the Account Key option for scanning":::
220
217
221
218
#### If using Service Principal
222
219
223
-
1. Provide a **Name** for the scan, choose the appropriate collection for the scan, and select the **+ New** under **Credential**
220
+
1. Provide a **Name** for the scan, select the Azure IR or your Self-Hosted IR depending on your configuration, choose the appropriate collection for the scan, and select the **+ New** under **Credential**
224
221
225
222
:::image type="content" source="media/register-scan-azure-blob-storage-source/register-blob-sp-option.png" alt-text="Screenshot that shows the option for service principal to enable scanning":::
226
223
@@ -318,13 +315,13 @@ To map a storage account asset in a received share, you need ONE of the followin
318
315
***Microsoft.Storage/storageAccounts/blobServices/containers/write** - This permission is available in the *Contributor*, *Owner*, *Storage Blob Data Contributor* and *Storage Blob Data Owner* role.
319
316
320
317
### Update shared data in source storage account
321
-
Updates you make to shared files or data in the shared folder from source storage account will be made available to recipient in target storage account in near real time. When you delete subfolder or files within the shared folder, they will disappear for recipient. To delete the shared folder, file or parent folders or containers, you need to first revoke access to all your shares from the source storage account.
318
+
Updates you make to shared files or data in the shared folder from source storage account will be made available to recipient in target storage account in near real time. When you delete subfolder or files within the shared folder, they'll disappear for recipient. To delete the shared folder, file or parent folders or containers, you need to first revoke access to all your shares from the source storage account.
322
319
323
320
### Access shared data in target storage account
324
321
The target storage account enables recipient to access the shared data read-only in near real time. You can connect analytics tools such as Synapse Workspace and Databricks to the shared data to perform analytics. Cost of accessing the shared data is charged to the target storage account.
325
322
326
323
### Service limit
327
-
Source storage account can support up to 20 targets, and target storage account can support up to 100 sources. If you require an increase in limit, please contact Support.
324
+
Source storage account can support up to 20 targets, and target storage account can support up to 100 sources. If you require an increase in limit, contact Support.
0 commit comments