Merge pull request #199060 from Blackmist/data-day-of

Data day of //Build update

Author: ShannonLeavitt

.openpublishing.redirection.json

@@ -21204,6 +21204,16 @@
       "redirect_url": "/azure/machine-learning/v1/reference-pipeline-yaml",
       "redirect_document_id": false
     },
+    {
+      "source_path_from_root": "/articles/machine-learning/how-to-create-register-datasets.md",
+      "redirect_url": "/azure/machine-learning/how-to-create-register-data-assets",
+      "redirect_document_id": false
+    },
+    {
+      "source_path_from_root": "/articles/machine-learning/how-to-access-data.md",
+      "redirect_url": "/azure/machine-learning/how-to-datastore",
+      "redirect_document_id": false
+    },
     {
       "source_path_from_root": "/articles/machine-learning/how-to-deploy-azure-container-instance.md",
       "redirect_url": "/azure/machine-learning/v1/how-to-deploy-azure-container-instance",

articles/machine-learning/concept-datastore.md

@@ -0,0 +1,51 @@
+---
+title: Azure Machine Learning datastores
+titleSuffix: Azure Machine Learning
+description: Learn how to securely connect to your data storage on Azure with Azure Machine Learning datastores.
+services: machine-learning
+ms.service: machine-learning
+ms.subservice: enterprise-readiness
+ms.topic: conceptual
+ms.reviewer: nibaccam
+author: blackmist
+ms.author: larryfr
+ms.date: 10/21/2021
+ms.custom: devx-track-python, data4ml
+
+# Customer intent: As an experienced Python developer, I need to securely access my data in my Azure storage solutions and use it to accomplish my machine learning tasks.
+---
+
+# Azure Machine Learning datastores
+
+Supported cloud-based storage services in Azure Machine Learning include:
+
++ Azure Blob Container
++ Azure File Share
++ Azure Data Lake
++ Azure Data Lake Gen2
+
+Azure Machine Learning allows you to connect to data directly by using a storage URI, for example: 
+
+- ```https://storageAccount.blob.core.windows.net/container/path/file.csv``` (Azure Blob Container)
+- ```abfss://[email protected]/base/path/folder1``` (Azure Data Lake Gen2). 
+
+Storage URIs use *identity-based* access that will prompt you for your Azure Active Directory token for data access authentication. This approach allows for data access management at the storage level and keeps credentials confidential.
+
+> [!NOTE]
+> When using Notebooks in Azure Machine Learning Studio, your Azure Active Directory token is automatically passed through to storage for data access authentication.
+
+Although storage URIs provide a convenient mechanism to access data, there may be cases where using an Azure Machine Learning *Datastore* is a better option:
+
+* **You need *credential-based* data access (for example: Service Principals, SAS Tokens, Account Name/Key).** Datastores are helpful because they keep the connection information to your data storage securely in an Azure Keyvault, so you don't have to code it in your scripts.
+* **You want team members to easily discover relevant datastores.** Datastores are registered to an Azure Machine Learning workspace making them easier for your team members to find/discover them.
+
+ [Register and create a datastore](how-to-datastore.md) to easily connect to your storage account, and access the data in your underlying storage service. 
+
+## Credential-based vs identity-based access
+
+Azure Machine Learning Datastores support both credential-based and identity-based access. In *credential-based* access, your authentication credentials are usually kept in a datastore, which is used to ensure you have permission to access the storage service. When these credentials are registered via datastores, any user with the workspace Reader role can retrieve them. That scale of access can be a security concern for some organizations. When you use *identity-based* data access, Azure Machine Learning prompts you for your Azure Active Directory token for data access authentication instead of keeping your credentials in the datastore. That approach allows for data access management at the storage level and keeps credentials confidential.
+
+
+## Next steps 
+
++ [How to create a datastore](how-to-datastore.md)

articles/machine-learning/concept-network-data-access.md

@@ -136,4 +136,4 @@ To secure communication between Azure Machine Learning and Azure SQL Database, t
 
 ## Next steps
 
-For information on enabling studio in a network, see [Use Azure Machine Learning studio in an Azure Virtual Network](how-to-enable-studio-virtual-network.md).
+For information on enabling studio in a network, see [Use Azure Machine Learning studio in an Azure Virtual Network](how-to-enable-studio-virtual-network.md).

articles/machine-learning/how-to-access-data.md

@@ -0,0 +1,109 @@
+---
+title: How to administrate data authentication
+titleSuffix: Azure Machine Learning
+description: Learn how to manage data access and how to authenticate in Azure Machine Learning
+services: machine-learning
+ms.service: machine-learning
+ms.subservice: enterprise-readiness
+ms.topic: how-to
+ms.author: xunwan
+author: xunwan
+ms.reviewer: larryfr
+ms.date: 05/24/2022
+
+# Customer intent: As an administrator, I need to administrate data access and set up authentication method for data scientists.
+---
+
+# How to authenticate data access
+Learn how to manage data access and how to authenticate in Azure Machine Learning
+[!INCLUDE [sdk v2](../../includes/machine-learning-sdk-v2.md)]
+[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
+
+> [!IMPORTANT]
+> The information in this article is intended for Azure administrators who are creating the infrastructure required for an Azure Machine Learning solution.
+
+In general, data access from studio involves the following checks:
+
+* Who is accessing?
+    - There are multiple different types of authentication depending on the storage type. For example, account key, token, service principal, managed identity, and user identity.
+    - If authentication is made using a user identity, then it's important to know *which* user is trying to access storage. Learn more about [identity-based data access](how-to-identity-based-data-access.md).
+* Do they have permission?
+    - Are the credentials correct? If so, does the service principal, managed identity, etc., have the necessary permissions on the storage? Permissions are granted using Azure role-based access controls (Azure RBAC).
+    - [Reader](../role-based-access-control/built-in-roles.md#reader) of the storage account reads metadata of the storage.
+    - [Storage Blob Data Reader](../role-based-access-control/built-in-roles.md#storage-blob-data-reader) reads data within a blob container.
+    - [Contributor](../role-based-access-control/built-in-roles.md#contributor) allows write access to a storage account.
+    - More roles may be required depending on the type of storage.
+* Where is access from?
+    - User: Is the client IP address in the VNet/subnet range?
+    - Workspace: Is the workspace public or does it have a private endpoint in a VNet/subnet?
+    - Storage: Does the storage allow public access, or does it restrict access through a service endpoint or a private endpoint?
+* What operation is being performed?
+    - Create, read, update, and delete (CRUD) operations on a data store/dataset are handled by Azure Machine Learning.
+    - Data Access calls (such as preview or schema) go to the underlying storage and need extra permissions.
+* Where is this operation being run; compute resources in your Azure subscription or resources hosted in a Microsoft subscription?
+    - All calls to dataset and datastore services (except the "Generate Profile" option) use resources hosted in a __Microsoft subscription__ to run the operations.
+    - Jobs, including the "Generate Profile" option for datasets, run on a compute resource in __your subscription__, and access the data from there. So the compute identity needs permission to the storage rather than the identity of the user submitting the job.
+
+The following diagram shows the general flow of a data access call. In this example, a user is trying to make a data access call through a machine learning workspace, without using any compute resource.
+
+:::image type="content" source="./media/concept-network-data-access/data-access-flow.svg" alt-text="Diagram of the logic flow when accessing data.":::
+
+## Scenarios and identities
+
+The following table lists what identities should be used for specific scenarios:
+
+| Scenario | Use workspace</br>Managed Service Identity (MSI) | Identity to use |
+|--|--|--|
+| Access from UI | Yes | Workspace MSI |
+| Access from UI | No | User's Identity |
+| Access from Job | Yes/No | Compute MSI |
+| Access from Notebook | Yes/No | User's identity |
+
+
+Data access is complex and it's important to recognize that there are many pieces to it. For example, accessing data from Azure Machine Learning studio is different than using the SDK. When using the SDK on your local development environment, you're directly accessing data in the cloud. When using studio, you aren't always directly accessing the data store from your client. Studio relies on the workspace to access data on your behalf.
+
+> [!TIP]
+> If you need to access data from outside Azure Machine Learning, such as using Azure Storage Explorer, _user_ identity is probably what is used. Consult the documentation for the tool or service you are using for specific information. For more information on how Azure Machine Learning works with data, see [Identity-based data access to storage services on Azure](how-to-identity-based-data-access.md).
+
+## Azure Storage Account
+
+When using an Azure Storage Account from Azure Machine Learning studio, you must add the managed identity of the workspace to the following Azure RBAC roles for the storage account:
+
+* [Blob Data Reader](../role-based-access-control/built-in-roles.md#storage-blob-data-reader)
+* If the storage account uses a private endpoint to connect to the VNet, you must grant the managed identity the [Reader](../role-based-access-control/built-in-roles.md#reader) role for the storage account private endpoint.
+
+For more information, see [Use Azure Machine Learning studio in an Azure Virtual Network](how-to-enable-studio-virtual-network.md).
+
+See the following sections for information on limitations when using Azure Storage Account with your workspace in a VNet.
+
+### Secure communication with Azure Storage Account 
+
+To secure communication between Azure Machine Learning and Azure Storage Accounts, configure storage to [Grant access to trusted Azure services](../storage/common/storage-network-security.md#grant-access-to-trusted-azure-services).
+
+### Azure Storage firewall
+
+When an Azure Storage account is behind a virtual network, the storage firewall can normally be used to allow your client to directly connect over the internet. However, when using studio it isn't your client that connects to the storage account; it's the Azure Machine Learning service that makes the request. The IP address of the service isn't documented and changes frequently. __Enabling the storage firewall will not allow studio to access the storage account in a VNet configuration__.
+
+### Azure Storage endpoint type
+
+When the workspace uses a private endpoint and the storage account is also in the VNet, there are extra validation requirements when using studio:
+
+* If the storage account uses a __service endpoint__, the workspace private endpoint and storage service endpoint must be in the same subnet of the VNet.
+* If the storage account uses a __private endpoint__, the workspace private endpoint and storage service endpoint must be in the same VNet. In this case, they can be in different subnets.
+
+## Azure Data Lake Storage Gen1
+
+When using Azure Data Lake Storage Gen1 as a datastore, you can only use POSIX-style access control lists. You can assign the workspace's managed identity access to resources just like any other security principal. For more information, see [Access control in Azure Data Lake Storage Gen1](../data-lake-store/data-lake-store-access-control.md).
+
+## Azure Data Lake Storage Gen2
+
+When using Azure Data Lake Storage Gen2 as a datastore, you can use both Azure RBAC and POSIX-style access control lists (ACLs) to control data access inside of a virtual network.
+
+**To use Azure RBAC**, follow the steps in the [Datastore: Azure Storage Account](how-to-enable-studio-virtual-network.md#datastore-azure-storage-account) section of the 'Use Azure Machine Learning studio in an Azure Virtual Network' article. Data Lake Storage Gen2 is based on Azure Storage, so the same steps apply when using Azure RBAC.
+
+**To use ACLs**, the managed identity of the workspace can be assigned access just like any other security principal. For more information, see [Access control lists on files and directories](../storage/blobs/data-lake-storage-access-control.md#access-control-lists-on-files-and-directories).
+
+
+## Next steps
+
+For information on enabling studio in a network, see [Use Azure Machine Learning studio in an Azure Virtual Network](how-to-enable-studio-virtual-network.md).