You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/machine-learning/concept-data.md
+18-12Lines changed: 18 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,30 +1,35 @@
1
1
---
2
2
title: Data in Azure Machine Learning
3
3
titleSuffix: Azure Machine Learning
4
-
description: Learn how Azure Machine Learning interacts with your data and how it's utilized across your machine learning experiments.
4
+
description: Learn how Azure Machine Learning securely connects to your data, and uses that data for machine learning tasks.
5
5
services: machine-learning
6
6
ms.service: machine-learning
7
7
ms.subservice: core
8
8
ms.topic: conceptual
9
9
ms.reviewer: nibaccam
10
10
author: nibaccam
11
11
ms.author: nibaccam
12
-
ms.date: 03/18/2020
12
+
ms.date: 03/20/2020
13
13
14
-
#Customer intent: As an experienced Python developer, I need to securely access my data in my Azure storage solutions and use it to accomplish my machine learning tasks.
14
+
#Customer intent: As an experienced Python developer, I need to securely access my data in my Azure storage solutions and use it to accomplish my machine learning tasks.
15
15
---
16
16
17
17
# Data access in Azure Machine Learning
18
18
19
-
In this article, you learn Azure Machine Learning's data access solutions for connecting to your data in Azure storage, and consuming it in machine learning tasks.
19
+
Azure Machine Learning makes it easy to connect to your data in the cloud. It provides an abstraction layer over the underlying storage service, so you can securely access and work with your data without having to write code specific to your storage type. Azure Machine Learning also provides the following data capabilities:
20
20
21
-
This article assumes you've already created an [Azure storage account](https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal) and [Azure storage service](https://docs.microsoft.com/azure/storage/common/storage-introduction).
21
+
* Versioning and tracking of data lineage
22
+
* Data labeling
23
+
* Data drift monitoring
24
+
* Interoperability with Pandas and Spark DataFrames
22
25
23
-
When you're ready to use the data in your Azure storage solution, we recommend the following data delivery workflow.
26
+
## Data workflow
24
27
25
-
1. Create an Azure Machine Learning datastore to store connection information to your Azure storage.
28
+
When you're ready to use the data in your cloud-based storage solution, we recommend the following data delivery workflow. This workflow assumes you have an [Azure storage account](https://docs.microsoft.com/azure/storage/common/storage-quickstart-create-account?tabs=azure-portal) and data in a cloud-based storage service in Azure.
26
29
27
-
2. From that datastore, create an Azure Machine Learning dataset to point to a specific file(s) in your underlying storage.
30
+
1. Create an [Azure Machine Learning datastore](#datastores) to store connection information to your Azure storage.
31
+
32
+
2. From that datastore, create an [Azure Machine Learning dataset](#datasets) to point to a specific file(s) in your underlying storage.
28
33
29
34
3. To use that dataset in your machine learning experiment you can either
30
35
1. Mount it to your experiment's compute target for model training.
@@ -33,7 +38,7 @@ When you're ready to use the data in your Azure storage solution, we recommend t
33
38
34
39
1. Consume it directly in Azure Machine Learning solutions like, automated machine learning (automated ML) experiment runs, machine learning pipelines, or the [Azure Machine Learning designer](concept-designer.md).
35
40
36
-
4. Create dataset monitors for your model output dataset to detect for data drift.
41
+
4. Create [dataset monitors](#data-drift) for your model output dataset to detect for data drift.
37
42
38
43
5. If data drift is detected, update your input dataset and retrain your model accordingly.
39
44
@@ -45,7 +50,7 @@ The following diagram provides a visual demonstration of this recommended workfl
45
50
46
51
Azure Machine Learning datastores securely keep the connection information to your Azure storage, so you don't have to code it in your scripts. [Register and create a datastore](how-to-access-data.md) to easily connect to your storage account, and access the data in your underlying Azure storage service.
47
52
48
-
Supported Azure storage services that can be registered as datastores:
53
+
Supported cloud-based storage services in Azure that can be registered as datastores:
49
54
50
55
+ Azure Blob Container
51
56
+ Azure File Share
@@ -60,7 +65,7 @@ Supported Azure storage services that can be registered as datastores:
60
65
61
66
Azure Machine Learning datasets are references that point to the data in your storage service. They aren't copies of your data, so no extra storage cost is incurred. To interact with your data in storage, [create a dataset](how-to-create-register-datasets.md) to package your data into a consumable object for machine learning tasks. Register the dataset to your workspace to share and reuse it across different experiments without data ingestion complexities.
62
67
63
-
Datasets can be created from local files, public urls, Azure Open Datasets, or specific file(s) in your datastores. To create a dataset from an in memory pandas dataframe, write the data to a local file, like a csv, and create your dataset from that file.
68
+
Datasets can be created from local files, public urls, [Azure Open Datasets](https://azure.microsoft.com/services/open-datasets/), or specific file(s) in your datastores. To create a dataset from an in memory pandas dataframe, write the data to a local file, like a csv, and create your dataset from that file.
64
69
65
70
The following diagram shows that if you don't have an Azure storage service, you can create a dataset directly from local files, public urls, or an Azure Open Dataset. Doing so connects your dataset to the default datastore that was automatically created with your experiment's [Azure Machine Learning workspace](concept-workspace.md).
66
71
@@ -97,13 +102,14 @@ Labeling large amounts of data has often been a headache in machine learning pro
97
102
98
103
Azure Machine Learning gives you a central location to create, manage, and monitor labeling projects. Labeling projects help coordinate the data, labels, and team members, allowing you to more efficiently manage the labeling tasks. Currently supported tasks are image classification, either multi-label or multi-class, and object identification using bounded boxes.
99
104
100
-
+Create a [data labeling project](how-to-create-labeling-projects.md), and output a dataset for use in machine learning experiments.
105
+
Create a [data labeling project](how-to-create-labeling-projects.md), and output a dataset for use in machine learning experiments.
101
106
102
107
<aname="drift"></a>
103
108
104
109
## Data drift
105
110
106
111
In the context of machine learning, data drift is the change in model input data that leads to model performance degradation. It is one of the top reasons model accuracy degrades over time, thus monitoring data drift helps detect model performance issues.
112
+
107
113
See the [Create a dataset monitor](how-to-monitor-datasets.md) article, to learn more about how to detect and alert to data drift on new data in a dataset.
Copy file name to clipboardExpand all lines: articles/machine-learning/how-to-access-data.md
+12-13Lines changed: 12 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,5 @@
1
1
---
2
-
title: Access data in Azure Storage services
2
+
title: Access data in Azure storage services
3
3
titleSuffix: Azure Machine Learning
4
4
description: Learn how to use datastores to securely connect to Azure storage services during training with Azure Machine Learning
5
5
services: machine-learning
@@ -12,16 +12,15 @@ ms.reviewer: nibaccam
12
12
ms.date: 02/27/2020
13
13
ms.custom: seodec18
14
14
15
-
# Customer intent: As an experienced Python developer, I need to make my data in Azure Storage available to my remote compute to train my machine learning models.
16
-
15
+
# Customer intent: As an experienced Python developer, I need to make my data in Azure storage available to my remote compute to train my machine learning models.
In this article, learn how to easily access your data in Azure Storage services via Azure Machine Learning datastores. Datastores store connection information, like your subscription ID and token authorization, so you can access your storage without having to hard code them in your scripts.
21
+
In this article, learn how to easily access your data in Azure storage services via Azure Machine Learning datastores. Datastores store connection information, like your subscription ID and token authorization, so you can access your storage without having to hard code them in your scripts.
23
22
24
-
You can create datastores from [these Azure Storage solutions](#matrix). For unsupported storage solutions, and to save data egress cost during machine learning experiments, we recommend that you [move your data](#move) to supported Azure Storage solutions.
23
+
You can create datastores from [these Azure storage solutions](#matrix). For unsupported storage solutions, and to save data egress cost during machine learning experiments, we recommend that you [move your data](#move) to supported Azure storage solutions.
25
24
26
25
## Prerequisites
27
26
@@ -54,10 +53,10 @@ Datastores currently support storing connection information to the storage servi
[Databricks File System](https://docs.microsoft.com/azure/databricks/data/databricks-file-system)| No authentication | | ✓** | ✓ ** |✓**
61
60
62
61
*MySQL is only supported for pipeline [DataTransferStep](https://docs.microsoft.com/python/api/azureml-pipeline-steps/azureml.pipeline.steps.datatransferstep?view=azure-ml-py). <br>
63
62
**Databricks is only supported for pipeline [DatabricksStep](https://docs.microsoft.com/python/api/azureml-pipeline-steps/azureml.pipeline.steps.databricks_step.databricksstep?view=azure-ml-py)
@@ -72,7 +71,7 @@ When you create a workspace, an Azure blob container and an Azure file share are
72
71
73
72
## Create and register datastores
74
73
75
-
When you register an Azure Storage solution as a datastore, you automatically create and register that datastore to a specific workspace. You can create and register datastores to a workspace by using the Python SDK or Azure Machine Learning studio.
74
+
When you register an Azure storage solution as a datastore, you automatically create and register that datastore to a specific workspace. You can create and register datastores to a workspace by using the Python SDK or Azure Machine Learning studio.
76
75
77
76
>[!IMPORTANT]
78
77
> As part of the initial datastore create and register process, Azure Machine Learning validates that the underlying storage service exists and that the user provided principal (username, service principal or SAS token) has access to that storage. For Azure Data Lake Storage Gen 1 and 2 datastores, however, this validation happens later, when data access methods like [`from_files()`](https://docs.microsoft.com/python/api/azureml-core/azureml.data.dataset_factory.filedatasetfactory?view=azure-ml-py) or [`from_delimited_files()`](https://docs.microsoft.com/python/api/azureml-core/azureml.data.dataset_factory.tabulardatasetfactory?view=azure-ml-py#from-parquet-files-path--validate-true--include-path-false--set-column-types-none--partition-format-none-) are called.
@@ -169,7 +168,7 @@ Create a new datastore in a few steps in Azure Machine Learning studio:
169
168
1. Sign in to [Azure Machine Learning studio](https://ml.azure.com/).
170
169
1. Select **Datastores** on the left pane under **Manage**.
171
170
1. Select **+ New datastore**.
172
-
1. Complete the form for a new datastore. The form intelligently updates itself based on your selections for Azure Storage type and authentication type.
171
+
1. Complete the form for a new datastore. The form intelligently updates itself based on your selections for Azure storage type and authentication type.
173
172
174
173
You can find the information that you need to populate the form on the [Azure portal](https://portal.azure.com). Select **Storage Accounts** on the left pane, and choose the storage account that you want to register. The **Overview** page provides information such as the account name, container, and file share name.
175
174
@@ -281,9 +280,9 @@ For situations where the SDK doesn't provide access to datastores, you might be
281
280
282
281
<aname="move"></a>
283
282
284
-
## Move data to supported Azure Storage solutions
283
+
## Move data to supported Azure storage solutions
285
284
286
-
Azure Machine Learning supports accessing data from Azure Blob storage, Azure Files, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, Azure SQL Database, and Azure Database for PostgreSQL. If you're using unsupported storage, we recommend that you move your data to supported Azure Storage solutions by using [Azure Data Factory and these steps](https://docs.microsoft.com/azure/data-factory/quickstart-create-data-factory-copy-data-tool). Moving data to supported storage can help you save data egress costs during machine learning experiments.
285
+
Azure Machine Learning supports accessing data from Azure Blob storage, Azure Files, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, Azure SQL Database, and Azure Database for PostgreSQL. If you're using unsupported storage, we recommend that you move your data to supported Azure storage solutions by using [Azure Data Factory and these steps](https://docs.microsoft.com/azure/data-factory/quickstart-create-data-factory-copy-data-tool). Moving data to supported storage can help you save data egress costs during machine learning experiments.
287
286
288
287
Azure Data Factory provides efficient and resilient data transfer with more than 80 prebuilt connectors at no additional cost. These connectors include Azure data services, on-premises data sources, Amazon S3 and Redshift, and Google BigQuery.
0 commit comments