Merge pull request #109796 from lobrien/1677813-data-pipelines

Moving data into and between pipeline steps

Author: v-albemi

articles/machine-learning/how-to-move-data-in-out-of-pipelines.md

@@ -0,0 +1,236 @@
+---
+title: 'Input and output data from ML pipelines'
+titleSuffix: Azure Machine Learning
+description: Prepare, consume, and generate data in Azure Machine Learning pipelines
+services: machine-learning
+ms.service: machine-learning
+ms.subservice: core
+ms.topic: conceptual
+ms.author: laobri
+author: lobrien
+ms.date: 04/01/2020
+# As a data scientist using Python, I want to get data into my pipeline and flowing between steps
+---
+
+# Moving data into and between ML pipeline steps (Python)
+
+[!INCLUDE [applies-to-skus](../../includes/aml-applies-to-basic-enterprise-sku.md)]
+
+Data is central to machine learning pipelines. This article provides code for importing, transforming, and moving data between steps in an Azure Machine Learning pipeline. For an overview of how data works in Azure Machine Learning, see [Access data in Azure storage services](how-to-access-data.md). For the benefits and structure of Azure Machine Learning pipelines, see [What are Azure Machine Learning pipelines?](concept-ml-pipelines.md).
+
+This article will show you how to:
+
+- Use `Dataset` objects for pre-existing data
+- Access data within your steps
+- Split `Dataset` data into subsets, such as training and validation subsets
+- Create `PipelineData` objects to transfer data to the next pipeline step
+- Use `PipelineData` objects as input to pipeline steps
+- Create new `Dataset` objects from `PipelineData` you wish to persist
+
+## Prerequisites
+
+You'll need:
+
+- An Azure subscription. If you don't have an Azure subscription, create a free account before you begin. Try the [free or paid version of Azure Machine Learning](https://aka.ms/AMLFree).
+
+- The [Azure Machine Learning SDK for Python](https://docs.microsoft.com/python/api/overview/azure/ml/intro?view=azure-ml-py), or access to [Azure Machine Learning studio](https://ml.azure.com/).
+
+- An Azure Machine Learning workspace.
+  
+  Either [create an Azure Machine Learning workspace](how-to-manage-workspace.md) or use an existing one via the Python SDK. Import the `Workspace` and `Datastore` class, and load your subscription information from the file `config.json` using the function `from_config()`. This function looks for the JSON file in the current directory by default, but you can also specify a path parameter to point to the file using `from_config(path="your/file/path")`.
+
+   ```python
+   import azureml.core
+   from azureml.core import Workspace, Datastore
+        
+   ws = Workspace.from_config()
+   ```
+
+- Some pre-existing data. This article briefly shows the use of an [Azure blob container](https://docs.microsoft.com/azure/storage/blobs/storage-blobs-overview).
+
+- Optional: An existing machine learning pipeline, such as the one described in [Create and run machine learning pipelines with Azure Machine Learning SDK](how-to-create-your-first-pipeline.md).
+
+## Use `Dataset` objects for pre-existing data 
+
+The preferred way to ingest data into a pipeline is to use a [Dataset](https://docs.microsoft.com/python/api/azureml-core/azureml.core.dataset%28class%29?view=azure-ml-py) object. `Dataset` objects represent persistent data available throughout a workspace.
+
+There are many ways to create and register `Dataset` objects. Tabular datasets are for delimited data available in one or more files. File datasets are for binary data (such as images) or for data that you'll parse. The simplest programmatic ways to create `Dataset` objects are to use existing blobs in workspace storage or public URLs:
+
+```python
+datastore = Datastore.get(workspace, 'training_data')
+iris_dataset = Dataset.Tabular.from_delimited_files(DataPath(datastore, 'iris.csv'))
+
+cats_dogs_dataset = Dataset.File.from_files(
+    paths='https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip',
+    archive_options=ArchiveOptions(archive_type=ArchiveType.ZIP, entry_glob='**/*.jpg')
+)
+```
+
+For more options on creating datasets with different options and from different sources, registering them and reviewing them in the Azure Machine Learning UI, understanding how data size interacts with compute capacity, and versioning them, see [Create Azure Machine Learning datasets](how-to-create-register-datasets.md). 
+
+### Pass datasets to your script
+
+To pass the dataset's path to your script, use the `Dataset` object's `as_named_input()` method. You can either pass the resulting `DatasetConsumptionConfig` object to your script as an argument or, by using the `inputs` argument to your pipeline script, you can retrieve the dataset using `Run.get_context().input_datasets[]`.
+
+Once you've created a named input, you can choose its access mode: `as_mount()` or `as_download()`. If your script processes all the files in your dataset and the disk on your compute resource is large enough for the dataset, the download access mode is the better choice. The download access mode will avoid the overhead of streaming the data at runtime. If your script accesses a subset of the dataset or it's too large for your compute, use the mount access mode. For more information, read [Mount vs. Download](https://docs.microsoft.com/azure/machine-learning/how-to-train-with-datasets#mount-vs-download)
+
+To pass a dataset to your pipeline step:
+
+1. Use `TabularDataset.as_named_inputs()` or `FileDataset.as_named_input()` (no 's' at end) to create a `DatasetConsumptionConfig` object
+1. Use `as_mount()` or `as_download()` to set the access mode
+1. Pass the datasets to your pipeline steps using either the `arguments` or the `inputs` argument
+
+The following snippet shows the common pattern of combining these steps within the `PythonScriptStep` constructor: 
+
+```python
+
+train_step = PythonScriptStep(
+    name="train_data",
+    script_name="train.py",
+    compute_target=cluster,
+    inputs=[iris_dataset.as_named_inputs('iris').as_mount()]
+)
+```
+
+You can also use methods such as `random_split()` and `take_sample()` to create multiple inputs or reduce the amount of data passed to your pipeline step:
+
+```python
+seed = 42 # PRNG seed
+smaller_dataset = iris_dataset.take_sample(0.1, seed=seed) # 10%
+train, test = smaller_dataset.random_split(percentage=0.8, seed=seed)
+
+train_step = PythonScriptStep(
+    name="train_data",
+    script_name="train.py",
+    compute_target=cluster,
+    inputs=[train.as_named_inputs('train').as_download(), test.as_named_inputs('test').as_download()]
+)
+```
+
+### Access datasets within your script
+
+Named inputs to your pipeline step script are available as a dictionary within the `Run` object. Retrieve the active `Run` object using `Run.get_context()` and then retrieve the dictionary of named inputs using `input_datasets`. If you passed the `DatasetConsumptionConfig` object using the `arguments` argument rather than the `inputs` argument, access the data using `ArgParser` code. Both techniques are demonstrated in the following snippet.
+
+```python
+# In pipeline definition script:
+# Code for demonstration only: It would be very confusing to split datasets between `arguments` and `inputs`
+train_step = PythonScriptStep(
+    name="train_data",
+    script_name="train.py",
+    compute_target=cluster,
+    arguments=['--training-folder', train.as_named_inputs('train').as_download()]
+    inputs=[test.as_named_inputs('test').as_download()]
+)
+
+# In pipeline script
+parser = argparse.ArgumentParser()
+parser.add_argument('--training-folder', type=str, dest='train_folder', help='training data folder mounting point')
+args = parser.parse_args()
+training_data_folder = args.train_folder
+
+testing_data_folder = Run.get_context().input_datasets['test']
+```
+
+The passed value will be the path to the dataset file(s).
+
+It's also possible to access a registered `Dataset` directly. Since registered datasets are persistent and shared across a workspace, you can retrieve them directly:
+
+```python
+run = Run.get_context()
+ws = run.experiment.workspace
+ds = Dataset.get_by_name(workspace=ws, name='mnist_opendataset')
+```
+
+## Use `PipelineData` for intermediate data
+
+While `Dataset` objects represent persistent data, [PipelineData](https://docs.microsoft.com/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinedata?view=azure-ml-py) objects are used for temporary data that is output from pipeline steps. Because the lifespan of a `PipelineData` object is longer than a single pipeline step, you define them in the pipeline definition script. When you create a `PipelineData` object, you must provide a name and a datastore at which the data will reside. Pass your `PipelineData` object(s) to your `PythonScriptStep` using _both_ the `arguments` and the `outputs` arguments:
+
+```python
+default_datastore = workspace.get_default_datastore()
+dataprep_output = PipelineData("clean_data", datastore=default_datastore)
+
+dataprep_step = PythonScriptStep(
+    name="prep_data",
+    script_name="dataprep.py",
+    compute_target=cluster,
+    arguments=["--output-path", dataprep_output]
+    inputs=[Dataset.get_by_name(workspace, 'raw_data')],
+    outputs=[dataprep_output]
+)
+```
+
+You may choose to create your `PipelineData` object using an access mode that provides an immediate upload. In that case, when you create your `PipelineData`, set the `upload_mode` to `"upload"` and use the `output_path_on_compute` argument to specify the path to which you'll be writing the data:
+
+```python
+PipelineData("clean_data", datastore=def_blob_store, output_mode="upload", output_path_on_compute="clean_data_output/")
+```
+
+### Use `PipelineData` as outputs of a training step
+
+Within your pipeline's `PythonScriptStep`, you can retrieve the available output paths using the program's arguments. If this step is the first and will initialize the output data, you must create the directory at the specified path. You can then write whatever files you wish to be contained in the `PipelineData`.
+
+```python
+parser = argparse.ArgumentParser()
+parser.add_argument('--output_path', dest='output_path', required=True)
+args = parser.parse_args()
+
+# Make directory for file
+os.makedirs(os.path.dirname(args.output_path), exist_ok=True)
+with open(args.output_path, 'w') as f:
+    f.write("Step 1's output")
+```
+
+If you created your `PipelineData` with the `is_directory` argument set to `True`, it would be enough to just perform the `os.makedirs()` call and then you would be free to write whatever files you wished to the path. For more details, see the [PipelineData](https://docs.microsoft.com/python/api/azureml-pipeline-core/azureml.pipeline.core.pipelinedata?view=azure-ml-py) reference documentation.
+
+### Read `PipelineData` as inputs to non-initial steps
+
+After the initial pipeline step writes some data to the `PipelineData` path and it becomes an output of that initial step, it can be used as an input to a later step:
+
+```python
+step1_output_data = PipelineData("processed_data", datastore=def_blob_store, output_mode="upload")
+
+step1 = PythonScriptStep(
+    name="generate_data",
+    script_name="step1.py",
+    runconfig = aml_run_config,
+    arguments = ["--output_path", step1_output_data],
+    inputs=[],
+    outputs=[step1_output_data]
+)
+
+step2 = PythonScriptStep(
+    name="read_pipeline_data",
+    script_name="step2.py",
+    compute_target=compute,
+    runconfig = aml_run_config,
+    arguments = ["--pd", step1_output_data],
+    inputs=[step1_output_data]
+)
+
+pipeline = Pipeline(workspace=ws, steps=[step1, step2])
+```
+
+The value of a `PipelineData` input is the path to the previous output. If, as shown previously, the first step wrote a single file, consuming it might look like: 
+
+```python
+parser = argparse.ArgumentParser()
+parser.add_argument('--pd', dest='pd', required=True)
+args = parser.parse_args()
+
+with open(args.pd) as f:
+    print(f.read())
+```
+
+## Convert `PipelineData` objects to `Dataset`s
+
+If you'd like to make your `PipelineData` available for longer than the duration of a run, use its `as_dataset()` function to convert it to a `Dataset`. You may then register the `Dataset`, making it a first-class citizen in your workspace. Since your `PipelineData` object will have a different path every time the pipeline runs, it's highly recommended that you set `create_new_version` to `True` when registering a `Dataset` created from a `PipelineData` object.
+
+```python
+step1_output_ds = step1_output_data.as_dataset()
+step1_output_ds.register(name="processed_data", create_new_version=True)
+```
+
+## Next steps
+
+* [Create an Azure machine learning dataset](how-to-create-register-datasets.md)
+* [Create and run machine learning pipelines with Azure Machine Learning SDK](how-to-create-your-first-pipeline.md)

articles/machine-learning/toc.yml

@@ -355,6 +355,8 @@
     items:
     - name: Create ML pipelines (Python)
       href: how-to-create-your-first-pipeline.md
+    - name: Moving data into and between ML pipeline steps (Python)
+      href: how-to-move-data-in-out-of-pipelines.md
     - name: Schedule a pipeline (Python)
       href: how-to-schedule-pipelines.md
     - name: Trigger a pipeline