Merge pull request #101714 from ScarlettFalls/tutorial-04

End to End tutorial - 04 train model

Author: TedA-M

articles/iot-edge/media/tutorial-machine-learning-edge-04-train-model/cross-platform-cli.png

@@ -4,34 +4,23 @@ description: 'In this tutorial, you will train a machine learning model using Az
 author: kgremban
 manager: philmea
 ms.author: kgremban
-ms.date: 11/11/2019
+ms.date: 2/10/2020
 ms.topic: tutorial
 ms.service: iot-edge
 services: iot-edge
 ---
-
 # Tutorial: Train and deploy an Azure Machine Learning model
 
 > [!NOTE]
 > This article is part of a series for a tutorial about using Azure Machine Learning on IoT Edge. If you have arrived at this article directly, we encourage you to begin with the [first article](tutorial-machine-learning-edge-01-intro.md) in the series for the best results.
 
-In this article, we use Azure Notebooks first to train a machine learning model using Azure Machine Learning and then package that model as a container image that can be deployed as an Azure IoT Edge Module. The Azure Notebooks take advantage of an Azure Machine Learning workspace, which is a foundational block used to experiment, train, and deploy machine learning models.
-
-The activities in this portion of the tutorial are broken up across two notebooks.
-
-* **01-turbofan\_regression.ipynb:** This notebook walks through the steps to train and publish a model using Azure Machine Learning. Broadly, the steps involved are:
+In this article, we do the following tasks:
 
-  1. Download, prepare, and explore the training data
-  2. Use the service workspace to create and run a machine learning experiment
-  3. Evaluate the model results from the experiment
-  4. Publish the best model to the service workspace
+* Use Azure Notebooks to train a machine learning model.
+* Package the trained model as a container image.
+* Deploy the container image as an Azure IoT Edge module.
 
-* **02-turbofan\_deploy\_model.ipynb:** This notebook takes the model created in the previous notebook and uses it to create a container image ready to be deployed to an Azure IoT Edge device.
-
-  1. Create a scoring script for the model
-  2. Create and publish the image
-  3. Deploy the image as a web service on Azure Container Instance
-  4. Use the web service to validate the model and the image work as expected
+The Azure Notebooks take advantage of an Azure Machine Learning workspace, a foundational block used to experiment, train, and deploy machine learning models.
 
 The steps in this article might be typically performed by data scientists.
 
@@ -42,76 +31,135 @@ We use Azure Notebooks to host the two Jupyter Notebooks and supporting files. H
 * **Quickstart:** [Create and share a notebook](../notebooks/quickstart-create-share-jupyter-notebook.md)
 * **Tutorial:** [Create and run a Jupyter notebook with Python](../notebooks/tutorial-create-run-jupyter-notebook.md)
 
-As with the developer virtual machine before, using Azure notebooks ensures a consistent environment for the exercise.
+Using Azure notebooks ensures a consistent environment for the exercise.
 
 > [!NOTE]
-> Once set up, the Azure Notebooks service can be accessed from any machine. During setup, you should use the development virtual machine, which has all of the files that you will need.
+> Once set up, the Azure Notebooks service can be accessed from any machine. During setup, you should use the development VM, which has all of the files that you will need.
 
 ### Create an Azure Notebooks account
 
-Azure Notebook accounts are independent from Azure subscriptions. To use Azure Notebooks, you need to create an account.
+To use Azure Notebooks, you need to create an account. Azure Notebook accounts are independent from Azure subscriptions.
 
 1. Navigate to [Azure notebooks](https://notebooks.azure.com).
 
-2. Click **Sign In** in the upper, right-hand corner of the page.
+1. Click **Sign In** in the upper right corner of the page.
 
-3. Sign in with either your work or school account (Azure Active Directory) or your personal account (Microsoft Account).
+1. Sign in with either your work or school account (Azure Active Directory) or your personal account (Microsoft Account).
 
-4. If you have not used Azure Notebooks before, you will be prompted to grant access for the Azure Notebooks app.
+1. If you have not used Azure Notebooks before, you will be prompted to grant access for the Azure Notebooks app.
 
-5. Create a user ID for Azure Notebooks.
+1. Create a user ID for Azure Notebooks.
 
-### Upload Jupyter notebooks files
+### Upload Jupyter notebook files
 
-In this step, we create a new Azure Notebooks project and upload files to it. Specifically, the files that we upload are:
+We will upload sample notebook files into a new Azure Notebooks project.
 
-* **01-turbofan\_regression.ipynb**: Jupyter notebook file that walks through the process of downloading the data generated by the device harness from the Azure storage account; exploring and preparing the data for training the classifier; training the model; testing the data using the test dataset found in the Test\_FD003.txt file; and, finally saving the classifier model in the Machine Learning service workspace.
+1. From the user page of your new account, select **My Projects** from the top menu bar.
 
-* **02-turbofan\_deploy\_model.ipynb:** Jupyter notebook that guides you through the process of using the classifier model saved in the Machine Learning service workspace to produce a container image. Once the image is created, the notebook walks you through the process of deploying the image as a web service so that you can validate it is working as expected. The validated image will be deployed to our IoT Edge device in the [Create and deploy custom IoT Edge modules](tutorial-machine-learning-edge-06-custom-modules.md) portion of this tutorial.
+1. On the **Create New Project** dialog  box, provide a **Project Name** that also automatically forms the **Project ID**.
 
-* **Test\_FD003.txt:** This file contains the data we will use as our test set when validating our trained classifier. We chose to use the test data as provided for the original contest as our test set for simplicity of the example.
+1. Leave **Public** and **README** unchecked as there is no need for the project to be public or to have a readme.
 
-* **RUL\_FD003.txt:** This file contains the RUL for the last cycle of each device in the Test\_FD003.txt file. See the **readme.txt** and the **Damage Propagation Modeling.pdf** files in the C:\\source\\IoTEdgeAndMlSample\\data\\Turbofan for a detailed explanation of the data.
+1. Select **Create**.
 
-* **Utils.py:** Contains a set of Python utility functions for working with data. The first notebook contains a detailed explanation of the functions.
+1. Select **Upload** (the up arrow icon) and choose **From Computer**.
 
-* **README.md:** Readme describing the use of the notebooks.
+1. Select **Choose files**.
 
-Create a new project and upload the files to your notebook.
+1. Navigate to **C:\source\IoTEdgeAndMlSample\AzureNotebooks**. Select all the files in the list and click **Open**.
 
-1. Select **My Projects** from the top menu bar.
+1. Select **Upload** to begin uploading and then select **Done** once the process is complete.
 
-1. Select **+ New Project**. Provide a name and an ID. There is no need for the project to be public or to have a readme.
+### Azure notebook files
 
-1. Select **Upload** and choose **From Computer**.
+Let's review the files you uploaded into your Azure Notebooks project. The activities in this portion of the tutorial span across two notebook files, which use a few supporting files.
 
-1. Select **Choose files**.
+* **01-turbofan\_regression.ipynb:** This notebook uses the Machine Learning service workspace to create and run a machine learning experiment. Broadly, the notebook does the following steps:
 
-1. Navigate to **C:\source\IoTEdgeAndMlSample\AzureNotebooks**. Select all files and click **Open**.
+  1. Downloads data from the Azure Storage account that was generated by the device harness.
+  1. Explores and prepare data for training and the classifier ad model.
+  1. Evaluate the model from the experiment using a test dataset (Test\_FD003.txt).
+  1. Publishes the best classifier model to the Machine Learning service workspace.
 
-1. Select **Upload** to begin uploading and then select **Done** once the process is complete.
+* **02-turbofan\_deploy\_model.ipynb:** This notebook takes the model created in the previous notebook and uses it to create a container image ready to be deployed to an Azure IoT Edge device. The notebook performs the following steps:
+
+  1. Creates a scoring script for the model.
+  1. Produces a container image using the classifier model that was saved in the Machine Learning service workspace.
+  1. Deploys the image as a web service on Azure Container Instance.
+  1. Uses the web service to validate the model and the image work as expected. The validated image will be deployed to our IoT Edge device in the [Create and deploy custom IoT Edge modules](tutorial-machine-learning-edge-06-custom-modules.md) portion of this tutorial.
+
+* **Test\_FD003.txt:** This file contains the data we will use as our test set when validating our trained classifier. We chose to use the test data, as provided for the original contest, as our test set for its simplicity.
+
+* **RUL\_FD003.txt:** This file contains the RUL for the last cycle of each device in the Test\_FD003.txt file. See the readme.txt and the Damage Propagation Modeling.pdf files in the C:\\source\\IoTEdgeAndMlSample\\data\\Turbofan for a detailed explanation of the data.
+
+* **Utils.py:** Contains a set of Python utility functions for working with data. The first notebook contains a detailed explanation of the functions.
+
+* **README.md:** Readme describing the use of the notebooks.  
 
 ## Run Azure Notebooks
 
-Now that the project is created, run the **01-turbofan\_regression.ipynb** notebook.
+Now that the project is created, you can run the notebooks. 
 
-1. From the turbofan project page, select **01-turbofan\_regression.ipynb**.
+1. From your project page, select **01-turbofan\_regression.ipynb**.
 
     ![Select first notebook to run](media/tutorial-machine-learning-edge-04-train-model/select-turbofan-regression-notebook.png)
 
-2. If prompted, choose the Python 3.6 Kernel from the dialog and select **Set Kernel**.
+1. If prompted, choose the Python 3.6 Kernel from the dialog and select **Set Kernel**.
+
+1. If the notebook is listed as **Not Trusted**, click on the **Not Trusted** widget in the top right of the notebook. When the dialog comes up, select **Trust**.
+
+1. In the notebook, scroll down to the cell that follows the **Set global properties** instructions and that begins with the code `AZURE_SUBSCRIPTION_ID =` and fill in the values for your Azure subscription, settings, and resources.
+
+    ![Set global properties in the notebook](media/tutorial-machine-learning-edge-04-train-model/set-global-properties.png)
+
+1. Run that cell by selecting **Run** on the toolbar.
+
+    When a cell is running, it displays an asterisk between the square brackets  ([\*]). When the cell's operation is complete, the asterisk is replaced with a number and relevant output may appear. The cells in a notebook build sequentially and only one can be running at a time.
+
+    Follow the instructions in the notebook. You can also use run options from the **Cell** menu, `Ctrl` + `Enter` to run a cell, and `Shift` + `Enter` to run a cell and advance to the next cell.
+
+    > [!TIP]
+    > For consistent cell operations, avoid running the same notebook from multiple tabs in your browser.
+
+1. Scroll down to the cell that immediately follows the **Create a workspace** overview text and run that cell. In the cell's output, look for the link that instructs you to sign in to authenticate. 
+
+    ![Sign in prompt for device authentication](media/tutorial-machine-learning-edge-04-train-model/sign-in-prompt.png)
+
+    Open the link and enter the specified code. This sign-in procedure authenticates the Jupyter notebook to access Azure resources using the Microsoft Azure Cross-Platform Command Line Interface.  
+
+    ![Authenticate application on device confirmation](media/tutorial-machine-learning-edge-04-train-model/cross-platform-cli.png)
+
+1. At this point, you can run the rest of the cells. It is optimal to run all the cells so that the code in the cells runs sequentially. Select **Run All** from the **Cell** menu. Scroll back up through the notebook and review how the cell operations are completed.
+
+    In the **Explore the data** section, you can review cells in the **Sensor readings and RUL** subsection that render scatterplots of sensor measurements.
+
+    ![Sensor readings scatterplots](media/tutorial-machine-learning-edge-04-train-model/sensor-readings.png)
+
+1. Save the notebook and return to your project page by clicking your project name in the upper-right corner of the notebook or going back in your browser.
+
+1. Open **02-turbofan\_deploy\_model.ipynb** and repeat the steps in this procedure to run the second notebook.
+
+1. Save the notebook and return to your project page by clicking your project name in the upper-right corner of the notebook or going back in your browser.
+
+### Verify success
+
+To verify that the notebooks have completed successfully, verify that a few items were created.
+
+1. On your Azure Notebooks project page, select **Show hidden items** so that item names that begin with a period appear.
 
-3. If the notebook is listed as **Not Trusted**, click on the **Not Trusted** widget in the top right of the notebook. When the dialog comes up, select **Trust**.
+1. Verify that the following files were created:
 
-4. Follow the instructions in the notebook.
+    | File | Description |
+    | --- | --- |
+    | ./aml_config/.azureml/config.json | Configuration file used to create the Azure Machine Learning Workspace. |
+    | ./aml_config/model_config.json | Configuration file that we will need to deploy the model in the **turbofanDemo** Machine Learning workspace in Azure. |
+    | myenv.yml| Provides information about the dependencies for the deployed Machine Learning model.|
 
-    * `Ctrl + Enter` runs a cell.
-    * `Shift + Enter` runs a cell and navigates to the next cell.
-    * When a cell is running, it has an asterisk between the square brackets, like **[\*]**. When it is complete, the asterisk will be replaced with a number and relevant output may be appear below. Since cells often build on the work of the previous ones, only one cell can run at a time.
+1. Verify in the Azure portal that the **turboFanDemo** Machine Learning workspace exists in your resource group.
 
-5. When you have finished running the **01-turbofan\_regression.ipynb** notebook, return to the project page.
+### Debugging
 
-6. Open **02-turbofan\_deploy\_model.ipynb** and repeat the steps in this section to run the second notebook.
+You can inset Python statements into the notebook for debugging, mainly the `print()` command. If you see variables or objects that are not defined, run the cells where they are first declared or instantiated.
 
 ## Next steps