Merge pull request #175475 from nibaccam/automl-test-valid

AutoML |  Test data + runs

Author: denrea

articles/machine-learning/concept-automated-ml.md

@@ -18,6 +18,8 @@ Automated machine learning, also referred to as automated ML or AutoML, is the p
 
 Traditional machine learning model development is resource-intensive, requiring significant domain knowledge and time to produce and compare dozens of models. With automated machine learning, you'll accelerate the time it takes to get production-ready ML models with great ease and efficiency.
 
+<a name="parity"></a>
+
 ## Ways to use AutoML in Azure Machine Learning
 
 Azure Machine Learning offers the following two experiences for working with automated ML. See the following sections to understand [feature availability in each experience](#parity).
@@ -28,10 +30,6 @@ Azure Machine Learning offers the following two experiences for working with aut
     * [Tutorial: Create a classification model with automated ML in Azure Machine Learning](tutorial-first-experiment-automated-ml.md).
     *  [Tutorial: Forecast demand with automated machine learning](tutorial-automated-ml-forecast.md)
 
-<a name="parity"></a>
-
-## AutoML settings and configuration
-
 ### Experiment settings 
 
 The following settings allow you to configure your automated ML experiment. 
@@ -65,7 +63,7 @@ These settings can be applied to the best model as a result of your automated ML
 |**Enable voting ensemble & stack ensemble models**| ✓|✓|
 |**Show best model based on non-primary metric**|✓||
 |**Enable/disable ONNX model compatibility**|✓||
-|**Test the model** | ✓| |
+|**Test the model** | ✓| ✓ (preview)|
 
 ### Run control settings
 
@@ -183,9 +181,65 @@ You can also inspect the logged run information, which [contains metrics](how-to
 
 While model building is automated, you can also [learn how important or relevant features are](how-to-configure-auto-train.md#explain) to the generated models.
 
-
 > [!VIDEO https://www.microsoft.com/videoplayer/embed/RE2Xc9t]
 
+<a name="local-remote"></a>
+
+## Guidance on local vs. remote managed ML compute targets
+
+The web interface for automated ML always uses a remote [compute target](concept-compute-target.md).  But when you use the Python SDK, you will choose either a local compute or a remote compute target for automated ML training.
+
+* **Local compute**: Training occurs on your local laptop or VM compute. 
+* **Remote compute**: Training occurs on Machine Learning compute clusters.  
+
+### Choose compute target
+Consider these factors when choosing your compute target:
+
+ * **Choose a local compute**: If your scenario is about initial explorations or demos using small data and short trains (i.e. seconds or a couple of minutes per child run), training on your local computer might be a better choice.  There is no setup time, the infrastructure resources (your PC or VM) are directly available.
+ * **Choose a remote ML compute cluster**: If you are training with larger datasets like in production training creating models which need longer trains, remote compute will provide much better end-to-end time performance because `AutoML` will parallelize trains across the cluster's nodes. On a remote compute, the start-up time for the internal infrastructure will add around 1.5 minutes per child run, plus additional minutes for the cluster infrastructure if the VMs are not yet up and running.
+
+### Pros and cons
+Consider these pros and cons when choosing to use local vs. remote.
+
+|  | Pros (Advantages)  |Cons (Handicaps)  |
+|---------|---------|---------|---------|
+|**Local compute target** |  <li> No environment start-up time   | <li>  Subset of features<li>  Can't parallelize runs <li> Worse for large data. <li>No data streaming while training <li>  No DNN-based featurization <li> Python SDK only |
+|**Remote ML compute clusters**|  <li> Full set of features <li> Parallelize child runs <li>   Large data support<li>  DNN-based featurization <li>  Dynamic scalability of compute cluster on demand <li> No-code experience (web UI) also available  |  <li> Start-up time for cluster nodes <li> Start-up time for each child run    |
+
+### Feature availability 
+
+More features are available when you use the remote compute, as shown in the table below. 
+
+| Feature                                                    | Remote | Local | 
+|------------------------------------------------------------|--------|-------|
+| Data streaming (Large data support, up to 100 GB)          | ✓      |       | 
+| DNN-BERT-based text featurization and training             | ✓      |       |
+| Out-of-the-box GPU support (training and inference)        | ✓      |       |
+| Image Classification and Labeling support                  | ✓      |       |
+| Auto-ARIMA, Prophet and ForecastTCN models for forecasting | ✓      |       | 
+| Multiple runs/iterations in parallel                       | ✓      |       |
+| Create models with interpretability in AutoML studio web experience UI      | ✓      |       |
+| Feature engineering customization in studio web experience UI| ✓      |       |
+| Azure ML hyperparameter tuning                             | ✓      |       |
+| Azure ML Pipeline workflow support                         | ✓      |       |
+| Continue a run                                             | ✓      |       |
+| Forecasting                                                | ✓      | ✓     |
+| Create and run experiments in notebooks                    | ✓      | ✓     |
+| Register and visualize experiment's info and metrics in UI | ✓      | ✓     |
+| Data guardrails                                            | ✓      | ✓     |
+
+## Training, validation and test data 
+
+With automated ML you provide the **training data** to train ML models, and you can specify what type of model validation to perform. Automated ML performs model validation as part of training. That is, automated ML uses **validation data** to tune model hyperparameters based on the applied algorithm to find the best combination that best fits the training data. However, the same validation data is used for each iteration of tuning, which introduces model evaluation bias since the model continues to improve and fit to the validation data. 
+
+To help confirm that such bias isn't applied to the final recommended model, automated ML supports the use of **test data** to evaluate the final model that automated ML recommends at the end of your experiment. When you provide test data as part of your AutoML experiment configuration, this recommended model is tested by default at the end of your experiment (preview). 
+
+>[!IMPORTANT]
+> Testing your models with a test dataset to evaluate generated models is a preview feature. This capability is an [experimental](/python/api/overview/azure/ml/#stable-vs-experimental) preview feature, and may change at any time.
+
+Learn how to [configure AutoML experiments to use test data (preview) with the SDK](how-to-configure-cross-validation-data-splits.md#provide-test-data-preview) or with the [Azure Machine Learning studio](how-to-use-automated-ml-for-ml-models.md#create-and-run-experiment).
+
+You can also [test any existing automated ML model (preview)](how-to-configure-auto-train.md#test-existing-automated-ml-model)), including models from child runs, by providing your own test data or by setting aside a portion of your training data. 
 
 ## Feature engineering
 
@@ -234,51 +288,6 @@ The [Caruana ensemble selection algorithm](http://www.niculescu-mizil.org/papers
 
 See the [how-to](how-to-configure-auto-train.md#ensemble) for changing default ensemble settings in automated machine learning.
 
-## <a name="local-remote"></a>Guidance on local vs. remote managed ML compute targets
-
-The web interface for automated ML always uses a remote [compute target](concept-compute-target.md).  But when you use the Python SDK, you will choose either a local compute or a remote compute target for automated ML training.
-
-* **Local compute**: Training occurs on your local laptop or VM compute. 
-* **Remote compute**: Training occurs on Machine Learning compute clusters.  
-
-### Choose compute target
-Consider these factors when choosing your compute target:
-
- * **Choose a local compute**: If your scenario is about initial explorations or demos using small data and short trains (i.e. seconds or a couple of minutes per child run), training on your local computer might be a better choice.  There is no setup time, the infrastructure resources (your PC or VM) are directly available.
- * **Choose a remote ML compute cluster**: If you are training with larger datasets like in production training creating models which need longer trains, remote compute will provide much better end-to-end time performance because `AutoML` will parallelize trains across the cluster's nodes. On a remote compute, the start-up time for the internal infrastructure will add around 1.5 minutes per child run, plus additional minutes for the cluster infrastructure if the VMs are not yet up and running.
-
-### Pros and cons
-Consider these pros and cons when choosing to use local vs. remote.
-
-|  | Pros (Advantages)  |Cons (Handicaps)  |
-|---------|---------|---------|
-|**Local compute target** |  <li> No environment start-up time   | <li>  Subset of features<li>  Can't parallelize runs <li> Worse for large data. <li>No data streaming while training <li>  No DNN-based featurization <li> Python SDK only |
-|**Remote ML compute clusters**|  <li> Full set of features <li> Parallelize child runs <li>   Large data support<li>  DNN-based featurization <li>  Dynamic scalability of compute cluster on demand <li> No-code experience (web UI) also available  |  <li> Start-up time for cluster nodes <li> Start-up time for each child run    |
-
-### Feature availability 
-
-More features are available when you use the remote compute, as shown in the table below. 
-
-| Feature                                                    | Remote | Local | 
-|------------------------------------------------------------|--------|-------|
-| Data streaming (Large data support, up to 100 GB)          | ✓      |       | 
-| DNN-BERT-based text featurization and training             | ✓      |       |
-| Out-of-the-box GPU support (training and inference)        | ✓      |       |
-| Image classification (preview) and labeling support        | ✓      |       |
-| Auto-ARIMA, Prophet and ForecastTCN models for forecasting | ✓      |       | 
-| Multiple runs/iterations in parallel                       | ✓      |       |
-| Create models with interpretability in AutoML studio web experience UI      | ✓      |       |
-| Feature engineering customization in studio web experience UI| ✓      |       |
-| Azure ML hyperparameter tuning                             | ✓      |       |
-| Azure ML Pipeline workflow support                         | ✓      |       |
-| Continue a run                                             | ✓      |       |
-| Forecasting                                                | ✓      | ✓     |
-| Computer vision (preview)                                  | ✓      |       |
-| Create and run experiments in notebooks                    | ✓      | ✓     |
-| Register and visualize experiment's info and metrics in UI | ✓      | ✓     |
-| Data guardrails                                            | ✓      | ✓     |
-
-
 <a name="use-with-onnx"></a>
 
 ## AutoML & ONNX

articles/machine-learning/how-to-configure-auto-train.md

@@ -8,7 +8,7 @@ ms.reviewer: nibaccam
 services: machine-learning
 ms.service: machine-learning
 ms.subservice: automl
-ms.date: 10/21/2021
+ms.date: 11/15/2021
 ms.topic: how-to
 ms.custom: devx-track-python,contperf-fy21q1, automl, contperf-fy21q4, FY21Q4-aml-seo-hack, contperf-fy22q1
 ---
@@ -86,7 +86,7 @@ dataset = Dataset.Tabular.from_delimited_files(data)
 
 ## Training, validation, and test data
 
-You can specify separate **training data and validation data sets** directly in the `AutoMLConfig` constructor. Learn more about [how to configure data splits and cross validation](how-to-configure-cross-validation-data-splits.md) for your AutoML experiments. 
+You can specify separate **training data and validation data sets** directly in the `AutoMLConfig` constructor. Learn more about [how to configure training, validation, cross validation, and test data](how-to-configure-cross-validation-data-splits.md) for your AutoML experiments. 
 
 If you do not explicitly specify a `validation_data` or `n_cross_validation` parameter, automated ML applies default techniques to determine how validation is performed. This determination depends on the number of rows in the dataset assigned to your `training_data` parameter. 
 
@@ -95,7 +95,15 @@ If you do not explicitly specify a `validation_data` or `n_cross_validation` par
 |**Larger than 20,000 rows**| Train/validation data split is applied. The default is to take 10% of the initial training data set as the validation set. In turn, that validation set is used for metrics calculation.
 |**Smaller&nbsp;than&nbsp;20,000&nbsp;rows**| Cross-validation approach is applied. The default number of folds depends on the number of rows. <br> **If the dataset is less than 1,000 rows**, 10 folds are used. <br> **If the rows are between 1,000 and 20,000**, then three folds are used.
 
-At this time, you need to provide your own **test data** for  model evaluation. For a code example of bringing your own test data for model evaluation see the **Test** section of [this Jupyter notebook](https://github.com/Azure/azureml-examples/blob/main/python-sdk/tutorials/automl-with-azureml/classification-credit-card-fraud/auto-ml-classification-credit-card-fraud.ipynb).
+
+> [!TIP] 
+> You can upload **test data (preview)** to evaluate models that automated ML generated for you. These features are  [experimental](/python/api/overview/azure/ml/#stable-vs-experimental) preview capabilities, and may change at any time.
+> Learn how to: 
+> * [Pass in test data to your AutoMLConfig object](how-to-configure-cross-validation-data-splits.md#provide-test-data-preview). 
+> * [Test the models automated ML generated for your experiment](#test-models-preview).
+>  
+> If you prefer a no-code experience, see [step 11 in Set up AutoML with the studio UI](how-to-use-automated-ml-for-ml-models.md#create-and-run-experiment)
+
 
 ### Large data 
 
@@ -509,9 +517,59 @@ RunDetails(run).show()
 
 ![Jupyter notebook widget for Automated Machine Learning](./media/how-to-configure-auto-train/azure-machine-learning-auto-ml-widget.png)
 
+## Test models (preview)
+
+>[!IMPORTANT]
+> Testing your models with a test dataset to evaluate automated ML generated models is a preview feature. This capability is an [experimental](/python/api/overview/azure/ml/#stable-vs-experimental) preview feature, and may change at any time.
+
+Passing the `test_data` or `test_size` parameters into the `AutoMLConfig`, automatically triggers a remote test run that uses the provided test data to evaluate the best model that automated ML recommends upon completion of the experiment. This remote test run is done at the end of the experiment, once the best model is determined. See how to [pass test data into your `AutoMLConfig`](how-to-configure-cross-validation-data-splits.md#provide-test-data-preview). 
+
+### Get test run results 
+
+You can get the predictions and metrics from the remote test run from the [Azure Machine Learning studio](how-to-use-automated-ml-for-ml-models.md#view-remote-test-run-results-preview) or with the following code. 
+
+```python
+best_run, fitted_model = remote_run.get_output()
+test_run = next(best_run.get_children(type='automl.model_test'))
+test_run.wait_for_completion(show_output=False, wait_post_processing=True)
+
+# Get test metrics
+test_run_metrics = test_run.get_metrics()
+for name, value in test_run_metrics.items():
+    print(f"{name}: {value}")
+
+# Get test predictions as a Dataset
+test_run_details = test_run.get_details()
+dataset_id = test_run_details['outputDatasets'][0]['identifier']['savedId']
+test_run_predictions = Dataset.get_by_id(workspace, dataset_id)
+predictions_df = test_run_predictions.to_pandas_dataframe()
+
+# Alternatively, the test predictions can be retrieved via the run outputs.
+test_run.download_file("predictions/predictions.csv")
+predictions_df = pd.read_csv("predictions.csv")
+
+```
+
+### Test existing automated ML model
+
+To test other existing automated ML models created, best run or child run, use [`ModelProxy()`](/python/api/azureml-train-automl-client/azureml.train.automl.model_proxy.modelproxy) to test a model after the main AutoML run has completed. `ModelProxy()` already returns the predictions and metrics and does not require further processing to retrieve the outputs.
+
+> [!NOTE]
+> ModelProxy is an [experimental](/python/api/overview/azure/ml/#stable-vs-experimental) preview class, and may change at any time.
+
+The following code demonstrates how to test a model from any run by using [ModelProxy.test()](/python/api/azureml-train-automl-client/azureml.train.automl.model_proxy.modelproxy#test-test-data--azureml-data-abstract-dataset-abstractdataset--include-predictions-only--bool---false-----typing-tuple-azureml-data-abstract-dataset-abstractdataset--typing-dict-str--typing-any--) method. In the test() method you have the option to specify if you only want to see the predictions of the test run with the `include_predictions_only` parameter. 
+
+```python
+from azureml.train.automl.model_proxy import ModelProxy
+
+model_proxy = ModelProxy(child_run=my_run, compute_target=cpu_cluster)
+predictions, metrics = model_proxy.test(test_data, include_predictions_only= True
+)
+```
+
 ## Register and deploy models
 
-You can register a model, so you can come back to it for later use. 
+After you test a model and confirm you want to use it in production, you can register it for later use and 
 
 To register a model from an automated ML run, use the [`register_model()`](/python/api/azureml-train-automl-client/azureml.train.automl.run.automlrun#register-model-model-name-none--description-none--tags-none--iteration-none--metric-none-) method.