Merge pull request #204766 from shivanissambare/inference-updates

Inference image and server updates

Author: Stacyrch140

articles/machine-learning/concept-prebuilt-docker-images-inference.md

@@ -7,19 +7,16 @@ ms.service: machine-learning
 ms.subservice: core
 ms.author: ssambare
 author: shivanissambare
-ms.date: 10/21/2021
+ms.date: 07/14/2022
 ms.topic: conceptual
 ms.reviewer: larryfr
-ms.custom: deploy, docker, prebuilt, curated environments
+ms.custom: deploy, docker, prebuilt
 ---
 
 # Prebuilt Docker images for inference
 
 Prebuilt Docker container images for inference are used when deploying a model with Azure Machine Learning.  The images are prebuilt with popular machine learning frameworks and Python packages. You can also extend the packages to add other packages by using one of the following methods:
 
-* [Add Python packages](how-to-prebuilt-docker-images-inference-python-extensibility.md).
-* [Use prebuilt inference image as base for a new Dockerfile](how-to-extend-prebuilt-docker-image-inference.md). Using this method, you can install both **Python packages and apt packages**.
-
 ## Why should I use prebuilt images?
 
 * Reduces model deployment latency.
@@ -29,9 +26,17 @@ Prebuilt Docker container images for inference are used when deploying a model w
 
 ## List of prebuilt Docker images for inference 
 
+> [!IMPORTANT]
+> The list provided below includes only **currently supported** inference docker images by Azure Machine Learning.
+
 [!INCLUDE [list-of-inference-prebuilt-docker-images](../../includes/aml-inference-list-prebuilt-docker-images.md)]
 
+## How to use inference prebuilt docker images?
+
+[Check examples in the Azure machine learning GitHub repository](https://github.com/Azure/azureml-examples/tree/main/cli/endpoints/online/custom-container)
+
 ## Next steps
 
-* [Add Python packages to prebuilt images](how-to-prebuilt-docker-images-inference-python-extensibility.md).
-* [Use a prebuilt package as a base for a new Dockerfile](how-to-extend-prebuilt-docker-image-inference.md).
+* [Deploy and score a machine learning model by using an online endpoint](how-to-deploy-managed-online-endpoints.md)
+* [Learn more about custom containers](how-to-deploy-custom-container.md)
+* [azureml-examples GitHub repository](https://github.com/Azure/azureml-examples/tree/main/cli/endpoints/online)

articles/machine-learning/how-to-inference-server-http.md

@@ -10,7 +10,7 @@ ms.service: machine-learning
 ms.subservice: core
 ms.topic: how-to
 ms.custom: inference server, local development, local debugging, devplatv2
-ms.date: 05/14/2021
+ms.date: 07/14/2022
 ---
 
 # Azure Machine Learning inference HTTP server (preview)
@@ -135,19 +135,57 @@ There are two ways to use Visual Studio Code (VSCode) and [Python Extension](htt
 1. User starts the AzureML Inference Server in a command line and use VSCode + Python Extension to attach to the process.
 1. User sets up the `launch.json` in the VSCode and start the AzureML Inference Server within VSCode.
 
+**launch.json**
+```json
+{
+    "name": "Debug score.py",
+    "type": "python",
+    "request": "launch",
+    "module": "azureml_inference_server_http.amlserver",
+    "args": [
+        "--entry_script",
+        "score.py"
+    ]
+}
+```
+
 In both ways, user can set breakpoint and debug step by step.
 
 ## Frequently asked questions
 
-### Do I need to reload the server when changing the score script?
+### 1. I encountered the following error during server startup:
+
+```bash
+
+TypeError: register() takes 3 positional arguments but 4 were given
+
+  File "/var/azureml-server/aml_blueprint.py", line 251, in register
+
+    super(AMLBlueprint, self).register(app, options, first_registration)
+
+TypeError: register() takes 3 positional arguments but 4 were given
+
+```
+
+You have **Flask 2** installed in your python environment but are running a server (< 7.0.0) that does not support Flask 2. To resolve, please upgrade to the latest version of server.
+
+### 2. I encountered an ``ImportError`` or ``ModuleNotFoundError`` on modules ``opencensus``, ``jinja2``, ``MarkupSafe``, or ``click`` during startup like the following:
+
+```bash
+ImportError: cannot import name 'Markup' from 'jinja2'
+```
+
+Older versions (<= 0.4.10) of the server did not pin Flask's dependency to compatible versions. This is fixed in the latest version of the server.
+
+### 3. Do I need to reload the server when changing the score script?
 
 After changing your scoring script (`score.py`), stop the server with `ctrl + c`. Then restart it with `azmlinfsrv --entry_script score.py`.
 
-### Which OS is supported?
+### 4. Which OS is supported?
 
 The Azure Machine Learning inference server runs on Windows & Linux based operating systems.
 
 ## Next steps
 
-* For more information on creating an entry script and deploying models, see [How to deploy a model using Azure Machine Learning](how-to-deploy-and-where.md).
+* For more information on creating an entry script and deploying models, see [How to deploy a model using Azure Machine Learning](how-to-deploy-managed-online-endpoints.md).
 * Learn about [Prebuilt docker images for inference](concept-prebuilt-docker-images-inference.md)

articles/machine-learning/resource-curated-environments.md

@@ -29,7 +29,7 @@ This article lists the curated environments with latest framework versions in Az
 >[!IMPORTANT] 
 > To view more information about curated environment packages and versions, visit the Environments tab in the Azure Machine Learning [studio](./how-to-manage-environments-in-studio.md). 
 
-## Training curated environments
+## Curated environments
 
 ### PyTorch
 
@@ -89,9 +89,5 @@ Azure ML pipeline training workflows that use AutoML automatically selects a cur
 
 For more information on AutoML and Azure ML pipelines, see [use automated ML in an Azure Machine Learning pipeline in Python](how-to-use-automlstep-in-pipelines.md).
 
-## Inference curated environments and prebuilt docker images
-
-[!INCLUDE [list-of-inference-prebuilt-docker-images](../../includes/aml-inference-list-prebuilt-docker-images.md)]
-
 ## Support
 Version updates for supported environments, including the base images they reference, are released every two weeks to address vulnerabilities no older than 30 days. Based on usage, some environments may be deprecated (hidden from the product but usable) to support more common machine learning scenarios.

includes/aml-inference-list-prebuilt-docker-images.md

@@ -7,47 +7,17 @@ ms.service: machine-learning
 ms.author: ssambare
 ms.custom: "include file"
 ms.topic: "include"
-ms.date: 10/07/2021
+ms.date: 07/14/2022
 ---
 
 * All the docker images run as non-root user.
-* We recommend using `latest` tag for docker images. Prebuilt docker images for inference are published to Microsoft container registry (MCR), to query list of tags available, follow [instructions on their GitHub repository](https://github.com/microsoft/ContainerRegistry#browsing-mcr-content).
+* We recommend using `latest` tag for docker images. Prebuilt docker images for inference are published to Microsoft container registry (MCR), to query list of tags available, follow [instructions on the GitHub repository](https://github.com/microsoft/ContainerRegistry#browsing-mcr-content).
+* If you want to use a specific tag for any inference docker image, we support from `latest` to the tag that is *6 months* old from the `latest`.  
 
-### TensorFlow
+### Inference minimal base images
 
-Framework version | CPU/GPU | Pre-installed packages | MCR Path | Curated environment
- --- | --- | --- | --- | --- |
- 1.15 | CPU | pandas==0.25.1 </br> numpy=1.20.1 | `mcr.microsoft.com/azureml/tensorflow-1.15-ubuntu18.04-py37-cpu-inference:latest`  | AzureML-tensorflow-1.15-ubuntu18.04-py37-cpu-inference | 
-2.4 | CPU | numpy>=1.16.0 </br> pandas~=1.1.x | `mcr.microsoft.com/azureml/tensorflow-2.4-ubuntu18.04-py37-cpu-inference:latest` | AzureML-tensorflow-2.4-ubuntu18.04-py37-cpu-inference |
-2.4 | GPU | numpy >= 1.16.0 </br> pandas~=1.1.x </br> CUDA==11.0.3 </br> CuDNN==8.0.5.39 | `mcr.microsoft.com/azureml/tensorflow-2.4-ubuntu18.04-py37-cuda11.0.3-gpu-inference:latest` | AzureML-tensorflow-2.4-ubuntu18.04-py37-cuda11.0.3-gpu-inference |
-
-### PyTorch
-
-Framework version | CPU/GPU | Pre-installed packages | MCR Path | Curated environment
- --- | --- | --- | --- | --- |
- 1.6 | CPU | numpy==1.20.1 </br> pandas==0.25.1 | `mcr.microsoft.com/azureml/pytorch-1.6-ubuntu18.04-py37-cpu-inference:latest` | AzureML-pytorch-1.6-ubuntu18.04-py37-cpu-inference |
-1.7 | CPU | numpy>=1.16.0 </br> pandas~=1.1.x | `mcr.microsoft.com/azureml/pytorch-1.7-ubuntu18.04-py37-cpu-inference:latest` | AzureML-pytorch-1.7-ubuntu18.04-py37-cpu-inference |
-
-### SciKit-Learn
-
-Framework version | CPU/GPU | Pre-installed packages | MCR Path | Curated environment
- --- | --- | --- | --- | --- |
-0.24.1  | CPU | scikit-learn==0.24.1 </br> numpy>=1.16.0 </br> pandas~=1.1.x | `mcr.microsoft.com/azureml/sklearn-0.24.1-ubuntu18.04-py37-cpu-inference:latest` | AzureML-sklearn-0.24.1-ubuntu18.04-py37-cpu-inference |
-
-### ONNX Runtime
-
-Framework version | CPU/GPU | Pre-installed packages | MCR Path | Curated environment
- --- | --- | --- | --- | --- |
-1.6 | CPU | numpy>=1.16.0 </br> pandas~=1.1.x | `mcr.microsoft.com/azureml/onnxruntime-1.6-ubuntu18.04-py37-cpu-inference:latest` |AzureML-onnxruntime-1.6-ubuntu18.04-py37-cpu-inference |
-
-### XGBoost
-
-Framework version | CPU/GPU | Pre-installed packages | MCR Path | Curated environment
- --- | --- | --- | --- | --- |
-0.9 | CPU | scikit-learn==0.23.2 </br> numpy==1.20.1 </br> pandas==0.25.1 | `mcr.microsoft.com/azureml/xgboost-0.9-ubuntu18.04-py37-cpu-inference:latest` | AzureML-xgboost-0.9-ubuntu18.04-py37-cpu-inference | 
-
-### No framework
-
-Framework version | CPU/GPU | Pre-installed packages | MCR Path | Curated environment
- --- | --- | --- | --- | --- |
-NA | CPU | NA | `mcr.microsoft.com/azureml/minimal-ubuntu18.04-py37-cpu-inference:latest` | AzureML-minimal-ubuntu18.04-py37-cpu-inference  |
+Framework version | CPU/GPU | Pre-installed packages | MCR Path
+ --- | --- | --- | --- |
+NA | CPU | NA | `mcr.microsoft.com/azureml/minimal-ubuntu18.04-py37-cpu-inference:latest`
+NA | GPU | NA | `mcr.microsoft.com/azureml/minimal-ubuntu18.04-py37-cuda11.0.3-gpu-inference:latest`
+NA | CPU | NA | `mcr.microsoft.com/azureml/minimal-ubuntu20.04-py38-cpu-inference:latest`