diff --git a/.jenkins/validate_tutorials_built.py b/.jenkins/validate_tutorials_built.py index 5c9e60e90bd..e02ea6028f7 100644 --- a/.jenkins/validate_tutorials_built.py +++ b/.jenkins/validate_tutorials_built.py @@ -48,7 +48,6 @@ "recipes_source/recipes/timer_quick_start", "recipes_source/recipes/amp_recipe", "recipes_source/recipes/Captum_Recipe", - "intermediate_source/flask_rest_api_tutorial", "intermediate_source/text_to_speech_with_torchaudio", "intermediate_source/tensorboard_profiler_tutorial", # reenable after 2.0 release. "advanced_source/semi_structured_sparse", # reenable after 3303 is fixed. diff --git a/.lycheeignore b/.lycheeignore index 3d86ae872de..e3113ca5152 100644 --- a/.lycheeignore +++ b/.lycheeignore @@ -13,8 +13,5 @@ https://pytorch.org/tutorials/beginner/colab/n # Ignore local host link from intermediate_source/tensorboard_tutorial.rst http://localhost:6006 -# Ignore local host link from recipes_source/deployment_with_flask.rst -http://localhost:5000/predict - # Ignore local host link from advanced_source/cpp_frontend.rst https://www.uber.com/blog/deep-neuroevolution/ diff --git a/index.rst b/index.rst index 54bd8b37466..2d202e0fe57 100644 --- a/index.rst +++ b/index.rst @@ -322,14 +322,6 @@ Welcome to PyTorch Tutorials .. Deploying PyTorch Models in Production - -.. customcarditem:: - :header: Deploying PyTorch in Python via a REST API with Flask - :card_description: Deploy a PyTorch model using Flask and expose a REST API for model inference using the example of a pretrained DenseNet 121 model which detects the image. - :image: _static/img/thumbnails/cropped/Deploying-PyTorch-in-Python-via-a-REST-API-with-Flask.png - :link: intermediate/flask_rest_api_tutorial.html - :tags: Production - .. customcarditem:: :header: Introduction to TorchScript :card_description: Introduction to TorchScript, an intermediate representation of a PyTorch model (subclass of nn.Module) that can then be run in a high-performance environment such as C++. @@ -1005,7 +997,6 @@ Additional Resources :caption: Deploying PyTorch Models in Production beginner/onnx/intro_onnx - intermediate/flask_rest_api_tutorial beginner/Intro_to_TorchScript_tutorial advanced/cpp_export advanced/super_resolution_with_onnxruntime diff --git a/intermediate_source/README.txt b/intermediate_source/README.txt index 0307e89a1a9..ecc8eb74af4 100644 --- a/intermediate_source/README.txt +++ b/intermediate_source/README.txt @@ -29,10 +29,6 @@ Intermediate tutorials Spatial Transformer Networks Tutorial https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html -8. flask_rest_api_tutorial.py - Deploying PyTorch and Building a REST API using Flask - https://pytorch.org/tutorials/intermediate/flask_rest_api_tutorial.html - -9. nvfuser_intro_tutorial.py +8. nvfuser_intro_tutorial.py Introduction to nvFuser https://pytorch.org/tutorials/intermediate/nvfuser_intro_tutorial.html diff --git a/intermediate_source/flask_rest_api_tutorial.py b/intermediate_source/flask_rest_api_tutorial.py deleted file mode 100644 index b6b46aed91d..00000000000 --- a/intermediate_source/flask_rest_api_tutorial.py +++ /dev/null @@ -1,333 +0,0 @@ -# -*- coding: utf-8 -*- -""" -Deploying PyTorch in Python via a REST API with Flask -======================================================== -**Author**: `Avinash Sajjanshetty `_ - -In this tutorial, we will deploy a PyTorch model using Flask and expose a -REST API for model inference. In particular, we will deploy a pretrained -DenseNet 121 model which detects the image. - -.. tip:: All the code used here is released under MIT license and is available on `Github `_. - -This represents the first in a series of tutorials on deploying PyTorch models -in production. Using Flask in this way is by far the easiest way to start -serving your PyTorch models, but it will not work for a use case -with high performance requirements. For that: - - - If you're already familiar with TorchScript, you can jump straight into our - `Loading a TorchScript Model in C++ `_ tutorial. - - - If you first need a refresher on TorchScript, check out our - `Intro a TorchScript `_ tutorial. -""" - - -###################################################################### -# API Definition -# -------------- -# -# We will first define our API endpoints, the request and response types. Our -# API endpoint will be at ``/predict`` which takes HTTP POST requests with a -# ``file`` parameter which contains the image. The response will be of JSON -# response containing the prediction: -# -# .. code-block:: sh -# -# {"class_id": "n02124075", "class_name": "Egyptian_cat"} -# -# - -###################################################################### -# Dependencies -# ------------ -# -# Install the required dependencies by running the following command: -# -# .. code-block:: sh -# -# pip install Flask==2.0.1 torchvision==0.10.0 - - -###################################################################### -# Simple Web Server -# ----------------- -# -# Following is a simple web server, taken from Flask's documentation - - -from flask import Flask -app = Flask(__name__) - - -@app.route('/') -def hello(): - return 'Hello World!' - -############################################################################### -# We will also change the response type, so that it returns a JSON response -# containing ImageNet class id and name. The updated ``app.py`` file will -# be now: - -from flask import Flask, jsonify -app = Flask(__name__) - -@app.route('/predict', methods=['POST']) -def predict(): - return jsonify({'class_id': 'IMAGE_NET_XXX', 'class_name': 'Cat'}) - - -###################################################################### -# Inference -# ----------------- -# -# In the next sections we will focus on writing the inference code. This will -# involve two parts, one where we prepare the image so that it can be fed -# to DenseNet and next, we will write the code to get the actual prediction -# from the model. -# -# Preparing the image -# ~~~~~~~~~~~~~~~~~~~ -# -# DenseNet model requires the image to be of 3 channel RGB image of size -# 224 x 224. We will also normalize the image tensor with the required mean -# and standard deviation values. You can read more about it -# `here `_. -# -# We will use ``transforms`` from ``torchvision`` library and build a -# transform pipeline, which transforms our images as required. You -# can read more about transforms `here `_. - -import io - -import torchvision.transforms as transforms -from PIL import Image - -def transform_image(image_bytes): - my_transforms = transforms.Compose([transforms.Resize(255), - transforms.CenterCrop(224), - transforms.ToTensor(), - transforms.Normalize( - [0.485, 0.456, 0.406], - [0.229, 0.224, 0.225])]) - image = Image.open(io.BytesIO(image_bytes)) - return my_transforms(image).unsqueeze(0) - -###################################################################### -# The above method takes image data in bytes, applies the series of transforms -# and returns a tensor. To test the above method, read an image file in -# bytes mode (first replacing `../_static/img/sample_file.jpeg` with the actual -# path to the file on your computer) and see if you get a tensor back: - -with open("../_static/img/sample_file.jpeg", 'rb') as f: - image_bytes = f.read() - tensor = transform_image(image_bytes=image_bytes) - print(tensor) - -###################################################################### -# Prediction -# ~~~~~~~~~~~~~~~~~~~ -# -# Now will use a pretrained DenseNet 121 model to predict the image class. We -# will use one from ``torchvision`` library, load the model and get an -# inference. While we'll be using a pretrained model in this example, you can -# use this same approach for your own models. See more about loading your -# models in this :doc:`tutorial `. - -from torchvision import models - -# Make sure to set `weights` as `'IMAGENET1K_V1'` to use the pretrained weights: -model = models.densenet121(weights='IMAGENET1K_V1') -# Since we are using our model only for inference, switch to `eval` mode: -model.eval() - - -def get_prediction(image_bytes): - tensor = transform_image(image_bytes=image_bytes) - outputs = model.forward(tensor) - _, y_hat = outputs.max(1) - return y_hat - -###################################################################### -# The tensor ``y_hat`` will contain the index of the predicted class id. -# However, we need a human readable class name. For that we need a class id -# to name mapping. Download -# `this file `_ -# as ``imagenet_class_index.json`` and remember where you saved it (or, if you -# are following the exact steps in this tutorial, save it in -# `tutorials/_static`). This file contains the mapping of ImageNet class id to -# ImageNet class name. We will load this JSON file and get the class name of -# the predicted index. - -import json - -imagenet_class_index = json.load(open('../_static/imagenet_class_index.json')) - -def get_prediction(image_bytes): - tensor = transform_image(image_bytes=image_bytes) - outputs = model.forward(tensor) - _, y_hat = outputs.max(1) - predicted_idx = str(y_hat.item()) - return imagenet_class_index[predicted_idx] - - -###################################################################### -# Before using ``imagenet_class_index`` dictionary, first we will convert -# tensor value to a string value, since the keys in the -# ``imagenet_class_index`` dictionary are strings. -# We will test our above method: - - -with open("../_static/img/sample_file.jpeg", 'rb') as f: - image_bytes = f.read() - print(get_prediction(image_bytes=image_bytes)) - -###################################################################### -# You should get a response like this: - -['n02124075', 'Egyptian_cat'] - -###################################################################### -# The first item in array is ImageNet class id and second item is the human -# readable name. -# - -###################################################################### -# Integrating the model in our API Server -# --------------------------------------- -# -# In this final part we will add our model to our Flask API server. Since -# our API server is supposed to take an image file, we will update our ``predict`` -# method to read files from the requests: -# -# .. code-block:: python -# -# from flask import request -# -# @app.route('/predict', methods=['POST']) -# def predict(): -# if request.method == 'POST': -# # we will get the file from the request -# file = request.files['file'] -# # convert that to bytes -# img_bytes = file.read() -# class_id, class_name = get_prediction(image_bytes=img_bytes) -# return jsonify({'class_id': class_id, 'class_name': class_name}) -# -# -###################################################################### -# The ``app.py`` file is now complete. Following is the full version; replace -# the paths with the paths where you saved your files and it should run: -# -# .. code-block:: python -# -# import io -# import json -# -# from torchvision import models -# import torchvision.transforms as transforms -# from PIL import Image -# from flask import Flask, jsonify, request -# -# -# app = Flask(__name__) -# imagenet_class_index = json.load(open('/imagenet_class_index.json')) -# model = models.densenet121(weights='IMAGENET1K_V1') -# model.eval() -# -# -# def transform_image(image_bytes): -# my_transforms = transforms.Compose([transforms.Resize(255), -# transforms.CenterCrop(224), -# transforms.ToTensor(), -# transforms.Normalize( -# [0.485, 0.456, 0.406], -# [0.229, 0.224, 0.225])]) -# image = Image.open(io.BytesIO(image_bytes)) -# return my_transforms(image).unsqueeze(0) -# -# -# def get_prediction(image_bytes): -# tensor = transform_image(image_bytes=image_bytes) -# outputs = model.forward(tensor) -# _, y_hat = outputs.max(1) -# predicted_idx = str(y_hat.item()) -# return imagenet_class_index[predicted_idx] -# -# -# @app.route('/predict', methods=['POST']) -# def predict(): -# if request.method == 'POST': -# file = request.files['file'] -# img_bytes = file.read() -# class_id, class_name = get_prediction(image_bytes=img_bytes) -# return jsonify({'class_id': class_id, 'class_name': class_name}) -# -# -# if __name__ == '__main__': -# app.run() -# -# -###################################################################### -# Let's test our web server! Run: -# -# .. code-block:: sh -# -# FLASK_ENV=development FLASK_APP=app.py flask run -# -####################################################################### -# We can use the -# `requests `_ -# library to send a POST request to our app: -# -# .. code-block:: python -# -# import requests -# -# resp = requests.post("http://localhost:5000/predict", -# files={"file": open('/cat.jpg','rb')}) -# - -####################################################################### -# Printing `resp.json()` will now show the following: -# -# .. code-block:: sh -# -# {"class_id": "n02124075", "class_name": "Egyptian_cat"} -# -###################################################################### -# Next steps -# -------------- -# -# The server we wrote is quite trivial and may not do everything -# you need for your production application. So, here are some things you -# can do to make it better: -# -# - The endpoint ``/predict`` assumes that always there will be a image file -# in the request. This may not hold true for all requests. Our user may -# send image with a different parameter or send no images at all. -# -# - The user may send non-image type files too. Since we are not handling -# errors, this will break our server. Adding an explicit error handing -# path that will throw an exception would allow us to better handle -# the bad inputs -# -# - Even though the model can recognize a large number of classes of images, -# it may not be able to recognize all images. Enhance the implementation -# to handle cases when the model does not recognize anything in the image. -# -# - We run the Flask server in the development mode, which is not suitable for -# deploying in production. You can check out `this tutorial `_ -# for deploying a Flask server in production. -# -# - You can also add a UI by creating a page with a form which takes the image and -# displays the prediction. -# - In this tutorial, we only showed how to build a service that could return predictions for -# a single image at a time. We could modify our service to be able to return predictions for -# multiple images at once. In addition, the `service-streamer `_ -# library automatically queues requests to your service and samples them into mini-batches -# that can be fed into your model. You can check out `this tutorial `_. -# -# - Finally, we encourage you to check out our other tutorials on deploying PyTorch models -# linked-to at the top of the page. -# diff --git a/recipes_source/deployment_with_flask.rst b/recipes_source/deployment_with_flask.rst deleted file mode 100644 index 213a326429c..00000000000 --- a/recipes_source/deployment_with_flask.rst +++ /dev/null @@ -1,284 +0,0 @@ -Deploying with Flask -==================== - -In this recipe, you will learn: - -- How to wrap your trained PyTorch model in a Flask container to expose - it via a web API -- How to translate incoming web requests into PyTorch tensors for your - model -- How to package your model’s output for an HTTP response - -Requirements ------------- - -You will need a Python 3 environment with the following packages (and -their dependencies) installed: - -- PyTorch 1.5 -- TorchVision 0.6.0 -- Flask 1.1 - -Optionally, to get some of the supporting files, you'll need git. - -The instructions for installing PyTorch and TorchVision are available at -`pytorch.org`_. Instructions for installing Flask are available on `the -Flask site`_. - -What is Flask? --------------- - -Flask is a lightweight web server written in Python. It provides a -convenient way for you to quickly set up a web API for predictions from -your trained PyTorch model, either for direct use, or as a web service -within a larger system. - -Setup and Supporting Files --------------------------- - -We're going to create a web service that takes in images, and maps them -to one of the 1000 classes of the ImageNet dataset. To do this, you'll -need an image file for testing. Optionally, you can also get a file that -will map the class index output by the model to a human-readable class -name. - -Option 1: To Get Both Files Quickly -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can pull both of the supporting files quickly by checking out the -TorchServe repository and copying them to your working folder. *(NB: -There is no dependency on TorchServe for this tutorial - it's just a -quick way to get the files.)* Issue the following commands from your -shell prompt: - -:: - - git clone https://github.com/pytorch/serve - cp serve/examples/image_classifier/kitten.jpg . - cp serve/examples/image_classifier/index_to_name.json . - -And you've got them! - -Option 2: Bring Your Own Image -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The ``index_to_name.json`` file is optional in the Flask service below. -You can test your service with your own image - just make sure it's a -3-color JPEG. - -Building Your Flask Service ---------------------------- - -The full Python script for the Flask service is shown at the end of this -recipe; you can copy and paste that into your own ``app.py`` file. Below -we'll look at individual sections to make their functions clear. - -Imports -~~~~~~~ - -:: - - import torchvision.models as models - import torchvision.transforms as transforms - from PIL import Image - from flask import Flask, jsonify, request - -In order: - -- We'll be using a pre-trained DenseNet model from - ``torchvision.models`` -- ``torchvision.transforms`` contains tools for manipulating your image - data -- Pillow (``PIL``) is what we'll use to load the image file initially -- And of course we'll need classes from ``flask`` - -Pre-Processing -~~~~~~~~~~~~~~ - -:: - - def transform_image(infile): - input_transforms = [transforms.Resize(255), - transforms.CenterCrop(224), - transforms.ToTensor(), - transforms.Normalize([0.485, 0.456, 0.406], - [0.229, 0.224, 0.225])] - my_transforms = transforms.Compose(input_transforms) - image = Image.open(infile) - timg = my_transforms(image) - timg.unsqueeze_(0) - return timg - -The web request gave us an image file, but our model expects a PyTorch -tensor of shape (N, 3, 224, 224) where *N* is the number of items in the -input batch. (We will just have a batch size of 1.) The first thing we -do is compose a set of TorchVision transforms that resize and crop the -image, convert it to a tensor, then normalize the values in the tensor. -(For more information on this normalization, see the documentation for -``torchvision.models_``.) - -After that, we open the file and apply the transforms. The transforms -return a tensor of shape (3, 224, 224) - the 3 color channels of a -224x224 image. Because we need to make this single image a batch, we use -the ``unsqueeze_(0)`` call to modify the tensor in place by adding a new -first dimension. The tensor contains the same data, but now has shape -(1, 3, 224, 224). - -In general, even if you're not working with image data, you will need to -transform the input from your HTTP request into a tensor that PyTorch -can consume. - -Inference -~~~~~~~~~ - -:: - - def get_prediction(input_tensor): - outputs = model.forward(input_tensor) - _, y_hat = outputs.max(1) - prediction = y_hat.item() - return prediction - -The inference itself is the simplest part: When we pass the input tensor -to them model, we get back a tensor of values that represent the model's -estimated likelihood that the image belongs to a particular class. The -``max()`` call finds the class with the maximum likelihood value, and -returns that value with the ImageNet class index. Finally, we extract -that class index from the tensor containing it with the ``item()`` call, and -return it. - -Post-Processing -~~~~~~~~~~~~~~~ - -:: - - def render_prediction(prediction_idx): - stridx = str(prediction_idx) - class_name = 'Unknown' - if img_class_map is not None: - if stridx in img_class_map is not None: - class_name = img_class_map[stridx][1] - - return prediction_idx, class_name - -The ``render_prediction()`` method maps the predicted class index to a -human-readable class label. It's typical, after getting the prediction -from your model, to perform post-processing to make the prediction ready -for either human consumption, or for another piece of software. - -Running The Full Flask App --------------------------- - -Paste the following into a file called ``app.py``: - -:: - - import io - import json - import os - - import torchvision.models as models - import torchvision.transforms as transforms - from PIL import Image - from flask import Flask, jsonify, request - - - app = Flask(__name__) - model = models.densenet121(pretrained=True) # Trained on 1000 classes from ImageNet - model.eval() # Turns off autograd - - - - img_class_map = None - mapping_file_path = 'index_to_name.json' # Human-readable names for Imagenet classes - if os.path.isfile(mapping_file_path): - with open (mapping_file_path) as f: - img_class_map = json.load(f) - - - - # Transform input into the form our model expects - def transform_image(infile): - input_transforms = [transforms.Resize(255), # We use multiple TorchVision transforms to ready the image - transforms.CenterCrop(224), - transforms.ToTensor(), - transforms.Normalize([0.485, 0.456, 0.406], # Standard normalization for ImageNet model input - [0.229, 0.224, 0.225])] - my_transforms = transforms.Compose(input_transforms) - image = Image.open(infile) # Open the image file - timg = my_transforms(image) # Transform PIL image to appropriately-shaped PyTorch tensor - timg.unsqueeze_(0) # PyTorch models expect batched input; create a batch of 1 - return timg - - - # Get a prediction - def get_prediction(input_tensor): - outputs = model.forward(input_tensor) # Get likelihoods for all ImageNet classes - _, y_hat = outputs.max(1) # Extract the most likely class - prediction = y_hat.item() # Extract the int value from the PyTorch tensor - return prediction - - # Make the prediction human-readable - def render_prediction(prediction_idx): - stridx = str(prediction_idx) - class_name = 'Unknown' - if img_class_map is not None: - if stridx in img_class_map is not None: - class_name = img_class_map[stridx][1] - - return prediction_idx, class_name - - - @app.route('/', methods=['GET']) - def root(): - return jsonify({'msg' : 'Try POSTing to the /predict endpoint with an RGB image attachment'}) - - - @app.route('/predict', methods=['POST']) - def predict(): - if request.method == 'POST': - file = request.files['file'] - if file is not None: - input_tensor = transform_image(file) - prediction_idx = get_prediction(input_tensor) - class_id, class_name = render_prediction(prediction_idx) - return jsonify({'class_id': class_id, 'class_name': class_name}) - - - if __name__ == '__main__': - app.run() - -To start the server from your shell prompt, issue the following command: - -:: - - FLASK_APP=app.py flask run - -By default, your Flask server is listening on port 5000. Once the server -is running, open another terminal window, and test your new inference -server: - -:: - - curl -X POST -H "Content-Type: multipart/form-data" http://localhost:5000/predict -F "file=@kitten.jpg" - -If everything is set up correctly, you should recevie a response similar -to the following: - -:: - - {"class_id":285,"class_name":"Egyptian_cat"} - -Important Resources -------------------- - -- `pytorch.org`_ for installation instructions, and more documentation - and tutorials -- The `Flask site`_ has a `Quick Start guide`_ that goes into more - detail on setting up a simple Flask service - -.. _pytorch.org: https://pytorch.org -.. _Flask site: https://flask.palletsprojects.com/en/1.1.x/ -.. _Quick Start guide: https://flask.palletsprojects.com/en/1.1.x/quickstart/ -.. _torchvision.models: https://pytorch.org/vision/stable/models.html -.. _the Flask site: https://flask.palletsprojects.com/en/1.1.x/installation/ diff --git a/recipes_source/recipes_index.rst b/recipes_source/recipes_index.rst index ce76e1931c2..bdafd705264 100644 --- a/recipes_source/recipes_index.rst +++ b/recipes_source/recipes_index.rst @@ -206,13 +206,6 @@ Recipes are bite-sized, actionable examples of how to use specific PyTorch featu :link: ../recipes/torchscript_inference.html :tags: TorchScript -.. customcarditem:: - :header: Deploying with Flask - :card_description: Learn how to use Flask, a lightweight web server, to quickly setup a web API from your trained PyTorch model. - :image: ../_static/img/thumbnails/cropped/using-flask-create-restful-api.png - :link: ../recipes/deployment_with_flask.html - :tags: Production,TorchScript - .. customcarditem:: :header: PyTorch Mobile Performance Recipes :card_description: List of recipes for performance optimizations for using PyTorch on Mobile (Android and iOS). @@ -505,7 +498,6 @@ Recipes are bite-sized, actionable examples of how to use specific PyTorch featu /recipes/compiling_optimizer /recipes/torch_compile_backend_ipex /recipes/torchscript_inference - /recipes/deployment_with_flask /recipes/distributed_rpc_profiling /recipes/zero_redundancy_optimizer /recipes/cuda_rpc