docs: General edits and reorganization, render notebooks in the

documentation

Signed-off-by: Naren Dasan <[email protected]>
Signed-off-by: Naren Dasan <[email protected]>

Author: narendasan

docsrc/Makefile

@@ -31,11 +31,14 @@ ifndef VERSION
 	rm -rf /tmp/trtorch_docs
 endif
 	rm -rf $(SOURCEDIR)/_cpp_api
+	rm -rf $(SOURCEDIR)/_notebooks
 	rm -rf $(SOURCEDIR)/_py_api
 	rm -rf $(SOURCEDIR)/_build
 	rm -rf $(SOURCEDIR)/_tmp
 
 html:
+	mkdir -p $(SOURCEDIR)/_notebooks
+	cp -r $(SOURCEDIR)/../notebooks/*.ipynb $(SOURCEDIR)/_notebooks
 	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 	mkdir -p $(DESTDIR)
 	cp -r $(BUILDDIR)/html/* $(DESTDIR)

docsrc/README.md

@@ -1,9 +1,9 @@
 # Building the Documentation
 
-We use Sphinx and Doxygen for documentation, so begin by installing the dependencies:
+We use Sphinx, Doxygen and pandoc for documentation, so begin by installing the dependencies:
 
 ```
-apt install doxygen
+apt install doxygen pandoc
 ```
 
 ```

docsrc/conf.py

@@ -33,6 +33,7 @@
 extensions = [
     'breathe',
     'exhale',
+    'nbsphinx',
     'sphinx.ext.napoleon',
     'sphinx.ext.intersphinx',
     'sphinx.ext.autosummary',
@@ -115,7 +116,7 @@
     'repo_name': 'TRTorch',
 
     # Visible levels of the global TOC; -1 means unlimited
-    'globaltoc_depth': 2,
+    'globaltoc_depth': 1,
     # If False, expand all TOC entries
     'globaltoc_collapse': False,
     # If True, show hidden TOC entries

docsrc/contributors/system_overview.rst

@@ -12,9 +12,13 @@ The repository is structured into:
 * core: Main compiler source code
 * cpp: C++ API
 * tests: tests of the C++ API, the core and converters
+* py: Python API
+* notebooks: Example applications built with TRTorch
 * docs: Documentation
 * docsrc: Documentation Source
 * third_party: BUILD files for dependency libraries
+* toolchains: Toolchains for different platforms
+
 
 The C++ API is unstable and subject to change until the library matures, though most work is done under the hood in the core.
 

docsrc/index.rst

@@ -36,6 +36,19 @@ Getting Started
    tutorials/getting_started
    tutorials/ptq
    tutorials/trtorchc
+   _notebooks/lenet
+
+Notebooks
+------------
+* :ref:`lenet`
+
+.. toctree::
+   :caption: Notebooks
+   :maxdepth: 1
+   :hidden:
+
+   _notebooks/lenet-getting-started
+
 
 Python API Documenation
 ------------------------

docsrc/requirements.txt

@@ -1,5 +1,6 @@
-sphinx>=2.0
-breathe>=4.13.0
+sphinx==3.1.2
+breathe==4.19.2
 exhale
 sphinx_rtd_theme==0.4.3
-sphinx-material>=0.0.29
+sphinx-material==0.0.30
+nbsphinx==0.7.1

docsrc/tutorials/getting_started.rst

@@ -5,15 +5,18 @@ Getting Started
 
 If you haven't already, aquire a tarball of the library by following the instructions in :ref:`Installation`
 
+Background
+*********************
+
 .. _creating_a_ts_mod:
 Creating a TorchScript Module
 ------------------------------
 
 Once you have a trained model you want to compile with TRTorch, you need to start by converting that model from Python code to TorchScript code.
 PyTorch has detailed documentation on how to do this https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html but briefly here is the
-here is key background and the process:
+here is key background information and the process:
 
-PyTorch programs are based around `Module`s which can be used to compose higher level modules. Modules contain a constructor to set up the modules, parameters and sub-modules
+PyTorch programs are based around ``Module`` s which can be used to compose higher level modules. ``Modules`` contain a constructor to set up the modules, parameters and sub-modules
 and a forward function which describes how to use the parameters and submodules when the module is invoked.
 
 For example, we can define a LeNet module like this:
@@ -130,12 +133,62 @@ TorchScript Modules are run the same way you run normal PyTorch modules. You can
 ``forward`` method or just calling the module ``torch_scirpt_module(in_tensor)`` The JIT compiler will compile
 and optimize the module on the fly and then returns the results.
 
+Saving TorchScript Module to Disk
+-----------------------------------
+
+For either traced or scripted modules, you can save the module to disk with the following command
+
+.. code-block:: python
+
+    import torch.jit
+
+    model = LeNet()
+    script_model = torch.jit.script(model)
+    script_model.save("lenet_scripted.ts")
+
+Using TRTorch
+*********************
+
+Now that there is some understanding of TorchScript and how to use it, we can now complete the pipeline and compile
+our TorchScript into TensorRT accelerated TorchScript. Unlike the PyTorch JIT compiler, TRTorch is an Ahead-of-Time
+(AOT) compiler. This means that unlike with PyTorch where the JIT compiler compiles from the high level PyTorch IR
+to kernel implementation at runtime, modules that are to be compiled with TRTorch are compiled fully before runtime
+(consider how you use a C compiler for an analogy). TRTorch has 3 main interfaces for using the compiler. You can
+use a CLI application similar to how you may use GCC called ``trtorchc``, or you can embed the compiler in a model
+freezing application / pipeline.
+
+.. _trtorch_quickstart:
+
+[TRTorch Quickstart] Compiling TorchScript Modules with ``trtorchc``
+---------------------------------------------------------------------
+
+An easy way to get started with TRTorch and to check if your model can be supported without extra work is to run it through
+``trtorchc``, which supports almost all features of the compiler from the command line including post training quantization
+(given a previously created calibration cache). For example we can compile our lenet model by setting our preferred operating
+precision and input size. This new TorchScript file can be loaded into Python (note: you need to ``import trtorch`` before loading
+these compiled modules because the compiler extends the PyTorch the deserializer and runtime to execute compiled modules).
+
+.. code-block:: shell
+
+    ❯ trtorchc -p f16 lenet_scripted.ts trt_lenet_scripted.ts "(1,1,32,32)"
+
+    ❯ python3
+    Python 3.6.9 (default, Apr 18 2020, 01:56:04)
+    [GCC 8.4.0] on linux
+    Type "help", "copyright", "credits" or "license" for more information.
+    >>> import torch
+    >>> import trtorch
+    >>> ts_model = torch.jit.load(“trt_lenet_scripted.ts”)
+    >>> ts_model(torch.randn((1,1,32,32)).to(“cuda”).half())
+
+You can learn more about ``trtorchc`` usage here: :ref:`trtorchc`
+
 .. _compile_py:
 
 Compiling with TRTorch in Python
 ---------------------------------
 
-To compile your TorchScript module with TRTorch, all you need to do is provide the module and some compiler settings
+To compile your TorchScript module with TRTorch embedded into Python, all you need to do is provide the module and some compiler settings
 to TRTorch and you will be returned an optimized TorchScript module to run or add into another PyTorch module. The
 only required setting is the input size or input range which is defined as a list of either list types like ``lists``, ``tuples``
 or PyTorch ``size`` objects or dictionaries of minimum, optimial and maximum sizes. You can also specify settings such as
@@ -386,14 +439,15 @@ Here is the graph that you get back after compilation is complete:
 
 .. code-block:: none
 
-    graph(%self.1 : __torch__.___torch_mangle_10.LeNet_trt,
-        %2 : Tensor):
-        %1 : int = prim::Constant[value=94106001690080]()
-        %3 : Tensor = trt::execute_engine(%1, %2)
-        return (%3)
-    (AddEngineToGraph)
+    graph(%self_1 : __torch__.lenet, %input_0 : Tensor):
+        %1 : ...trt.Engine = prim::GetAttr[name="lenet"](%self_1)
+        %3 : Tensor[] = prim::ListConstruct(%input_0)
+        %4 : Tensor[] = trt::execute_engine(%3, %1)
+        %5 : Tensor = prim::ListUnpack(%4)
+        return (%5)
+
 
-You can see the call where the engine is executed, based on a constant which is the ID of the engine, telling JIT how to find the engine and the input tensor which will be fed to TensorRT.
+You can see the call where the engine is executed, after extracting the attribute containing the engine and constructing a list of inputs, then returns the tensors back to the user.
 
 .. _unsupported_ops:
 
@@ -404,7 +458,7 @@ TRTorch is a new library and the PyTorch operator library is quite large, so the
 shown above to make modules are fully TRTorch supported and ones that are not and stitch the modules together in the deployment application or you can register converters for missing ops.
 
     You can check support without going through the full compilation pipleine using the ``trtorch::CheckMethodOperatorSupport(const torch::jit::Module& module, std::string method_name)`` api
-    to see what operators are not supported.
+    to see what operators are not supported. ``trtorchc`` automatically checks modules with this method before starting compilation and will print out a list of operators that are not supported.
 
 .. _custom_converters:
 

docsrc/tutorials/installation.rst

@@ -53,21 +53,20 @@ Dependencies for Compilation
 
 TRTorch is built with Bazel, so begin by installing it.
 
-The easiest way is to install bazelisk using the method of you choosing https://github.com/bazelbuild/bazelisk
+    * The easiest way is to install bazelisk using the method of you choosing https://github.com/bazelbuild/bazelisk
+    * Otherwise you can use the following instructions to install binaries https://docs.bazel.build/versions/master/install.html
+    * Finally if you need to compile from source (e.g. aarch64 until bazel distributes binaries for the architecture) you can use these instructions
 
-Otherwise you can use the following instructions to install binaries https://docs.bazel.build/versions/master/install.html
+    .. code-block:: shell
 
-Finally if you need to compile from source (e.g. aarch64 until bazel distributes binaries for the architecture) you can use these instructions
+        export BAZEL_VERSION=$(cat <PATH_TO_TRTORCH_ROOT>/.bazelversion)
+        mkdir bazel
+        cd bazel
+        curl -fSsL -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-dist.zip
+        unzip bazel-$BAZEL_VERSION-dist.zip
+        bash ./compile.sh
+        cp output/bazel /usr/local/bin/
 
-```sh
-export BAZEL_VERSION=3.3.1
-mkdir bazel
-cd bazel
-curl -fSsL -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-dist.zip
-unzip bazel-$BAZEL_VERSION-dist.zip
-bash ./compile.sh
-cp output/bazel /usr/local/bin/
-```
 
 You will also need to have CUDA installed on the system (or if running in a container, the system must have
 the CUDA driver installed and the container must have CUDA)
@@ -231,12 +230,25 @@ Debug Build
 
 This also compiles a debug build of ``libtrtorch.so``
 
-Building natively on aarch64 platform
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+**Building Natively on aarch64 (Jetson)**
+-------------------------------------------
+
+Prerequisites
+^^^^^^^^^^^^^^
+
+Install or compile a build of PyTorch/LibTorch for aarch64
+
+NVIDIA hosts builds the latest release branch for Jetson here:
 
-To build natively on aarch64-linux-gnu platform, configure the WORKSPACE with local available dependencies.
+    https://forums.developer.nvidia.com/t/pytorch-for-jetson-nano-version-1-5-0-now-available/72048
 
-1. Disable the rules with http_archive for x86_64 platform by commenting rules as:
+
+Enviorment Setup
+^^^^^^^^^^^^^^^^^
+
+To build natively on aarch64-linux-gnu platform, configure the ``WORKSPACE`` with local available dependencies.
+
+1. Disable the rules with ``http_archive`` for x86_64 by commenting the following rules:
 
 .. code-block:: shell
 
@@ -277,7 +289,7 @@ To build natively on aarch64-linux-gnu platform, configure the WORKSPACE with lo
     #)
 
 
-2. Disable the pip3_import rules as:
+2. Disable Python API testing dependencies:
 
 .. code-block:: shell
 
@@ -298,19 +310,27 @@ To build natively on aarch64-linux-gnu platform, configure the WORKSPACE with lo
     #pip_install()
 
 
-3. Configuring the local available dependencies by using new_local_repository rules as:
+3. Configure the correct paths to directory roots containing local dependencies in the ``new_local_repository`` rules:
+
+    NOTE: If you installed PyTorch using a pip package, the correct path is the path to the root of the python torch package.
+    In the case that you installed with ``sudo pip install`` this will be ``/usr/local/lib/python3.6/dist-packages/torch``.
+    In the case you installed with ``pip install --user`` this will be ``$HOME/.local/lib/python3.6/site-packages/torch``.
+
+In the case you are using NVIDIA compiled pip packages, set the path for both libtorch sources to the same path. This is because unlike
+PyTorch on x86_64, NVIDIA aarch64 PyTorch uses the CXX11-ABI. If you compiled for source using the pre_cxx11_abi and only would like to
+use that library, set the paths to the same path but when you compile make sure to add the flag ``--config=pre_cxx11_abi``
 
 .. code-block:: shell
 
     new_local_repository(
         name = "libtorch",
-        path = "/usr/local/lib/python3.6/site-packages/torch",
+        path = "/usr/local/lib/python3.6/dist-packages/torch",
         build_file = "third_party/libtorch/BUILD"
     )
 
     new_local_repository(
         name = "libtorch_pre_cxx11_abi",
-        path = "/usr/local/lib/python3.6/site-packages/torch",
+        path = "/usr/local/lib/python3.6/dist-packages/torch",
         build_file = "third_party/libtorch/BUILD"
     )
 
@@ -326,12 +346,22 @@ To build natively on aarch64-linux-gnu platform, configure the WORKSPACE with lo
         build_file = "@//third_party/tensorrt/local:BUILD"
     )
 
+Compile C++ Library and Compiler CLI
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Compile TRTorch library using bazel command:
+
+.. code-block:: shell
 
-Note: If pip library for torch is installed using --user, configure the new_local_repository path for torch accordingly.
+   bazel build //:libtrtorch
 
+Compile Python API
+^^^^^^^^^^^^^^^^^^^^
 
-Compile TRTorch library using bazel command as:
+Compile the Python API using the following command from the ``//py`` directory:
 
 .. code-block:: shell
 
-   bazel build //:libtrtorch
+    python3 setup.py install --use-cxx11-abi
+
+If you have a build of PyTorch that uses Pre-CXX11 ABI drop the ``--use-cxx11-abi`` flag

docsrc/tutorials/trtorchc.rst

@@ -14,7 +14,7 @@ All that is required to run the program after compilation is for C++ linking aga
 or in Python importing the trtorch package. All other aspects of using compiled modules are identical
 to standard TorchScript. Load with ``torch.jit.load()`` and run like you would run any other module.
 
-.. code-block:: txt
+.. code-block::txt
 
     trtorchc [input_file_path] [output_file_path]
         [input_shapes...] {OPTIONS}
@@ -86,6 +86,6 @@ to standard TorchScript. Load with ``torch.jit.load()`` and run like you would r
 
 e.g.
 
-.. code-block:: txt
+.. code-block:: shell
 
     trtorchc tests/modules/ssd_traced.jit.pt ssd_trt.ts "[(1,3,300,300); (1,3,512,512); (1, 3, 1024, 1024)]" -p f16