Skip to content

Commit 4d606bc

Browse files
committed
Merge branch 'readme' into 'main'
chore: Update Docker instructions in main README.md See merge request nvidia/trtorch/Torch-TensorRT-Preview!2
2 parents 6619371 + c4723a9 commit 4d606bc

File tree

1 file changed

+47
-18
lines changed

1 file changed

+47
-18
lines changed

README.md

Lines changed: 47 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -10,9 +10,24 @@ More Information / System Architecture:
1010

1111
- [GTC 2020 Talk](https://developer.nvidia.com/gtc/2020/video/s21671)
1212

13+
14+
15+
## Building a docker container for Torch-TensorRT Preview
16+
17+
We provide a `Dockerfile` in `docker/` directory. We build `Torch-TensorRT` on top of a `Pytorch NGC container` which provide basic dependencies (like CUDA, CUDNN, CUBLAS, TensorRT, Pytorch and others) The dependency libraries in the container can be found in the <a href="https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/index.html">release notes</a>.
18+
19+
Please follow this instruction to build a Docker container.
20+
21+
```bash
22+
docker build -f docker/Dockerfile -t torch_tensorrt_preview:latest .
23+
```
24+
25+
If you would like to build outside a docker container, please follow the section [Compiling Torch-TensorRT](#compiling-torch-tensorrt)
26+
1327
## Example Usage
1428

1529
### C++
30+
1631
```c++
1732
#include "torch/script.h"
1833
#include "torch_tensorrt/torch_tensorrt.h"
@@ -21,12 +36,12 @@ More Information / System Architecture:
2136
// Set input datatypes. Allowerd options torch::{kFloat, kHalf, kChar, kInt32, kBool}
2237
// Size of input_dtypes should match number of inputs to the network.
2338
// If input_dtypes is not set, default precision follows traditional PyT / TRT rules
24-
auto input = torch_tensorrt::CompileSpec::Input(dims, torch::kHalf)
25-
auto compile_settings = torch_tensorrt::CompileSpec({input});
39+
auto input = torch_tensorrt::Input(dims, torch::kHalf)
40+
auto compile_settings = torch_tensorrt::ts::CompileSpec({input});
2641
// FP16 execution
2742
compile_settings.enabled_precisions = {torch::kHalf};
2843
// Compile module
29-
auto trt_mod = torch_tensorrt::CompileGraph(ts_mod, compile_settings);
44+
auto trt_mod = torch_tensorrt::ts::CompileModule(ts_mod, compile_settings);
3045
// Run like normal
3146
auto results = trt_mod.forward({in_tensor});
3247
// Save module for later
@@ -35,6 +50,7 @@ trt_mod.save("trt_torchscript_module.ts");
3550
```
3651
3752
### Python
53+
3854
```py
3955
import torch_tensorrt
4056
@@ -58,23 +74,25 @@ torch.jit.save(trt_ts_module, "trt_torchscript_module.ts")
5874
```
5975

6076
> Notes on running in lower precisions:
77+
>
6178
> - Enabled lower precisions with compile_spec.enabled_precisions
6279
> - The module should be left in FP32 before compilation (FP16 can support half tensor models)
6380
> - In FP16 only input tensors by default should be FP16, other precisions use FP32. This can be overrided by setting Input::dtype
6481
6582
## Platform Support
6683

67-
| Platform | Support |
68-
| -------- | ------- |
69-
| Linux AMD64 / GPU | **Supported** |
84+
| Platform | Support |
85+
| ------------------- | ------------------------------------------------ |
86+
| Linux AMD64 / GPU | **Supported** |
7087
| Linux aarch64 / GPU | **Native Compilation Supported on JetPack-4.4+** |
7188
| Linux aarch64 / DLA | **Native Compilation Supported on JetPack-4.4+** |
72-
| Windows / GPU | **Unofficial Support** |
73-
| Linux ppc64le / GPU | - |
89+
| Windows / GPU | **Unofficial Support** |
90+
| Linux ppc64le / GPU | - |
7491

7592
> Note: Refer NVIDIA NGC container(https://ngc.nvidia.com/catalog/containers/nvidia:l4t-pytorch) for PyTorch libraries on JetPack.
7693
7794
### Dependencies
95+
7896
These are the following dependencies used to verify the testcases. Torch-TensorRT can work with other versions, but the tests are not guaranteed to pass.
7997

8098
- Bazel 4.0.0
@@ -118,10 +136,11 @@ then you have two options.
118136
> Make sure when running Torch-TensorRT that these versions of the libraries are prioritized in your `$LD_LIBRARY_PATH`
119137
120138
1. You need to download the tarball distributions of TensorRT and cuDNN from the NVIDIA website.
121-
- https://developer.nvidia.com/cudnn
122-
- https://developer.nvidia.com/tensorrt
139+
- https://developer.nvidia.com/cudnn
140+
- https://developer.nvidia.com/tensorrt
123141
2. Place these files in a directory (the directories `third_party/dist_dir/[x86_64-linux-gnu | aarch64-linux-gnu]` exist for this purpose)
124142
3. Compile using:
143+
125144
``` shell
126145
bazel build //:libtorchtrt --compilation_mode opt --distdir third_party/dist_dir/[x86_64-linux-gnu | aarch64-linux-gnu]
127146
```
@@ -133,6 +152,7 @@ bazel build //:libtorchtrt --compilation_mode opt --distdir third_party/dist_dir
133152
134153
1. Install TensorRT, CUDA and cuDNN on the system before starting to compile.
135154
2. In `WORKSPACE` comment out
155+
136156
```py
137157
# Downloaded distributions to use with --distdir
138158
http_archive(
@@ -153,7 +173,9 @@ http_archive(
153173
strip_prefix = "TensorRT-<VERSION>"
154174
)
155175
```
176+
156177
and uncomment
178+
157179
```py
158180
# Locally installed dependencies
159181
new_local_repository(
@@ -168,28 +190,33 @@ new_local_repository(
168190
build_file = "@//third_party/tensorrt/local:BUILD"
169191
)
170192
```
193+
171194
3. Compile using:
195+
172196
``` shell
173197
bazel build //:libtorchtrt --compilation_mode opt
174198
```
175199

176200
### Debug build
201+
177202
``` shell
178203
bazel build //:libtorchtrt --compilation_mode=dbg
179204
```
180205

181206
### Native compilation on NVIDIA Jetson AGX
207+
182208
``` shell
183209
bazel build //:libtorchtrt --distdir third_party/dist_dir/aarch64-linux-gnu
184210
```
211+
185212
> Note: Please refer [installation](docs/tutorials/installation.html) instructions for Pre-requisites
186213
187214
A tarball with the include files and library can then be found in bazel-bin
188215

189216
### Running Torch-TensorRT on a JIT Graph
190217

191218
> Make sure to add LibTorch to your LD_LIBRARY_PATH <br>
192-
>`export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/bazel-Torch-TensorRT/external/libtorch/lib`
219+
> `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/bazel-Torch-TensorRT/external/libtorch/lib`
193220
194221
``` shell
195222
bazel run //cpp/bin/torchtrtc -- $(realpath <PATH TO GRAPH>) out.ts <input-size>
@@ -200,9 +227,11 @@ bazel run //cpp/bin/torchtrtc -- $(realpath <PATH TO GRAPH>) out.ts <input-size>
200227
To compile the python package for your local machine, just run `python3 setup.py install` in the `//py` directory.
201228
To build wheel files for different python versions, first build the Dockerfile in ``//py`` then run the following
202229
command
230+
203231
```
204232
docker run -it -v$(pwd)/..:/workspace/Torch-TensorRT build_torch_tensorrt_wheel /bin/bash /workspace/Torch-TensorRT/py/build_whl.sh
205233
```
234+
206235
Python compilation expects using the tarball based compilation strategy from above.
207236

208237
## How do I add support for a new op...
@@ -219,13 +248,13 @@ You can register a converter for your op using the `NodeConverterRegistry` insid
219248

220249
## Structure of the repo
221250

222-
| Component | Description |
223-
| ------------- | ------------------------------------------------------------ |
224-
| [**core**](core) | Main JIT ingest, lowering, conversion and runtime implementations |
225-
| [**cpp**](cpp) | C++ API and CLI source |
226-
| [**examples**](examples) | Example applications to show different features of Torch-TensorRT |
227-
| [**py**](py) | Python API for Torch-TensorRT |
228-
| [**tests**](tests) | Unit tests for Torch-TensorRT |
251+
| Component | Description |
252+
| ------------------------ | ------------------------------------------------------------ |
253+
| [**core**](core) | Main JIT ingest, lowering, conversion and runtime implementations |
254+
| [**cpp**](cpp) | C++ API and CLI source |
255+
| [**examples**](examples) | Example applications to show different features of Torch-TensorRT |
256+
| [**py**](py) | Python API for Torch-TensorRT |
257+
| [**tests**](tests) | Unit tests for Torch-TensorRT |
229258

230259
## Contributing
231260

0 commit comments

Comments
 (0)