Skip to content

Commit dc572cb

Browse files
narendasannarendasan
committed
Merge branch 'main' of https://gitlab.com/nvidia/trtorch/Torch-TensorRT-Preview into release/1.0
2 parents 2f91fb6 + cfbfbb4 commit dc572cb

File tree

8 files changed

+782
-748
lines changed

8 files changed

+782
-748
lines changed

‎docker/README.md

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,16 @@
1+
# Building a Torch-TensorRT container
12

2-
## Use the container for pytorch1.10+cuda11.1+trt8.0.3.4
3+
### Install Docker and NVIDIA Container Toolkit
4+
5+
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html
6+
7+
### Build Container
8+
9+
> From root of Torch-TensorRT repo
310
411
```
512
# Build:
6-
docker build -f docker/Dockerfile -t torch_tensorrt1.0:latest .
13+
docker build --build-arg BASE={PyTorch Base Container Version} -f docker/Dockerfile -t torch_tensorrt1.0:latest .
714
815
# Run:
916
docker run --gpus all -it \

‎notebooks/README.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,18 +14,20 @@ git clone https://github.com/NVIDIA/Torch-TensorRT
1414
Next, build the NVIDIA Torch-TensorRT container (from repo root):
1515

1616
```
17+
cd Torch-TensorRT
1718
docker build -t torch_tensorrt -f ./docker/Dockerfile .
1819
```
1920

2021
Then launch the container with:
2122

2223
```
23-
docker run --runtime=nvidia -it --rm --ipc=host --net=host torch_tensorrt
24+
docker run --gpus=all --rm -it -v $PWD:/Torch-TensorRT --net=host torch_tensorrt bash
2425
```
2526

2627
Within the docker interactive bash session, start Jupyter with
2728

2829
```
30+
cd /Torch-TensorRT/notebooks
2931
jupyter notebook --allow-root --ip 0.0.0.0 --port 8888
3032
```
3133

@@ -38,8 +40,7 @@ in, for example:
3840
```http://[host machine]:8888/?token=aae96ae9387cd28151868fee318c3b3581a2d794f3b25c6b```
3941

4042

41-
Within the container, the notebooks themselves are located at `/workspace/torch_tensorrt/notebooks`. To reach them in Jupyter, click on the folder marked
42-
`torch_tensorrt`, then the folder marked `notebooks`.
43+
Within the container, the notebooks themselves are located at `/Torch-TensorRT/notebooks`.
4344

4445
## 2. Notebook list
4546

‎notebooks/Resnet50-example.ipynb

Lines changed: 432 additions & 84 deletions
Large diffs are not rendered by default.

‎notebooks/lenet-getting-started.ipynb

Lines changed: 109 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -215,43 +215,27 @@
215215
{
216216
"cell_type": "code",
217217
"execution_count": 5,
218-
"metadata": {
219-
"scrolled": false
220-
},
218+
"metadata": {},
221219
"outputs": [
222220
{
223221
"name": "stdout",
224222
"output_type": "stream",
225223
"text": [
226-
"Warm up ...\n"
227-
]
228-
},
229-
{
230-
"name": "stderr",
231-
"output_type": "stream",
232-
"text": [
233-
"/opt/conda/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at ../c10/core/TensorImpl.h:1153.)\n",
234-
" return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)\n"
235-
]
236-
},
237-
{
238-
"name": "stdout",
239-
"output_type": "stream",
240-
"text": [
224+
"Warm up ...\n",
241225
"Start timing ...\n",
242-
"Iteration 100/1000, ave batch time 5.14 ms\n",
243-
"Iteration 200/1000, ave batch time 5.13 ms\n",
244-
"Iteration 300/1000, ave batch time 5.13 ms\n",
245-
"Iteration 400/1000, ave batch time 5.13 ms\n",
246-
"Iteration 500/1000, ave batch time 5.13 ms\n",
247-
"Iteration 600/1000, ave batch time 5.13 ms\n",
248-
"Iteration 700/1000, ave batch time 5.13 ms\n",
249-
"Iteration 800/1000, ave batch time 5.13 ms\n",
250-
"Iteration 900/1000, ave batch time 5.13 ms\n",
251-
"Iteration 1000/1000, ave batch time 5.13 ms\n",
226+
"Iteration 100/1000, ave batch time 5.22 ms\n",
227+
"Iteration 200/1000, ave batch time 5.22 ms\n",
228+
"Iteration 300/1000, ave batch time 5.22 ms\n",
229+
"Iteration 400/1000, ave batch time 5.22 ms\n",
230+
"Iteration 500/1000, ave batch time 5.22 ms\n",
231+
"Iteration 600/1000, ave batch time 5.21 ms\n",
232+
"Iteration 700/1000, ave batch time 5.22 ms\n",
233+
"Iteration 800/1000, ave batch time 5.21 ms\n",
234+
"Iteration 900/1000, ave batch time 5.21 ms\n",
235+
"Iteration 1000/1000, ave batch time 5.21 ms\n",
252236
"Input shape: torch.Size([1024, 1, 32, 32])\n",
253237
"Output features size: torch.Size([1024, 10])\n",
254-
"Average batch time: 5.13 ms\n"
238+
"Average batch time: 5.21 ms\n"
255239
]
256240
}
257241
],
@@ -315,19 +299,19 @@
315299
"text": [
316300
"Warm up ...\n",
317301
"Start timing ...\n",
318-
"Iteration 100/1000, ave batch time 5.13 ms\n",
319-
"Iteration 200/1000, ave batch time 5.13 ms\n",
320-
"Iteration 300/1000, ave batch time 5.13 ms\n",
321-
"Iteration 400/1000, ave batch time 5.13 ms\n",
322-
"Iteration 500/1000, ave batch time 5.13 ms\n",
323-
"Iteration 600/1000, ave batch time 5.13 ms\n",
324-
"Iteration 700/1000, ave batch time 5.13 ms\n",
325-
"Iteration 800/1000, ave batch time 5.13 ms\n",
326-
"Iteration 900/1000, ave batch time 5.13 ms\n",
327-
"Iteration 1000/1000, ave batch time 5.13 ms\n",
302+
"Iteration 100/1000, ave batch time 5.22 ms\n",
303+
"Iteration 200/1000, ave batch time 5.21 ms\n",
304+
"Iteration 300/1000, ave batch time 5.21 ms\n",
305+
"Iteration 400/1000, ave batch time 5.21 ms\n",
306+
"Iteration 500/1000, ave batch time 5.22 ms\n",
307+
"Iteration 600/1000, ave batch time 5.22 ms\n",
308+
"Iteration 700/1000, ave batch time 5.22 ms\n",
309+
"Iteration 800/1000, ave batch time 5.22 ms\n",
310+
"Iteration 900/1000, ave batch time 5.22 ms\n",
311+
"Iteration 1000/1000, ave batch time 5.22 ms\n",
328312
"Input shape: torch.Size([1024, 1, 32, 32])\n",
329313
"Output features size: torch.Size([1024, 10])\n",
330-
"Average batch time: 5.13 ms\n"
314+
"Average batch time: 5.22 ms\n"
331315
]
332316
}
333317
],
@@ -398,19 +382,19 @@
398382
"text": [
399383
"Warm up ...\n",
400384
"Start timing ...\n",
401-
"Iteration 100/1000, ave batch time 5.16 ms\n",
402-
"Iteration 200/1000, ave batch time 5.15 ms\n",
403-
"Iteration 300/1000, ave batch time 5.14 ms\n",
404-
"Iteration 400/1000, ave batch time 5.14 ms\n",
405-
"Iteration 500/1000, ave batch time 5.14 ms\n",
406-
"Iteration 600/1000, ave batch time 5.14 ms\n",
407-
"Iteration 700/1000, ave batch time 5.14 ms\n",
408-
"Iteration 800/1000, ave batch time 5.14 ms\n",
409-
"Iteration 900/1000, ave batch time 5.14 ms\n",
410-
"Iteration 1000/1000, ave batch time 5.14 ms\n",
385+
"Iteration 100/1000, ave batch time 5.21 ms\n",
386+
"Iteration 200/1000, ave batch time 5.21 ms\n",
387+
"Iteration 300/1000, ave batch time 5.27 ms\n",
388+
"Iteration 400/1000, ave batch time 5.28 ms\n",
389+
"Iteration 500/1000, ave batch time 5.27 ms\n",
390+
"Iteration 600/1000, ave batch time 5.26 ms\n",
391+
"Iteration 700/1000, ave batch time 5.26 ms\n",
392+
"Iteration 800/1000, ave batch time 5.25 ms\n",
393+
"Iteration 900/1000, ave batch time 5.25 ms\n",
394+
"Iteration 1000/1000, ave batch time 5.25 ms\n",
411395
"Input shape: torch.Size([1024, 1, 32, 32])\n",
412396
"Output features size: torch.Size([1024, 10])\n",
413-
"Average batch time: 5.14 ms\n"
397+
"Average batch time: 5.25 ms\n"
414398
]
415399
}
416400
],
@@ -437,9 +421,29 @@
437421
},
438422
{
439423
"cell_type": "code",
440-
"execution_count": 11,
424+
"execution_count": 17,
441425
"metadata": {},
442-
"outputs": [],
426+
"outputs": [
427+
{
428+
"name": "stderr",
429+
"output_type": "stream",
430+
"text": [
431+
"WARNING: [Torch-TensorRT] - For input x.1, found user specified input dtype as Float16, however when inspecting the graph, the input type expected was inferred to be Float\n",
432+
"The compiler is going to use the user setting Float16\n",
433+
"This conflict may cause an error at runtime due to partial compilation being enabled and therefore\n",
434+
"compatibility with PyTorch's data type convention is required.\n",
435+
"If you do indeed see errors at runtime either:\n",
436+
"- Remove the dtype spec for x.1\n",
437+
"- Disable partial compilation by setting require_full_compilation to True\n",
438+
"WARNING: [Torch-TensorRT TorchScript Conversion Context] - The logger passed into createInferBuilder differs from one already provided for an existing builder, runtime, or refitter. TensorRT maintains only a single logger pointer at any given time, so the existing value, which can be retrieved with getLogger(), will be used instead. In order to use a new logger, first destroy all existing builder, runner or refitter objects.\n",
439+
"\n",
440+
"WARNING: [Torch-TensorRT] - Dilation not used in Max pooling converter\n",
441+
"WARNING: [Torch-TensorRT] - Dilation not used in Max pooling converter\n",
442+
"WARNING: [Torch-TensorRT] - Detected invalid timing cache, setup a local cache instead\n",
443+
"WARNING: [Torch-TensorRT] - Max value of this profile is not valid\n"
444+
]
445+
}
446+
],
443447
"source": [
444448
"import torch_tensorrt\n",
445449
"\n",
@@ -451,10 +455,10 @@
451455
" max_shape=[1024, 1, 34, 34],\n",
452456
" dtype=torch.half\n",
453457
" )],\n",
454-
" \"enabled_precisions\": {torch.float, torch.half} # Run with FP16\n",
458+
" \"enabled_precisions\": {torch.half} # Run with FP16\n",
455459
"}\n",
456460
"\n",
457-
"trt_ts_module = torch_tensorrt.compile(traced_model, compile_settings)\n",
461+
"trt_ts_module = torch_tensorrt.compile(traced_model, **compile_settings)\n",
458462
"\n",
459463
"input_data = torch.randn((1024, 1, 32, 32))\n",
460464
"input_data = input_data.half().to(\"cuda\")\n",
@@ -466,7 +470,7 @@
466470
},
467471
{
468472
"cell_type": "code",
469-
"execution_count": 12,
473+
"execution_count": 20,
470474
"metadata": {},
471475
"outputs": [
472476
{
@@ -475,19 +479,19 @@
475479
"text": [
476480
"Warm up ...\n",
477481
"Start timing ...\n",
478-
"Iteration 100/1000, ave batch time 2.00 ms\n",
479-
"Iteration 200/1000, ave batch time 1.97 ms\n",
480-
"Iteration 300/1000, ave batch time 1.98 ms\n",
481-
"Iteration 400/1000, ave batch time 1.98 ms\n",
482-
"Iteration 500/1000, ave batch time 1.99 ms\n",
483-
"Iteration 600/1000, ave batch time 1.99 ms\n",
484-
"Iteration 700/1000, ave batch time 1.99 ms\n",
485-
"Iteration 800/1000, ave batch time 1.99 ms\n",
486-
"Iteration 900/1000, ave batch time 1.98 ms\n",
487-
"Iteration 1000/1000, ave batch time 1.98 ms\n",
482+
"Iteration 100/1000, ave batch time 2.47 ms\n",
483+
"Iteration 200/1000, ave batch time 2.40 ms\n",
484+
"Iteration 300/1000, ave batch time 2.35 ms\n",
485+
"Iteration 400/1000, ave batch time 2.35 ms\n",
486+
"Iteration 500/1000, ave batch time 2.35 ms\n",
487+
"Iteration 600/1000, ave batch time 2.35 ms\n",
488+
"Iteration 700/1000, ave batch time 2.36 ms\n",
489+
"Iteration 800/1000, ave batch time 2.35 ms\n",
490+
"Iteration 900/1000, ave batch time 2.91 ms\n",
491+
"Iteration 1000/1000, ave batch time 2.85 ms\n",
488492
"Input shape: torch.Size([1024, 1, 32, 32])\n",
489493
"Output features size: torch.Size([1024, 10])\n",
490-
"Average batch time: 1.98 ms\n"
494+
"Average batch time: 2.85 ms\n"
491495
]
492496
}
493497
],
@@ -506,9 +510,29 @@
506510
},
507511
{
508512
"cell_type": "code",
509-
"execution_count": 13,
513+
"execution_count": 21,
510514
"metadata": {},
511-
"outputs": [],
515+
"outputs": [
516+
{
517+
"name": "stderr",
518+
"output_type": "stream",
519+
"text": [
520+
"WARNING: [Torch-TensorRT] - For input x.1, found user specified input dtype as Float16, however when inspecting the graph, the input type expected was inferred to be Float\n",
521+
"The compiler is going to use the user setting Float16\n",
522+
"This conflict may cause an error at runtime due to partial compilation being enabled and therefore\n",
523+
"compatibility with PyTorch's data type convention is required.\n",
524+
"If you do indeed see errors at runtime either:\n",
525+
"- Remove the dtype spec for x.1\n",
526+
"- Disable partial compilation by setting require_full_compilation to True\n",
527+
"WARNING: [Torch-TensorRT TorchScript Conversion Context] - The logger passed into createInferBuilder differs from one already provided for an existing builder, runtime, or refitter. TensorRT maintains only a single logger pointer at any given time, so the existing value, which can be retrieved with getLogger(), will be used instead. In order to use a new logger, first destroy all existing builder, runner or refitter objects.\n",
528+
"\n",
529+
"WARNING: [Torch-TensorRT] - Dilation not used in Max pooling converter\n",
530+
"WARNING: [Torch-TensorRT] - Dilation not used in Max pooling converter\n",
531+
"WARNING: [Torch-TensorRT] - Detected invalid timing cache, setup a local cache instead\n",
532+
"WARNING: [Torch-TensorRT] - Max value of this profile is not valid\n"
533+
]
534+
}
535+
],
512536
"source": [
513537
"import torch_tensorrt\n",
514538
"\n",
@@ -520,10 +544,10 @@
520544
" max_shape=[1024, 1, 34, 34],\n",
521545
" dtype=torch.half\n",
522546
" )],\n",
523-
" \"enabled_precisions\": {torch.float, torch.half} # Run with FP16\n",
547+
" \"enabled_precisions\": {torch.half} # Run with FP16\n",
524548
"}\n",
525549
"\n",
526-
"trt_script_module = torch_tensorrt.compile(script_model, compile_settings)\n",
550+
"trt_script_module = torch_tensorrt.compile(script_model, **compile_settings)\n",
527551
"\n",
528552
"input_data = torch.randn((1024, 1, 32, 32))\n",
529553
"input_data = input_data.half().to(\"cuda\")\n",
@@ -535,7 +559,7 @@
535559
},
536560
{
537561
"cell_type": "code",
538-
"execution_count": 14,
562+
"execution_count": 22,
539563
"metadata": {},
540564
"outputs": [
541565
{
@@ -544,19 +568,19 @@
544568
"text": [
545569
"Warm up ...\n",
546570
"Start timing ...\n",
547-
"Iteration 100/1000, ave batch time 2.02 ms\n",
548-
"Iteration 200/1000, ave batch time 1.98 ms\n",
549-
"Iteration 300/1000, ave batch time 1.97 ms\n",
550-
"Iteration 400/1000, ave batch time 1.96 ms\n",
551-
"Iteration 500/1000, ave batch time 1.96 ms\n",
552-
"Iteration 600/1000, ave batch time 1.96 ms\n",
553-
"Iteration 700/1000, ave batch time 1.96 ms\n",
554-
"Iteration 800/1000, ave batch time 1.96 ms\n",
555-
"Iteration 900/1000, ave batch time 1.96 ms\n",
556-
"Iteration 1000/1000, ave batch time 1.96 ms\n",
571+
"Iteration 100/1000, ave batch time 2.34 ms\n",
572+
"Iteration 200/1000, ave batch time 2.34 ms\n",
573+
"Iteration 300/1000, ave batch time 2.35 ms\n",
574+
"Iteration 400/1000, ave batch time 2.36 ms\n",
575+
"Iteration 500/1000, ave batch time 2.58 ms\n",
576+
"Iteration 600/1000, ave batch time 3.26 ms\n",
577+
"Iteration 700/1000, ave batch time 3.13 ms\n",
578+
"Iteration 800/1000, ave batch time 3.02 ms\n",
579+
"Iteration 900/1000, ave batch time 2.96 ms\n",
580+
"Iteration 1000/1000, ave batch time 2.90 ms\n",
557581
"Input shape: torch.Size([1024, 1, 32, 32])\n",
558582
"Output features size: torch.Size([1024, 10])\n",
559-
"Average batch time: 1.96 ms\n"
583+
"Average batch time: 2.90 ms\n"
560584
]
561585
}
562586
],
@@ -579,7 +603,7 @@
579603
],
580604
"metadata": {
581605
"kernelspec": {
582-
"display_name": "Python 3",
606+
"display_name": "Python 3 (ipykernel)",
583607
"language": "python",
584608
"name": "python3"
585609
},
@@ -593,7 +617,7 @@
593617
"name": "python",
594618
"nbconvert_exporter": "python",
595619
"pygments_lexer": "ipython3",
596-
"version": "3.6.13"
620+
"version": "3.8.10"
597621
}
598622
},
599623
"nbformat": 4,

0 commit comments

Comments
 (0)