How to use a traced torchscript model from Python?

[fracpete]

I built the docker images from here:
https://github.com/facebookresearch/detectron2/tree/master/tools/deploy

And traced a torchscript model (export-method=caffe2_tracing, format=torchscript), using the following command as basis:

./export_model.py --config-file ../../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml \
    --output ./output --export-method caffe2_tracing --format torchscript \
    MODEL.WEIGHTS detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl \
    MODEL.DEVICE cpu

Calling the torchscript_traced_mask_rcnn binary with appropriate parameters works on GPU and CPU:

./build/torchscript_traced_mask_rcnn output/model.ts input.jpg caffe2_tracing

The binary outputs something like this:

Latency (should vary with different inputs): 3.21286 seconds
Number of output tensors: 4
bbox: CPUFloatType [4, 4]
scores: CPUFloatType [4]
pred_classes: CPUFloatType [4]
pred_masks: CPUFloatType [4, 80, 28, 28]
 138.8425  145.1582  152.2856  173.8196
 124.2041  146.0827  138.4888  174.0243
 137.3747  166.2999  495.9241  321.6906
 112.4978  149.2721  125.9721  177.1775
[ CPUFloatType{4,4} ]

Since I'm not a C++ developer, I'm trying to replicate the C++ code that underpins torchscript_traced_mask_rcnn in Python for the caffe2_tracing method:
https://github.com/facebookresearch/detectron2/blob/master/tools/deploy/torchscript_traced_mask_rcnn.cpp

So far, I've come up with this:

import cv2
import numpy as np
import torch

model_file = "/some/where/model.ts"
image_file = "/some/where/input.jpg"

with torch.no_grad():
    model = torch.jit.load(model_file)
    device = None
    for b in model.buffers():
        device = b.device
        break
    if device is None:
        raise Exception("No buffers?")

    input_img = cv2.imread(image_file)
    height, width = input_img.shape[:2]
    channels = 3
    assert(height % 32 == 0 and width % 32 == 0)
    inp = torch.from_numpy(input_img)
    print("inp", inp.shape)
    inp = torch.as_tensor(input_img.astype("float32")).permute(2, 0, 1).unsqueeze(0)
    print("inp", inp.shape)
    im_info = torch.from_numpy(np.array([float(height), float(width), 1.0])).unsqueeze(0)
    print("im_info", im_info.shape)
    inputs = (inp, im_info)
    output = model.forward(inputs)

However, the line model.forward(inputs) generates the following error message:

RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/detectron2/export/caffe2_modeling.py", line 17, in forward
    x = torch.div(torch.sub(data, _4, alpha=1), _3)
    _5, _6, _7, _8, _9, _10, _11, = (_2).forward(x, )
    _12 = (_1).forward(_5, _6, _7, _8, _9, im_info, )
           ~~~~~~~~~~~ <--- HERE
    _13 = (_0).forward(_12, _5, _6, _7, _10, im_info, _11, )
    _14, _15, _16, _17, = _13
  File "code/__torch__/detectron2/export/c10.py", line 24, in forward
    scores = torch.detach(_6)
    bbox_deltas = torch.detach(_7)
    _16, _17 = ops._caffe2.GenerateProposals(scores, bbox_deltas, im_info, _4, 0.25, 1000, 1000, 0.69999999999999996, 0., True, -180, 180, 1., False, None)
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    scores0 = torch.detach(_8)
    bbox_deltas0 = torch.detach(_9)

Traceback of TorchScript, original code (most recent call last):
/home/appuser/detectron2_repo/detectron2/export/c10.py(203): _generate_proposals
/home/appuser/detectron2_repo/detectron2/export/c10.py(256): forward
/home/appuser/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward
/home/appuser/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl
/home/appuser/detectron2_repo/detectron2/export/caffe2_modeling.py(272): forward
/usr/lib/python3.6/contextlib.py(52): inner
/home/appuser/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(860): _slow_forward
/home/appuser/.local/lib/python3.6/site-packages/torch/nn/modules/module.py(887): _call_impl
/home/appuser/.local/lib/python3.6/site-packages/torch/jit/_trace.py(940): trace_module
/home/appuser/.local/lib/python3.6/site-packages/torch/jit/_trace.py(742): trace
/home/appuser/detectron2_repo/detectron2/export/api.py(134): export_torchscript
./export_model.py(53): export_caffe2_tracing
./export_model.py(172): <module>
RuntimeError: [enforce fail at generate_proposals_op.cc:291] im_info_tensor.template IsType<float>(). double

This is where I'm currently stuck, as I have no idea how to interpret this error...

My Python virtual environment that I used:

absl-py==0.12.0
cachetools==4.2.1
certifi==2020.12.5
chardet==4.0.0
cloudpickle==1.6.0
cycler==0.10.0
Cython==0.29.22
detectron2==0.4+cpu
future==0.18.2
fvcore==0.1.3.post20210317
google-auth==1.28.0
google-auth-oauthlib==0.4.3
grpcio==1.36.1
idna==2.10
importlib-metadata==3.7.3
iopath==0.1.6
kiwisolver==1.3.1
Markdown==3.3.4
matplotlib==3.3.4
numpy==1.20.1
oauthlib==3.1.0
omegaconf==2.0.6
opencv-python==4.5.1.48
Pillow==8.1.2
pkg-resources==0.0.0
portalocker==2.2.1
protobuf==3.15.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.2
pydot==1.4.2
pyparsing==2.4.7
python-dateutil==2.8.1
PyYAML==5.4.1
requests==2.25.1
requests-oauthlib==1.3.0
rsa==4.7.2
six==1.15.0
tabulate==0.8.9
tensorboard==2.4.1
tensorboard-plugin-wit==1.8.0
termcolor==1.1.0
torch==1.8.0
torchvision==0.9.0
tqdm==4.59.0
typing-extensions==3.7.4.3
urllib3==1.26.4
Werkzeug==1.0.1
yacs==0.1.8
zipp==3.4.1

Thanks for any insight!

[ppwwyyxx]

RuntimeError: [enforce fail at generate_proposals_op.cc:291] im_info_tensor.template IsType(). double

The error says im_info has to be float but double is given.

[fracpete]

Awesome, that solved it! :-)

So instead of using:

im_info = torch.from_numpy(np.array([float(height), float(width), 1.0])).unsqueeze(0)

Converting the tensor for float32 works:

im_info = torch.from_numpy(np.array([float(height), float(width), 1.0])).unsqueeze(0).to(torch.float32)