How to remove in-place copy_ operator from the traced model?

[apivovarov]

I traced the model from the tutorial (COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x)
I noticed that the resulting graph has in-place copy_ operators.

Is it possible to somehow remove/replace this copy_ operator from the graph?
Maybe jit has an option to produce a graph without in-place operators (such as copy_)?

[ppwwyyxx]

I think this would require rewriting Box2BoxTransform.apply_deltas in an equivalent form to not use in-place operations in the end (but some concat + reshape instead).

[apivovarov]

Thank you for the idea!

BTW, what you think about using torch._C._jit_pass_onnx_prepare_inplace_ops_for_onnx and torch._C._jit_pass_dce_allow_deleting_nodes_with_side_effects?
Can it automatically convert the graph to equivalent graph but without in-place ops?

[ppwwyyxx]

Yeah the set of passes in torch/onnx/utils.py might also work, but we have not used them.
Also the graph automatically produced for this case probably isn't as optimized as a manual rewrite by human.

[apivovarov]

Yuxin, I opened a PR to replace copy_ op in Box2BoxTransform.apply_delta #2685

[apivovarov]

Another place which probably should be rewritten is boxes.clip() - https://github.com/facebookresearch/detectron2/blob/master/detectron2/structures/boxes.py#L205

[apivovarov]

The fix for boxes.clip() - #2716

[ppwwyyxx]

Hi, I'm wondering what's the status of tvm support of our model? Does the model now successfully run under tvm?

[apivovarov]

The model consist of 3 main sub-Modules:

• backbone
• proposal_generator
• roi_heads

I was able to successfully compile the backbone submodule.

proposal_generator has the following issues:

PG Tracing done
from_pytorch PG
Traceback (most recent call last):
  File "model_summary.py", line 160, in <module>
    mod, params = relay.frontend.from_pytorch(pg_net_traced, pg_shape_list)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tvm-0.8.dev970+g5a2ded777-py3.8-macosx-10.15-x86_64.egg/tvm/relay/frontend/pytorch.py", line 3284, in from_pytorch
    ret = converter.convert_operators(_get_operator_nodes(graph.nodes()), outputs, ret_name)[0]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tvm-0.8.dev970+g5a2ded777-py3.8-macosx-10.15-x86_64.egg/tvm/relay/frontend/pytorch.py", line 2705, in convert_operators
    relay_out = relay_op(
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tvm-0.8.dev970+g5a2ded777-py3.8-macosx-10.15-x86_64.egg/tvm/relay/frontend/pytorch.py", line 1736, in clamp
    return _op.clip(data, amin, amax)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tvm-0.8.dev970+g5a2ded777-py3.8-macosx-10.15-x86_64.egg/tvm/relay/op/tensor.py", line 1059, in clip
    return _make.clip(a, a_min, a_max)
  File "tvm/_ffi/_cython/./packed_func.pxi", line 322, in tvm._ffi._cy3.core.PackedFuncBase.__call__
  File "tvm/_ffi/_cython/./packed_func.pxi", line 257, in tvm._ffi._cy3.core.FuncCall
  File "tvm/_ffi/_cython/./packed_func.pxi", line 246, in tvm._ffi._cy3.core.FuncCall3
  File "tvm/_ffi/_cython/./base.pxi", line 160, in tvm._ffi._cy3.core.CALL
tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (5) 6   libtvm.dylib                        0x00000001395f3826 TVMFuncCall + 70
  [bt] (4) 5   libtvm.dylib                        0x0000000139161e7e void tvm::runtime::TypedPackedFunc<tvm::RelayExpr (tvm::RelayExpr, double, double)>::AssignTypedLambda<tvm::RelayExpr (*)(tvm::RelayExpr, double, double)>(tvm::RelayExpr (*)(tvm::RelayExpr, double, double), std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >)::'lambda'(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)::operator()(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*) const + 830
  [bt] (3) 4   libtvm.dylib                        0x0000000138626057 tvm::runtime::TVMMovableArgValueWithContext_::operator double<double>() const + 23
  [bt] (2) 3   libtvm.dylib                        0x0000000138460567 tvm::runtime::TVMPODValue_::operator double() const + 471
  [bt] (1) 2   libtvm.dylib                        0x000000013845c619 tvm::runtime::detail::LogFatal::Entry::Finalize() + 89
  [bt] (0) 1   libtvm.dylib                        0x00000001396063f8 tvm::runtime::Backtrace() + 24
  File "/Users/pivovaa/workspace/tvm/include/tvm/runtime/packed_func.h", line 713
TVMError: In function relay.op._make.clip: error while converting argument 2: [14:27:31] /Users/pivovaa/workspace/tvm/include/tvm/runtime/packed_func.h:493: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------

  Check failed: type_code_ == kDLFloat (8 vs. 2) : expected float but got Object

roi_heads has the following issues:

RH Tracing done
RH Shapes: [('data0', ([1, 3, 800, 1216], 'float32')), ('data1', ([2], 'int64')), ('data2', ([1, 256, 200, 304], 'float32')), ('data3', ([1, 256, 100, 152], 'float32')), ('data4', ([1, 256, 50, 76], 'float32')), ('data5', ([1, 256, 25, 38], 'float32')), ('data6', ([1, 256, 13, 19], 'float32')), ('data7', ([1000, 4], 'float32')), ('data8', ([1000], 'float32'))]
from_pytorch RH
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tvm-0.8.dev970+g5a2ded777-py3.8-macosx-10.15-x86_64.egg/tvm/runtime/object.py", line 63, in __getattr__
    return _ffi_node_api.NodeGetAttr(self, name)
  File "tvm/_ffi/_cython/./packed_func.pxi", line 322, in tvm._ffi._cy3.core.PackedFuncBase.__call__
  File "tvm/_ffi/_cython/./packed_func.pxi", line 257, in tvm._ffi._cy3.core.FuncCall
  File "tvm/_ffi/_cython/./packed_func.pxi", line 246, in tvm._ffi._cy3.core.FuncCall3
  File "tvm/_ffi/_cython/./base.pxi", line 160, in tvm._ffi._cy3.core.CALL
AttributeError: Traceback (most recent call last):
  [bt] (5) 6   libtvm.dylib                        0x00000001335fe826 TVMFuncCall + 70
  [bt] (4) 5   libtvm.dylib                        0x00000001327665cb std::__1::__function::__func<void (*)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), std::__1::allocator<void (*)(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>, void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&) + 59
  [bt] (3) 4   libtvm.dylib                        0x00000001327637d6 tvm::NodeGetAttr(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) + 502
  [bt] (2) 3   libtvm.dylib                        0x0000000132761882 tvm::ReflectionVTable::GetAttr(tvm::runtime::Object*, tvm::runtime::String const&) const + 770
  [bt] (1) 2   libtvm.dylib                        0x0000000132467619 tvm::runtime::detail::LogFatal::Entry::Finalize() + 89
  [bt] (0) 1   libtvm.dylib                        0x00000001336113f8 tvm::runtime::Backtrace() + 24
  File "/Users/pivovaa/workspace/tvm/src/node/reflection.cc", line 110
AttributeError: relay.Call object has no attributed type_annotation

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "model_summary.py", line 219, in <module>
    mod, params = relay.frontend.from_pytorch(rh_net_traced, rh_shape_list)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tvm-0.8.dev970+g5a2ded777-py3.8-macosx-10.15-x86_64.egg/tvm/relay/frontend/pytorch.py", line 3284, in from_pytorch
    ret = converter.convert_operators(_get_operator_nodes(graph.nodes()), outputs, ret_name)[0]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tvm-0.8.dev970+g5a2ded777-py3.8-macosx-10.15-x86_64.egg/tvm/relay/frontend/pytorch.py", line 2690, in convert_operators
    unpacked = _unpack_tuple(inputs[0])
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tvm-0.8.dev970+g5a2ded777-py3.8-macosx-10.15-x86_64.egg/tvm/relay/frontend/pytorch.py", line 3108, in _unpack_tuple
    elif isinstance(tup.type_annotation, TupleType):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/tvm-0.8.dev970+g5a2ded777-py3.8-macosx-10.15-x86_64.egg/tvm/runtime/object.py", line 65, in __getattr__
    raise AttributeError("%s has no attribute %s" % (str(type(self)), name))
AttributeError: <class 'tvm.relay.expr.Call'> has no attribute type_annotation

The script to split Detectron2 model into submodules and compile them with TVM is here: https://gist.github.com/apivovarov/211b85fb5bfb136709ebd62313a83ba2