[ ERROR ] There is no registered "infer" function for node "Unpooling_53" with op = "Unpooling". Please implement this function in the extensions.

[Rakshith2597]

Hi @dkurt ,
Your solution to overcome a pytorch to IR model conversion issue here was really helpful, however we couldn't proceed with the PR due to dataset license issues. I am facing issue with model conversion in another model, this time it is due to maxunpooling layer in the decoder. I came across your really good extension. However, I am still not able to resolve the problem.

Receiving the following the error while converting model with maxunpool.

[ ERROR ]  Cannot infer shapes or values for node "Unpooling_53".
[ ERROR ]  There is no registered "infer" function for node "Unpooling_53" with op = "Unpooling". Please implement this function in the extensions. 
 For more information please refer to Model Optimizer FAQ, question #37. (https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=37#question-37)
[ ERROR ]  
[ ERROR ]  It can happen due to bug in custom shape infer function <UNKNOWN>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'openvino.tools.mo.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "Unpooling_53" node. 
 For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)

Input shape provided in mo is [1,1,512,512].

Environment Specs:

torch==1.12.0
torchvision==0.13.0
numpy==1.19.5
openvino-dev[onnx]==2022.1.0
onnxruntime==1.10.0

You can find the model definition, model weights, onnx file, error and debug log from mo here.

[dkurt]

@likholat, @alexeyhorkin, do you still have an alternative solution with MO Python extension which allows replace MaxUpooling to known ops? Cannot find in openvinotoolkit/openvino_notebooks#487

Update: Never mind

@Rakshith2597, may I ask you to try this approach which let you don't use custom extensions at all: openvinotoolkit/openvino#11400. You need create an alias for unpooling layer, export PyTorch model with it and then convert to OpenVINO IR using Model Optimized extension from the PR.

[Rakshith2597]

@dkurt , Here are the steps I did, please let me know if I am doing it right.

1. Created max_unpool_2d_decomposition at tools/mo/openvino/tools/mo/front/onnx/max_unpool2d_decomposition.py based on link.
2. Exported the pytorch model to ONNX as mentioned in the PR.
3. Tried converting the ONNX model to OpenVINO IR.

Got the following error:

OpenVINO runtime found in:      /home/deeptensor/rakshith_codes/training_extensions/misc/pytorch_toolkit/lung_nodule_detection/venv/lib/python3.9/site-packages/openvino
OpenVINO runtime version:       2022.1.0-7019-cdb9bec7210-releases/2022/1
Model Optimizer version:        2022.1.0-7019-cdb9bec7210-releases/2022/1
[ ERROR ]  Cannot infer shapes or values for node "ATen_53".
[ ERROR ]  There is no registered "infer" function for node "ATen_53" with op = "ATen". Please implement this function in the extensions. 
 For more information please refer to Model Optimizer FAQ, question #37. (https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=37#question-37)
[ ERROR ]  
[ ERROR ]  It can happen due to bug in custom shape infer function <UNKNOWN>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID" (<class 'openvino.tools.mo.middle.PartialInfer.PartialInfer'>): Stopped shape/value propagation at "ATen_53" node. 
 For more information please refer to Model Optimizer FAQ, question #38. (https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html?question=38#question-38)

I think i might have missed something here

You need create an alias for unpooling layer, export PyTorch model with it

Could you please elaborate this? Sorry, I'm new to this.

[dkurt]

Seems fine, but can you try operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK?

[Rakshith2597]

I'm converting my PyTorch model to ONNX using,

torch.onnx.export(model, dummy_input, res_path,
                          input_names=['input'], output_names=['output'],
                          operator_export_type=torch.onnx.OperatorExportTypes.ONNX_ATEN_FALLBACK,
                          verbose=False)

This raises the error mentioned in the previous comment.

[dkurt]

@Rakshith2597, got it, thanks! Is there a way to share model definition so I can reproduce and propose a solution?

[Rakshith2597]

@dkurt , Sure, below is the model definition.

Additionally, I have provided the model definition, model weights, onnx file, error and debug log from mo in GDrive if you need it here.
Thank you for helping. I really appreciate it.

class SUMNet(nn.Module):
    def __init__(self,in_ch,out_ch):
        super().__init__()

        self.encoder   = models.vgg11_bn(pretrained = True).features
        self.preconv   = nn.Conv2d(in_ch, 3, 1)
        self.conv1     = self.encoder[0]
        self.bn1       = self.encoder[1]
        self.pool1     = nn.MaxPool2d(2, 2, return_indices = True)
        self.conv2     = self.encoder[4]
        self.bn2       = self.encoder[5]
        self.pool2     = nn.MaxPool2d(2, 2, return_indices = True)
        self.conv3a    = self.encoder[8]
        self.bn3       = self.encoder[9]
        self.conv3b    = self.encoder[11]
        self.bn4       = self.encoder[12]
        self.pool3     = nn.MaxPool2d(2, 2, return_indices = True)
        self.conv4a    = self.encoder[15]
        self.bn5       = self.encoder[16]
        self.conv4b    = self.encoder[18]
        self.bn6       = self.encoder[19]
        self.pool4     = nn.MaxPool2d(2, 2, return_indices = True)
        self.conv5a    = self.encoder[22]
        self.bn7       = self.encoder[23]
        self.conv5b    = self.encoder[25]
        self.bn8       = self.encoder[26]
        self.pool5     = nn.MaxPool2d(2, 2, return_indices = True)

        self.unpool5   = nn.MaxUnpool2d(2, 2)
        self.donv5b    = nn.Conv2d(1024, 512, 3, padding = 1)
        self.donv5a    = nn.Conv2d(512, 512, 3, padding = 1)
        self.unpool4   = nn.MaxUnpool2d(2, 2)
        self.donv4b    = nn.Conv2d(1024, 512, 3, padding = 1)
        self.donv4a    = nn.Conv2d(512, 256, 3, padding = 1)
        self.unpool3   = nn.MaxUnpool2d(2, 2)
        self.donv3b    = nn.Conv2d(512, 256, 3, padding = 1)
        self.donv3a    = nn.Conv2d(256,128, 3, padding = 1)
        self.unpool2   = nn.MaxUnpool2d(2, 2)
        self.donv2     = nn.Conv2d(256, 64, 3, padding = 1)
        self.unpool1   = nn.MaxUnpool2d(2, 2)
        self.donv1     = nn.Conv2d(128, 32, 3, padding = 1)
        self.output    = nn.Conv2d(32, out_ch, 1)

    def forward(self, x):
        preconv        = F.relu(self.preconv(x), inplace = True)
        conv1          = F.relu(self.bn1(self.conv1(preconv)), inplace = True)
        pool1, idxs1   = self.pool1(conv1)
        conv2          = F.relu(self.bn2(self.conv2(pool1)), inplace = True)
        pool2, idxs2   = self.pool2(conv2)
        conv3a         = F.relu(self.bn3(self.conv3a(pool2)), inplace = True)
        conv3b         = F.relu(self.bn4(self.conv3b(conv3a)), inplace = True)
        pool3, idxs3   = self.pool3(conv3b)
        conv4a         = F.relu(self.bn5(self.conv4a(pool3)), inplace = True)
        conv4b         = F.relu(self.bn6(self.conv4b(conv4a)), inplace = True)
        pool4, idxs4   = self.pool4(conv4b)
        conv5a         = F.relu(self.bn7(self.conv5a(pool4)), inplace = True)
        conv5b         = F.relu(self.bn8(self.conv5b(conv5a)), inplace = True)
        pool5, idxs5   = self.pool5(conv5b)

        unpool5        = torch.cat([self.unpool5(pool5, idxs5), conv5b], 1)
        donv5b         = F.relu(self.donv5b(unpool5), inplace = True)
        donv5a         = F.relu(self.donv5a(donv5b), inplace = True)
        unpool4        = torch.cat([self.unpool4(donv5a, idxs4), conv4b], 1)
        donv4b         = F.relu(self.donv4b(unpool4), inplace = True)
        donv4a         = F.relu(self.donv4a(donv4b), inplace = True)
        unpool3        = torch.cat([self.unpool3(donv4a, idxs3), conv3b], 1)
        donv3b         = F.relu(self.donv3b(unpool3), inplace = True)
        donv3a         = F.relu(self.donv3a(donv3b))
        unpool2        = torch.cat([self.unpool2(donv3a, idxs2), conv2], 1)
        donv2          = F.relu(self.donv2(unpool2), inplace = True)
        unpool1        = torch.cat([self.unpool1(donv2, idxs1), conv1], 1)
        donv1          = F.relu(self.donv1(unpool1), inplace = True)
        output         = self.output(donv1)
        return output

[dkurt]

@Rakshith2597, seems like the following steps helped me convert a model:

1. Restore normal unpooling usage:

self.unpool1   = nn.MaxUnpool2d(2, stride=2)
...
unpool1        = torch.cat([self.unpool1(donv2, idxs1), conv1], 1)  # not .apply

2. Workaround some internal bug by adding intermediate op between pool5 and unpool5 (have not looked how to fix it). Required only for upool5.

unpool5        = torch.cat([self.unpool5(pool5 + 1e-10, idxs5), conv5b], 1)

3. Convert to ONNX with operator_export_type=torch.onnx.OperatorExportTypes.ONNX_FALLTHROUGH
4. Modify max_unpool2d_decomposition.py:

--- a/mo_extensions/front/onnx/max_unpool2d_decomposition.py
+++ b/mo_extensions/front/onnx/max_unpool2d_decomposition.py
@@ -20,7 +20,7 @@ class MaxUnpoolFrontReplacer(FrontReplacementSubgraph):
             nodes=[
                 ("max_pool0", dict(op="MaxPool")),
                 ("max_pool1", dict(op="MaxPool")),
-                ("slice", dict(op="AttributedSlice")),
+                ("slice", dict(op="Slice")),
                 ("sub", dict(op="Sub")),
                 ("unpool", dict(op="max_unpool2d")),
             ],

[Rakshith2597]

@dkurt Thanks a lot for your help. I am able to convert the model to IR.

[Rakshith2597]

@dkurt Hi, on trying to do inference with the ONNX model generated, I'm getting the following error.

[ONNXRuntimeError] : 1 : FAIL : Load model from downloads/model_weights/stage1/lung_seg.onnx failed:Fatal error: max_unpool2d is not a registered function/op

Code used to do inference

net = onnxruntime.InferenceSession('model.onnx')
ort_inputs = {net.get_inputs()[0].name: to_numpy(inputs)}
net_out = net.run(None, ort_inputs)

Code used to create ONNX model

torch.onnx.export(model, dummy_input, res_path,
                          input_names=['input'], output_names=['output'],
                          operator_export_type=torch.onnx.OperatorExportTypes.ONNX_FALLTHROUGH,
                          verbose=False)

Am I missing something?

[dkurt]

Not sure that ONNX Runtime can handle this layer. I've tested OpenVINO 2022.2 and inference was fine.