Incorrect output shape when setting pool_size as (odd, even) on MaxPooling2D

[maybeLee]

I was trying to convert a Keras to ONNX then to PyTorch through onnx2pytorch but failed.

The model I was trying to convert is as follows with input shape as (1, 32, 32, 3):

layer_stack = [
    keras.layers.MaxPooling2D(pool_size=(2,1), strides=1, padding='same'),
    keras.layers.Flatten(),
    keras.layers.Dense(100)
]

After adding the patch from this pr: #28
There still occurs one error as follows:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1846     if has_torch_function_variadic(input, weight, bias):
   1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-> 1848     return torch._C._nn.linear(input, weight, bias)
   1849 
   1850 

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x3069 and 3072x100)

After some investigations, I think the bug lies in how onnx2pytorch infers the padding on some special cases.
Below is what I have done for debugging:
I find that when setting the pool_size parameter to be (odd, even) or (even, odd), the actual MaxPooling2D's output size is not (1, 32, 32, 3) even though we set the padding to be 'same' (which means the output size should be the same as input size) And that's why crash happens. Run below code to see what's going on:

# generate model
import keras
x = keras.layers.Input((32,32,3))
layer_stack = [
    keras.layers.MaxPooling2D(pool_size=(2,1), strides=1, padding='same'),
]
layer_input = x
for layer in layer_stack:
  y = layer(layer_input)
  layer_input = y
model = keras.Model(x, y)
model.summary()

# convert model to onnx
import tensorflow as tf
import tf2onnx
# transform model
input_shape = model.layers[0].input_shape[0]
spec = (tf.TensorSpec(input_shape, tf.float32, name="input"),)
model, _ = tf2onnx.convert.from_keras(model, input_signature=spec, \
    opset=15, output_path="temp.onnx")

# load onnx to pytorch using onnx2pytorch
import onnx
from onnx2pytorch import ConvertModel
import numpy as np
import torch
onnx_model = onnx.load("temp.onnx")
model = ConvertModel(onnx_model)
input = np.random.rand(1, 32, 32, 3)
torch_input = torch.from_numpy(input)
model.double()
res = model(torch_input)
res.size()

The output torch size is [1, 32, 33, 3] instead of [1, 32, 32, 3]. I initially thought it may be the bug of tf2onnx that incorrectly generate the padding parameter. However, when I directly inference the onnx model using onnxruntime, it can output the correct size [1, 32, 32, 3]. The code to inference onnx model is as follows:

import onnxruntime as ort
session = ort.InferenceSession("temp.onnx")
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name
input = np.random.rand(1, 32, 32, 3).astype('float32')
result = session.run([output_name], {input_name: input})
print(result[0].shape)

After such cross-checking, I then think it may be a bug inside onnx2pytorch. So I checked the generated model and find the following information:

ConvertModel(
  (Transpose_max_pooling2d_1/MaxPool__6:0): Transpose()
  (MaxPool_max_pooling2d_1/MaxPool_raw_output___4:0): Sequential(
    (0): ConstantPad2d(padding=(0, 1, 0, 0), value=0)
    (1): MaxPool2d(kernel_size=(2, 1), stride=(1, 1), padding=0, dilation=1, ceil_mode=False)
  )
  (Transpose_max_pooling2d_1): Transpose()
)

I then manually try to change the padding parameter and find out that if we change the padding parameter in ConstantPad2d from (0, 1, 0, 0) to (0, 0, 1, 0). Pytorch can predict the correct shape.

In conclusion, I assume it may be a bug inside onnx2pytorch's padding inference phase.
All mentioned code above can be accessed through this colab link:
https://colab.research.google.com/drive/1LnPiSZy08rmZFRsGoEPjsCJFyBPL0_K5?usp=sharing