Mil model Backbone input channel as a parameter

[relyativist]

Thanks for making MIL model tutorial public, I am trying to experiment with monai mil_model on my data. I have an frames of images with tensor shape (b, n, c, h, w), x = (16, 97, 1, 128, 128), thus my frames of images have only 1 channel. I face with the Runtime Error when I init the model and try run model(x):

--> 453 return F.conv2d(input, weight, bias, self.stride,
    454                 self.padding, self.dilation, self.groups)

File ~/miniconda3/envs/torch1-13_cuda11-3/lib/python3.9/site-packages/monai/data/meta_tensor.py:268, in MetaTensor.__torch_function__(cls, func, types, args, kwargs)
    266 if kwargs is None:
    267     kwargs = {}
--> 268 ret = super().__torch_function__(func, types, args, kwargs)
    269 # if `out` has been used as argument, metadata is not copied, nothing to do.
    270 # if "out" in kwargs:
    271 #     return ret
    272 if _not_requiring_metadata(ret):

File ~/miniconda3/envs/torch1-13_cuda11-3/lib/python3.9/site-packages/torch/_tensor.py:1121, in Tensor.__torch_function__(cls, func, types, args, kwargs)
   1118     return NotImplemented
   1120 with _C.DisableTorchFunction():
-> 1121     ret = func(*args, **kwargs)
   1122     if func in get_default_nowrap_functions():
   1123         return ret

RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[97, 1, 128, 128] to have 3 channels, but got 1 channels instead

I assume the Backbone ResNet have input channel of 3, and this throws the Error. Then I tried to

model = milmodel.MILModel(num_classes=11, pretrained=True, mil_mode="att")
model.net.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
model.to(device)

But this also throws the error:

--> 453 return F.conv2d(input, weight, bias, self.stride,
    454                 self.padding, self.dilation, self.groups)

File ~/miniconda3/envs/torch1-13_cuda11-3/lib/python3.9/site-packages/monai/data/meta_tensor.py:268, in MetaTensor.__torch_function__(cls, func, types, args, kwargs)
    266 if kwargs is None:
    267     kwargs = {}
--> 268 ret = super().__torch_function__(func, types, args, kwargs)
    269 # if `out` has been used as argument, metadata is not copied, nothing to do.
    270 # if "out" in kwargs:
    271 #     return ret
    272 if _not_requiring_metadata(ret):

File ~/miniconda3/envs/torch1-13_cuda11-3/lib/python3.9/site-packages/torch/_tensor.py:1121, in Tensor.__torch_function__(cls, func, types, args, kwargs)
   1118     return NotImplemented
   1120 with _C.DisableTorchFunction():
-> 1121     ret = func(*args, **kwargs)
   1122     if func in get_default_nowrap_functions():
   1123         return ret

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

I assume the second error is considered with input type Error. However, can input channel be a parameter of MIL model?
Thanks

[KumoLiu]

Hi @relyativist, thanks for your interest here.
I agree that since we were previously using torchvision.model, we may need to refactor the entire model to support adding 'input_channel'.

Considering your particular issue, I guess you may forget to send the data to the same device with the model.
Hope it helps, thanks!

[relyativist]

@KumoLiu Yes, I was able to resolve an issue with Input Type, for now I changed input conv channel dimension in a way I mentioned above.