Model with multiple features

[Aishuvenkat09]

Hi ! Maybe a rookie question but how do I customize the model to take two input features

I'm working on an BoneAge Prediction problem. I want to feed the model with 2 inputs (Image and Gender) but not sure how to do that. Do I have to completely customize the monai model architecture to let the forward() function to take multiple inputs ? Any resources to such usecase would be a great help thank you!

[Nic-Ma]

Hi @Aishuvenkat09 ,

Thanks for your interest here.
May I know what network you are trying to use for your task?

Thanks.

[Aishuvenkat09]

I’m just trying out with DenseNet but ideally I would like use Pretrained Vgg16, inception and xception models to benchmark performance compared to models built on tensorflow. Regards, Aishwarya Vantipuli

…

________________________________ From: Nic Ma ***@***.***> Sent: Tuesday, March 22, 2022 6:17:49 PM To: Project-MONAI/MONAI ***@***.***> Cc: Aishwarya ***@***.***>; Mention ***@***.***> Subject: Re: [Project-MONAI/MONAI] Model with multiple features (Discussion #3981) Hi @Aishuvenkat09<https://github.com/Aishuvenkat09> , Thanks for your interest here. May I know what network you are trying to use for your task? Thanks. — Reply to this email directly, view it on GitHub<#3981 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJBMFS7WYSFO33X5V4CKF6DVBJBI3ANCNFSM5RMD7TOQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[ebrahimebrahim]

What you are trying to do is called multimodal fusion, so that would be a good thing to read up on.

This article is useful and lists some of your options: https://www.nature.com/articles/s41746-020-00341-z

[Aishuvenkat09]

Thank you @ebrahimebrahim. I have done something similar to early fusion in PyTorch. But I want to do the same in MONAI but not sure how to do that.
Also, can I use any network like (inception) and implement that in MONAI ? or is it advisable to use MONAI networks to get most out of them.