Synthetic data in monai based on a monai.apps.SyntheticDataset class

[RichardObi]

Hello 👋
The following proposal is based on several discussions initiated by @nvahmadi about a potential integration of the medigan library into MONAI. In medigan, users can generate their own synthetic datasets based on pretrained generative models. We think this could be a useful concept to bring to MONAI. In particular, MONAI could have a synthetic dataset class that allows users to generate, load, and interact with synthetic datasets. Apart from synthetic data generation, another potential use-case of this class is the translation of user data from one domain to another using pretrained generative models (e.g., cycleGAN).

For this purpose, we thought that a potential extension of monai.apps could be interesting, as this module already contains classes that implement specific datasets such as the monai.apps.DecathlonDataset and the monai.apps.TciaDataset. We suggest to define a general monai.apps.SyntheticDataset class as a central point for handling synthetic datasets in MONAI. Any specific synthetic datasets such as, for example, a monai.apps.MediganDataset or a BrainLDMDataset (e.g. a dataset generated by running a generative model from the model zoo) could inherit from this base SyntheticDataset class.

This suggestion was previously discussed with @wyli @pdogra89 @daguangxu @nvahmadi (29.11.2022) and with the Project-MONAI/GenerativeModels team (28.02.2023) @Warvito @ericspod.

[RichardObi]

In the case of a potential medigan integration, the existing code of medigan can be adjusted to fit, for example, any of these two scenarios:

1. One could import medigan (as optional external dependency) inside monai.apps.MediganDataset
2. Or, similarly to monai.apps.tcia, one could create a monai.apps.medigan (or, more general, monai.apps.synthesis) module into which all of the necessary code from medigan could be migrated.

[wyli]

I think option 1 a MediganDataset wrapping the ModelExecutor.generator makes sense, https://github.com/RichardObi/medigan/blob/main/src/medigan/execute_model/model_executor.py#L283

perhaps as a first step you could create a jupyter notebook with the dataset class and a simple end-to-end monai+medigan model training demo in https://github.com/Project-MONAI/tutorials? hopefully it'll increase the visibility of your work for the monai users and we can do further integrations according to the users feedback.

[nvahmadi]

Sounds good. What about @RichardObi's initial idea to have a parent class monai.apps.SyntheticDataset, from which a class MediganDataset(SyntheticDataset) would derive? I found that elegant. On the other hand, it would take longer, because the teams would need to define and agree on a suitable structure for the SyntheticDataset class?

[wyli]

yes agreed, a base class is useful for introducing the basic concepts and defining the capability/scope, would be great to have a draft pull request and then we can invite more people to discuss...

(and a MediganDataset demo could be a quick prototype and when developing it we'll have a better idea about how to shape the base class)