MONAI vs. MONAI Lable vs. MONAI Deploy

[khamyl]

I am studying the MONAI framework. I started with MONAI Deploy. As the idea was quite compelling I started to explore the MONAI and MONAI Label. But I found out that they do not follow the MONAI Deploy idea i.e. for example the MONAI Label apps (MONAI Core apps) are not Apps built using MONAI Deploy Operators. What I am trying to understand is the whole picture. What is the relationship between MONAI Core/Label/Deploy? Is MONAI core a library of useful classes, MONAI Label is few apps laveraging MONAI Core classes? And if I want I can rewrite MONAI Label apps using MONAI Deploy FW?

Thanks for any help to understand the relations among these frameworks. :)

[vikashg]

Hi @khamyl This is a very relevant question and we are working on building a documentation that can help you understand the differences. Give us few days to compile and provide the answers. However in the mean time you can think of the whole infrastructure as follows

1. MONAI Label: is used for labeling and is available as a plugin in Slicer or OHIF. The idea is that if you have a model behind it which is packaged as a MONAI Bundle you can use that bundle to power those labeling apps.
2. MONAI Core is the algorithmic library that is used for building and training models.
   These models can then be packaged as a model bundle @ericspod is working on creating documentation for building a monai model bundle. Here is the link
3. Now comes the MONAI Deploy App SDK which is another way to handle model deployment. You can wrap a monai bundle in deploy SDK and build a docker container using it or wrap your torchscript model in a monai deploy app SDK and make a docker package out of it for sharing too. MONAI Deploy was developed before MONAI bundle and is geared more towards clinical setting (where radiologists are using clinical viewers for labeling generally).

So your question though valid that why are we not using monai deploy in monai label, that simple answer is bit historical as deploy existed much before bundles and when bundles came through as a self contained package which can also train and do inference, it was plugged into the backend of MONAI label for real-time inferencing during labeling.
Hope this answers your question .
Also here is a quick breakdown of bundle vs MAPS and when to use which,

		Bundle	MAP
	When to use	To share a model with enough information about the pre and post transforms that someone can use to write their application	To share an application package which can be used with minimal effort to deploy an inference in an institutions. It contains piece of code with pre transforms. Although a model might be trained on Nifti JPEG image, the MAPs are geared towards dicom i/o
	Challenges	To ensure that right amount of information is present in the model schema contained in the bundle so that the users can write their code to retrain the model or use it for inferenceing/evaluating	Have enough flexibility to account for different dicom IO and communication scenario. Some of them were heavily discussed on github for example dicom series selector, dicom sr writer etc.
	Research	Mostly research use for building new models and sharing	The research is about building new DICOM pipelines. and writing self contained turnkey applications
	Clinical	Learning from clinical use case need to write code (transforms) to incorporate new datatypes (stl, geometric annotations) etc	Needs research for incorporating messaging, formats like FHIR, HL7 and integrate it in the clinical operations and MONAI
	Training	More or protential research in caching/ better GPU utilization and data pipelining	Not applicable
	Validation	Need to incorporate any non standard metric someone may desire to compute	Not applicable
	Inference	A bundle can be potentially used for inference. Bundle defines the requirements for execution	Main Application of a MAP
	DICOM I/O	Although a dicom file can be used for training it slows down the training process and GPU utilization by a factor of 3. So maybe tools need to be written which account for a proper conversion of a DICOM file to a JPEG or numpy array while maintaining a proper orientation/geomtery spacing etc. There needs to be tools which will incorporate more visual means of verifying that the conversion is correct. It is not easy as dicom formats vary a lot but something to be thought of.	Thre are lot of dicom tools in MAP Thanks to @Ming and @gigon. However it is an evolving thing where new dicom formats specially SR Seg and RTs need to be incorporated in a systematic way. The problem is that there are more disagreements how data should be written and displayed across different viewers and different vendors. This part is challenging.

Hope this clarifies a bit. Adding @wyli @dbericat @pdogra89 for more comments/visibility