Models Inconsistency

[blaginin]

• TIA Toolbox version: 1.3.3
• Python version: 3.10.8

Description

Tiatoolbox has several pre-trained models helpful for data processing. However, models differ in how they handle input and output, making them confusing to use (especially when customizing):

• Activation functions apply at different steps for different models. Sometimes in forward method (e.g. CNNModel), sometimes forward returns a raw layer output and the transformation applies in infer_batch (e.g. UNetModel).
• Moreover, activation functions are hardcoded. To customize, you can't simply change an attribute; you must overwrite the whole method (different for each model).
• Data normalization distributes across methods: HoVerNet uses it in forward, UNetModel in _transform, MicroNet in preproc, and vanilla models rely on the user to do so.
• Data preprocessing also lacks consistency. Even though it should happen in preproc_func/_preproc functions, UNetModel uses its own _transform, unrelated to the standard methods. Yet, its behavior could implement in _preproc.

What to do

Refactoring the code will significantly improve readability:

• Decompose the pipeline into small granular methods in ModelABC: one method for normalization, activation function as an attribute, etc.
• Explain ModelABC methods in their documentation: does infer_batch rely on postproc_func? Can infer_batch be used for training? How?
• Reorganize custom model methods to match the new ModelABC structure.
• Add a new page to the documentation explaining the Tiatoolbox models pipeline: how is it related to the PyTorch pipeline? How to evaluate a model? How to train a model?

[blaginin]

I guess it will be fixed by #635

[Ahmad-Tamim-Hamad]

I have my own model ViT (vision transformer) I have trained my model and saved the best weight of the model, I want to run tia toolbox on my own data how can I use it my own model and weight of the model patches and WSI, please help me.

[GeorgeBatch]

I have my own model ViT (vision transformer) I have trained my model and saved the best weight of the model, I want to run tia toolbox on my own data how can I use it my own model and weight of the model patches and WSI, please help me.

@tamim1992 You can wrap your ViT into a format compatible with tiatoolbox like shown in this notebook: https://github.com/TissueImageAnalytics/tiatoolbox/blob/develop/examples/07-advanced-modeling.ipynb

[Ahmad-Tamim-Hamad]

Thank you for your response. The model performs well at the patch level, but when I apply it to overlay on my own data, the results aren't as good. Do you have any suggestions?