cv2 error

[pr4deepr]

Hi

I was trying faststain on some tif images and ran into this error

Traceback (most recent call last):
  File "C:\ProgramData\Miniconda3\envs\faststain\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\ProgramData\Miniconda3\envs\faststain\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Miniconda3\envs\faststain\Scripts\faststainnorm.exe\__main__.py", line 7, in <module>
  File "C:\ProgramData\Miniconda3\envs\faststain\lib\site-packages\source\__main__.py", line 217, in main
    run(*vars(ret).values())
  File "C:\ProgramData\Miniconda3\envs\faststain\lib\site-packages\source\__main__.py", line 75, in run
    target = cv2.cvtColor(cv2.imread(reference_image_filename), cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(4.5.3) C:\Users\runneradmin\AppData\Local\Temp\pip-req-build-q3d_8t8e\opencv\modules\imgcodecs\src\loadsave.cpp:77: error: (-215:Assertion failed) pixels <= CV_IO_MAX_IMAGE_PIXELS in function 'cv::validateInputImageSize'

It looks like an opencv error related to input image size. The images are around 800 MB and 1.5GB each (input and reference image).
I tried editing the _ main _.py file and including this
os.environ["OPENCV_IO_MAX_IMAGE_PIXELS"] = str(pow(pow(2,40))) before importing cv2. This doesn't fix it unfortunately.

Cheers
Pradeep

[andreped]

I'm assuming that you are trying to stain normalize the same image as mentioned here:
AICAN-Research/FAST-Pathology#13 (comment)

If so that won't work as the TIFF image you are trying to use is just too large to be kept in memory. But if they are not, could you give me exact information regarding which format this image is stored in and what its image pixel dimensions are.

OpenCV is not meant to handle such large images. However, FAST can given that the image is stored in a tiled, pyramidal format.

Another problem however is that this plugin was not meant to be run on full WSIs, but rather on already extracted image patches. Patches that had for instance been extracted from QuPath.

The reason why I haven't made full WSI image support, is because there are so many scenarios where the normalization could fail, but more importantly, normalization is an extremely slow operation and especially if you wish to write a full normalized WSI.

Regarding catching where macenko fails and handling that is actually something I am working on in another project, and it seems to be working. What I am essentially doing is skipping patches were macenko was unable to convert, due to for instance the H and E channels not being properly represented in a given image patch.

[pr4deepr]

Thanks Andre.
With normalisation of whole WSIs, is it customary to do that?
Also, what is the recommendation for selecting a "reference image" ? Do you select one that has "good" staining and then stick to that for the whole project?

[andreped]

With normalisation of whole WSIs, is it customary to do that?

As with any algorithm applied on WSIs it is almost always necessary to apply methods in a patch-wise fashion. The same applies for stain normalization. A problem with this for this exact problem, is that it can occur patches which are not suitable for the stain normalization algorithm and which it might fail or produce strange results. If you were to apply stain normalization across the entire WSI you should observe this on some tissue types and regions.

Also, what is the recommendation for selecting a "reference image" ? Do you select one that has "good" staining and then stick to that for the whole project?

Regarding reference image, if you are training a model for a specific task and staining (e.g., HE), you should definitely use the same reference image for all future patches and WSIs. That is the whole point of using a reference image. If you suddenly use a different one, for the same model, the colors might change too much for what the model is used to during training, and the whole point of doing "normalization" is gone - you might even get worse performance than if you did no normalization at all! Networks tend to be very sensitive to this, and that is why I most often do stain augmentation instead. But there are use cases where stain normalization has proven to be more useful, such as for mitosis detection, where I believe the task is so difficult that to properly guide the model it is necessary to normalize instead of augment, which preserves information in a standardized manner instead of potentially removing it but making the model more robust.

But it can be said that there has been proposed smarter methods of performing WSI-level stain normalization, but these methods tend to be slow and/or are tricky to implement without a reference implementation, so I have not spent time on that.

Regarding which reference image to use, I don't have a straight forward answer. But I would choose one that best represents the stain color you think is ideal, and at a resolution that makes sense. If you apply a method on x10 magnification, you probably want to use a reference image at the same resolution. But if I were you I would refer to a recent literature study on this exact topic to see what makes the most sense and what other people have tried.

Hope it helps! When I get the time, I will try to add a way to catch faulty patches, to avoid this CLI crashing when applying it on lots of patches. I could either 1) remove the patch entirely or 2) keep the original colors. What do you prefer? Could add an option to toggle between these options.

[pr4deepr]

Thanks a lot @andreped . This is super detailed.
I wasn't aware of the challenge with mitosis detection and how you use normalize instead of augment..

In regard to catching faulty patches, I think adding an option to toggle between the two approaches would be a good idea.