VolPy Mask_RCNN doesn't work

[bobqianic]

My supervisor gave me a piece of code. I ran it on the latest version of CaImAn, but Mask R-CNN consistently produces no ROIs, while it works well with the TensorFlow version utils.mrcnn_inference using the same config.

   ############ Segmentation

   # Create mean and correlation images
   img = mean_image(mc.mmap_file[0], window = 1000, dview=dview)
   img = (img-np.mean(img))/np.std(img)


   Cn = local_correlations_movie_offline(mc.mmap_file[0], fr=fr, window=fr*4, #### [0] ???
                                         stride=fr*4, winSize_baseline=fr, 
                                         remove_baseline=True, gaussian_blur=gaussian_blur,
                                         dview=dview).max(axis=0)
   img_corr = (Cn-np.mean(Cn))/np.std(Cn)
   summary_images = np.stack([img, img, img_corr], axis=0).astype(np.float32)

   # Save summary images which could be further used in the VolPy GUI
   cm.movie(summary_images).save(cells_savedir+ '//summary_images.tif')
   print(cells_savedir+ '//summary_images.tif')

   use_maskrcnn = True  # set to True to predict the ROIs using the mask R-CNN
   if not use_maskrcnn:                 # use manual annotations
       with h5py.File(path_ROIs, 'r') as fl:
           ROIs = fl['mov'][()]  # load ROIs
   else:
       weights_path = "/root/caiman_data/model/mask_rcnn"  
   
       ROIs = utils.mrcnn_inference_pytorch(img=summary_images.transpose([1, 2, 0]), size_range=size_range,#[5, 22], size_range=[5,45]
                                     weights_path=weights_path, display_result=True) # size parameter decides size range of masks to be selected
       cm.movie(ROIs).save(cells_savedir + '//mrcnn_ROIs.hdf5')
       print(cells_savedir + '//mrcnn_ROIs.hdf5')

[bobqianic]

The summary images produced are identical

sha256sum

26cb7dd32991d4b734d1744bfb132227e88632f63a1829a09af56ae7c5ab80c0  latest
26cb7dd32991d4b734d1744bfb132227e88632f63a1829a09af56ae7c5ab80c0  old

[pgunn]

@manuelpaeza Can you give this a look?

[bobqianic]

demo_pipeline_voltage_imaging.ipynb

PyTorch-CalmAn 1.13.1:

Tensorflow-CalmAn 1.12.2:

[bobqianic]

@pgunn Thank you for looking into this issue. This problem cost me two days and caused quite a bit of confusion. Initially, I tested the demo and saw that there was some ROIs, but it couldn’t identify all cells. I thought it was a threshold issue. However, when I tried it on my experimental data, there was zero ROI.

I suspected there might be a bug in my code, so I asked my supervisor for his code, but it still didn’t work. That’s when I realized there must be a bug in the package itself.

I didn’t expect this to happen in this project. I thought this was a well-established and actively maintained piece of software with over 700 stars. I’ll look into this further, try to identify the problem, and propose a pull request. I’ll also add this to the CI to prevent similar regressions in the future.

[pgunn]

@bobqianic Ah, sorry about that; the rewrite of the machine learning code from tensorflow to pytorch touched a lot of components of the codebase, and volpy is unfortunately not the most heavily-maintained or well-understood part of the codebase (it was dropped off by a contributor some years back and is not really fully integrated into the rest of caiman). We do our best, but volpy is a weak spot in terms of maintenance and support.

I hope longer-term we can find a way to integrate it better with the rest of the code and improve its testing.

[pgunn]

@bobqianic The person who ported the code to torch has a possible fix which he's testing for comparison; if things look good we may be able to merge a fix into dev soon.

[pgunn]

@bobqianic Could you give this branch a try with your data?
#1568

[bobqianic]

@pgunn I just tested that branch on demo data. As you can see, the result still looks off, there’s more ROI, but some are in the wrong position. This didn’t happen in the TF version (see image I posted earlier)

[bobqianic]

I also tested this on my experimental data. It looks like we’re only getting one ROI? The mask shape is (400, 512), but it’s hard to tell because there are some internal errors (shape mismatch) in the new code. Either way, it’s still way off compared to the TF version, I can get 16 correct ROIs there.

[manuelpaeza]

Hello. Apologies for the lateness on this, I will be looking at this and giving you an update soon. If you have anything additional to share, please let me know.