Combine PatchDataset and SmartCacheDataset

[jxchen01]

Is there a way to combine PatchDataset and SmartCacheDataset?

Use case: I have 2000 large 3D microscopy images to train on. From each image, I want to randomly sample 5 patches each time. Apparently, loading or caching all 2000 x 5 = 10000 patches is not quite feasible. So, I want to borrow SmartCacheDataset, so that each epoch, I will cache 200 3D volumes, each with 5 randomly sampled patches (so, 1000 patches each epoch, which is doable for my machines), then in the next epoch, I can replace maybe 50 out of 200 3D volumes from all the 2000 3D images.

From the online resources, I still cannot figure out how to do this with current functions.

Many thanks,
Jianxu

[Nic-Ma]

Hi @jxchen01 ,

I think you can combine PatchDataset and SmartCacheDataset like:

patch_dataset = PatchDataset(...)
dataset = SmartCacheDataset(data=patch_dataset, persistent_rate=...)

Thanks.

[jxchen01]

Woohoo! Thanks @Nic-Ma ! I will try it out and report back here if it works.

[jxchen01]

@Nic-Ma I read through more tutorials online. I realize that the patch extraction step is always inserted as one of the transformation steps for training, such as in the spleen_segmentation_3d.ipynb. This actually makes it very easy to use patch based sampling and SmartCacheDataset together. Whatever in the cache , each will generate certain number of samples by the sampling transform.

Then, I am a little puzzled by the underlying design logic between PatchDataset() and the random sampling transformations (such as https://docs.monai.io/en/stable/transforms.html#randweightedcropd). If I understand correctly, with the sampling transformations, the PatchData will not be quite useful anymore. Is that correct?

Thanks!

[Nic-Ma]

Hi @wyli ,

Could you please help share some typical use cases for PatchDataset?

Thanks in advance.