Re-using cache data of PersistentDataset on new runs (when using Azure blob storage).

[jhairgallardo]

Hello,
I am using PersistentDataset on a large dataset. Is it possible to re-use the cache data on a new run instead of creating the cache data again? I notice that, if I use the same folder where I cached the data before, it won't re-use the files; instead, it will create new files (increasing the disk memory usage of that folder). I am new at using Monai, so I am not sure if I am missing some input argument or configuration that can allow PersistentDataset to re-use cache data generated on previous runs. (The transformations don't change between runs). Thank you!

[ebrahimebrahim]

It is indeed supposed to re-use the data that it caches to disk. Like if the data is there in the directory that you pointed it to, then it should use that data rather than creating new files. So it sounds like something is wrong.

[jhairgallardo]

I see. My cache folder is a blob storage in azure. I will test this using a local folder instead. Maybe using the blob storage folder is causing some issues.

[wyli]

the persistent dataset finds cache content by hash of input dictionaries (in this format

MONAI/monai/data/dataset.py

Lines 179 to 182 in 317ef1f

	[{ { {
	'image': 'image1.nii.gz', 'image': 'image2.nii.gz', 'image': 'image3.nii.gz',
	'label': 'label1.nii.gz', 'label': 'label2.nii.gz', 'label': 'label3.nii.gz',
	'extra': 123 'extra': 456 'extra': 789

) perhaps in your use cases there are changes in the dictionaries?

[jhairgallardo]

Thank you for your prompt response! When I use a local folder in cache_dir, monai re-uses the same files from a previous run. so, it works in that case! However, it seems that it doesn't reuse the files when I am pointing to a blob storage folder and do a second run there. (The first one created the files in the blob storage folder)

@wyli yes, I am using that format. Since it works just fine when using a local folder, I think the issue is related to using a remote folder that is mounted on. It could also be related to run the code in an azure cluster. Everything works fine locally in my own PC, but when I run it on azure (and use a blob storage folder for cache_dir) it doesn't reuse the files.

[jhairgallardo]

I found the issue when using a blob storage for PersistentDataset. It doesn't re-use the files because when mounting the blob, it creates a new mounted path everytime in the azure cluster node, causing the hash to not match the new folder. It is not an issue with MONAI, but with the way mounting a folder in azure cluster instances works. I will post here if I find a work around, but I think we can close this discussion for now since it is not a MONAI's issue. Thank you!