Reading data from cloud storage

[St3V0Bay]

Dear MONAI Community,

a question regarding the input for ImageReader (and more specifically WSIReader):

is possible to also use paths from some sort of cloud storage (e.g. S3 or GCS)?

Given the size of WSIs it would be so nice to not copy the data back and forth anymore. In case it's not supported, do you see a chance that this is technically feasible without running into some I/O bottleneck?

Curious about your thoughts.

Thanks in advance.

[pdogra89]

Hi Steffen,

Thanks for your interest. We are looking into this and learning more from our CSP partners, if there are cloud storage APIs that we can leverage to implement this capability in the WSI Image reader.

[steffenvogler]

Hi Prerna,

I just learned that pandas and dask use gcsfuse and s3fs under the hood to accept paths similar to "s3://" and "gcs://". In case I get a little prototype going I'll let you know.

[nvahmadi]

Hi Steffen,

Just wanted to check back whether you were able to resolve this issue, e.g. via gcsfuse or s3fs. We have talked to AWS SAs about this and identified two possibilities

• Using AWS's DICOMweb interface for image loading (accelerated by AWS team). This is an option especially if your WSI data is in DICOM format. Please note: this would require writing a new WSIReader backend in MONAI.
• By combining 1) fast compute node with GPUs (e.g. A100 80GB), and 2) mounting a remote file storage, to stream WSI data in combination with WSIReader cuCIM backend, analoguous to local data. This would require a high performance file system on AWS that can be used to cache files on S3 (FSx on Lustre, with fully managed file storage, see recent re:Invent announcement).

Please let us know if you still need assistance on this, or whether/how you solved the I/O bottleneck?