use MONAI for analyzing the camelyon16 datasets

[yqyuhao]

when I run this code for analyzing the camelyon16 datasets, I meet the error. How can I solve this problem?
CUDA_VISIBLE_DEVICES=0,1 python3 -u code/mil_train_evaluate_pytorch_gpu.py
--data_root=dataset
--dataset_json=./datalist_camelyon16.json
--amp
--distributed
--mil_mode=att_trans
--batch_size=32
--epochs=50
--logdir=./logs

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
fn(i, *args)
File "/workspace/notebooks_monai/code/mil_train_evaluate_pytorch_gpu.py", line 411, in main_worker
train_loss, train_acc = train_epoch(model, train_loader, optimizer, scaler=scaler, epoch=epoch, args=args)
File "/workspace/notebooks_monai/code/mil_train_evaluate_pytorch_gpu.py", line 74, in train_epoch
for idx, batch_data in enumerate(loader):
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 530, in next
data = self._next_data()
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1225, in _next_data
return self._process_data(data)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1251, in _process_data
data.reraise()
File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 463, in reraise
raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/opt/monai/monai/transforms/transform.py", line 89, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "/opt/monai/monai/transforms/transform.py", line 53, in _apply_transform
return transform(parameters)
File "/opt/monai/monai/transforms/io/dictionary.py", line 131, in call
data = self._loader(d[key], reader)
File "/opt/monai/monai/transforms/io/array.py", line 240, in call
img_array, meta_data = reader.get_data(img)
File "/opt/monai/monai/data/image_reader.py", line 832, in get_data
region = self._extract_region(img, location=location, size=size, level=level, dtype=dtype)
File "/opt/monai/monai/data/image_reader.py", line 910, in _extract_region
region = img_obj.asarray(level=level)
File "/opt/conda/lib/python3.8/site-packages/tifffile/tifffile.py", line 3851, in asarray
result = page0.asarray(out=out, maxworkers=maxworkers)
File "/opt/conda/lib/python3.8/site-packages/tifffile/tifffile.py", line 7661, in asarray
for _ in self.segments(
File "/opt/conda/lib/python3.8/site-packages/tifffile/tifffile.py", line 7497, in segments
yield from executor.map(decode, segments)
File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/opt/conda/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/opt/conda/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/conda/lib/python3.8/site-packages/tifffile/tifffile.py", line 7472, in decode
return func(keyframe.decode(*args, **decodeargs))
File "/opt/conda/lib/python3.8/site-packages/tifffile/tifffile.py", line 7651, in func
out[
IndexError: index 1 is out of bounds for axis 0 with size 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/monai/monai/transforms/transform.py", line 89, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "/opt/monai/monai/transforms/transform.py", line 53, in apply_transform
return transform(parameters)
File "/opt/monai/monai/transforms/compose.py", line 173, in call
input = apply_transform(transform, input, self.map_items, self.unpack_items, self.log_stats)
File "/opt/monai/monai/transforms/transform.py", line 113, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.io.dictionary.LoadImaged object at 0x7f8879bd4790>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
data = fetcher.fetch(index)
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/monai/monai/data/dataset.py", line 97, in getitem
return self._transform(index)
File "/opt/monai/monai/data/dataset.py", line 83, in _transform
return apply_transform(self.transform, data_i) if self.transform is not None else data_i
File "/opt/monai/monai/transforms/transform.py", line 113, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f8879bd4760>

[Nic-Ma]

Hi @yqyuhao ,

Thanks for your interest here.
Could you please help share the test program and sample image for further analysis?
We just totally refactored the pathology module recently, so maybe you can try it again with the latest MONAI v0.9 at end of next week.
CC @drbeh .

Thanks.

[yqyuhao]

CUDA_VISIBLE_DEVICES=0,1 python3 -u code/mil_train_evaluate_pytorch_gpu.py
--data_root=dataset
--dataset_json=./datalist_camelyon16.json
--amp
--distributed
--mil_mode=att_trans
--batch_size=32
--epochs=50
--logdir=./logs

code:
mil_train_evaluate_pytorch_gpu.py
datalist_camelyon16.json

camelyon16_dataset:
https://camelyon17.grand-challenge.org/

[Nic-Ma]

Hi @myron ,

As @drbeh is on vacation now, could you please help take a look at this question?
I think maybe the newly refactored APIs don't have this issue?

Thanks in advance.

[bhashemian]

Hi @yqyuhao,

It seems that there is a problem with loading the images that you have with the tifffile. To isolate the issue, can you do the following checks:

1. simply replace "TiffFile" with "cuCIM" in the code (lines 355 and 375) and see if the error persisits?

LoadImageD(keys=["image"], reader=WSIReader, backend="TiffFile", dtype=np.uint8, level=1, image_only=True),

2. Also change the level to a lower resolution like level=6 to see if you don't have CPU memory issue.
3. At the end, can you check your whole slide images and see if they are fine? You can just load them individually at level=1 (if you have enough memory) to see if there is any problem with loading them?
   Either directly using TiffFile or cuCIM or WSIReader in monai as follow:

from monai.data import WSIReader
filename="your_wsi_file.tiff"
reader = WSIReader(backend="cucim")
obj = reader.read(filename)
img, _ = reader.get_data(obj, level=1)

Please let me know the result of your checks. Thanks

[yqyuhao]

I use the four nvidia gpu P100 devices to running the following code, but the result is not best. How can I adjust the para to get the best result.

LoadImageD(keys=["image"], reader=WSIReader, backend="cuCIM", dtype=np.uint8, level=6, image_only=True)

CUDA_VISIBLE_DEVICES=0,1 python3 -u code/mil_train_evaluate_pytorch_gpu.py
Argument values:
data_root => dataset
dataset_json => ./datalist_camelyon16.json
num_classes => 2
mil_mode => att_trans
tile_count => 49
tile_size => 256
checkpoint => None
validate => False
logdir => ./logs
epochs => 100
batch_size => 4
optim_lr => 3e-05
weight_decay => 0
amp => True
val_every => 1
workers => 2
distributed => True
world_size => 1
rank => 0
dist_url => tcp://127.0.0.1:23456
dist_backend => nccl
quick => False

[yqyuhao]

The results is here
run-logs-tag-val_qwk.csv
run-logs-tag-val_loss.csv
run-logs-tag-val_acc.csv
run-logs-tag-train_loss.csv
run-logs-tag-train_acc.csv

[bhashemian]

Hi @myron,

Do you have any insight to share with @yqyuhao to improve their MIL model?

Thanks

[ChengXianhang]

Hello, I also encountered this problem, how did you solve it