Running out shared memory.

[YerePhy]

Hi,

I'm training a monai.networks.nets.DynUNet with deep supervision in a SLURM cluster, my dataset is a big folder of files. After 5 epochs my training crash due to an error realated to shared memory. In short, the errors I get are the following:

• RuntimeError: unable to open shared memory object </torch_3368351_1991753163> in read-write mode
• RuntimeError: Caught RuntimeError in DataLoader worker process 0.

Until here, this question might be better placed in the PyTorch forum. However, I'm getting a lot of messages like this one:

dev_collate - CRITICAL - >>> collate/stack a list of x, where x=numpy arrays, list of tensors, "xyzt_units", "cal_max" out of 42 keys, etc.

I think that this is related to MONAI. You can see the whole error trace here

Has someone faced something similar?

[yiheng-wang-nv]

Hi @YerePhy , the error message is logged from monai.data.utils.dev_collate and enhances pytorch's default_collate. It is used for diagnostic. It seems the issue is in the data loading part, and could you provide more information, such as what are your transforms? The input shape, gpu memory, cpu memory, monai version and so on.
In addition, according to the warnings, the cropping transform may be wrongly used.

[YerePhy]

Hi @yiheng-wang-nv,

Thanks for your answer, my hardware is :

• CPU count 64 (don't have more info about CPU, but according to the guidelines given to us 32 CPU cores should be enough)
• GPU count 1
• GPU type NVIDIA A100-PCIE-40GB
• My dependencies (I must use that PyTorch version, cannot upgrade bacuse it won't work):

ipython==7.12.0
entrypoints==0.4
traitlets==5.3.0
-f https://download.pytorch.org/whl/torch_stable.html
torch==1.9.0+cu111
-f https://download.pytorch.org/whl/torch_stable.html
torchvision==0.10.0+cu111
pytorch-lightning==1.6.5
torchmetrics==0.9.1
monai==0.9.0
torchio==0.18.80
kornia==0.6.6
hydra-core==1.2.0
numpy==1.22.4
scikit-learn==1.1
scikit-image==0.19.3
nibabel==4.0.1
wandb==0.12.21
matplotlib===3.5.2
setuptools==59.5.0

• Shapes:
  • INPUTS: 4 tensors of (192, 192, 192) concatenated into one tensor of (4, 192, 192, 192) then 4 patches of (4, 96, 96, 96) are extracted.
  • GROUND TRUTH: 1 tensor of (192, 192, 192) then it becomes a tensor with shape (3, 192, 192, 192) then 4 patches of (3, 96, 96, 96) are extracted as in INPUTS.
• These are my transforms:

_target_: monai.transforms.Compose
transforms:
  - _target_: monai.transforms.LoadImaged
    keys: [T1, T1CE, T2, FLAIR, GROUND_TRUTH]
  - _target_: monai.transforms.AddChanneld
    keys: [T1, T1CE, T2, FLAIR, GROUND_TRUTH]
  - _target_: monai.transforms.ConcatItemsd
    keys: [T1, T1CE, T2, FLAIR]
    name: INPUTS
  - _target_: monai.transforms.ConvertToMultiChannelBasedOnBratsClassesd
    keys: [GROUND_TRUTH]
  - _target_: monai.transforms.BorderPadd
    keys: [INPUTS, GROUND_TRUTH]
    spatial_border: 48
  - _target_: monai.transforms.CastToTyped
    keys: [GROUND_TRUTH]
    dtype: float

My dataset is built as follows (well, I'm using a pytorch_lightning.LightningDataModule, but this is the genral idea):

def build_monai_patch_dataset(
    subjects: Sequence[Subject],
    transforms: transforms.Compose,
    patch_fn: Optional[transforms.Transform] = None,
    patch_transforms: Optional[transforms.Compose] = None,
) -> data.Dataset:
    """
    Builds a patch dataset or a dataset with complete volumes
    depending on the aguments given.

    Args:
        subjects: dictionary data for building the dataset.
        transforms: transformations applied to the images.
            Must be ``monai.transforms.Transform`` instances
            encapsulated in a ``monai.transforms.Compose``.
        patch_fn: a ``monai.transforms.Transform`` that
            yields lists of patches from a volume, per example
            ``transforms.RandCropByLabelClassesd``.
        patch_transforms: transformations applied to the patches.
            Must be ``monai.transforms.Transform`` instances
            encapsulated in a ``monai.transforms.Compose``.

    Returns:
        A ``monai.data.Dataset`` instance with the transformations applied.

    """
    subjects_ = data.Dataset(subjects, transforms)

    if patch_fn is not None:
        assert hasattr(
            patch_fn, "num_samples"
        ), "``patch_func`` must have ``num_samples`` attribute."
        return data.PatchDataset(
            subjects_,
            patch_func=patch_fn,
            samples_per_image=patch_fn.num_samples,
            transform=patch_transforms,
        )

    return subjects_
# then fed the output to monai.data.DataLoader