the process would be blocking in Validation step

[lyingflatDDD]

Problem:
I have a problem. when i train the model, the process would be blocking in validation step(or validation sanity check) but in the training step, it can work. And I debug in the pytorch-lightning , i found when loading data from validation dataloader it would be blocking. I am not sure what problem.

Environment:
docker; the lastest pytorch-lighting;gpu a100

log:
INFO Using validation DataLoader3DOffset with {}
INFO Building Sampling Cache for Dataloder
Sampling Cache: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1445.07it/s]
INFO Using 5 num_processes and 2 num_cached_per_queue for augmentation. | 0/2 [00:00<?, ?it/s]
INFO VALIDATION KEYS:
odict_keys(['case_0', 'case_7'])
using pin_memory on device 0

---

I test the validation step and it world jam

can you help me, thank you !!!!!

[akihironitta]

i found when loading data from validation dataloader it would jam.

Hi @clearknow, by "jam", you mean the process hangs? Would it be possible for you to provide your env details and script?

[lyingflatDDD]

thank you for your answer! sorry, I mean the process would be blocken.
I am very confused because the training step can work and validation step would block.

trainer = pl.Trainer(
    gpus=list(range(num_gpus)) if num_gpus > 1 else num_gpus,
    accelerator=cfg["trainer_cfg"]["accelerator"],
    precision=cfg["trainer_cfg"]["precision"],
    amp_backend=cfg["trainer_cfg"]["amp_backend"],
    amp_level=cfg["trainer_cfg"]["amp_level"],
    benchmark=cfg["trainer_cfg"]["benchmark"],
    deterministic=cfg["trainer_cfg"]["deterministic"],
    callbacks=callbacks,
    logger=pl_logger,
    max_epochs=module.max_epochs,
    progress_bar_refresh_rate=None if bool(int(os.getenv("det_verbose", 1))) else 0,
    reload_dataloaders_every_epoch=False,
    num_sanity_val_steps=10,
    weights_summary='full',
    plugins=plugins,
    terminate_on_nan=True,  # TODO: make modular
    move_metrics_to_cpu=False,
    **trainer_kwargs
)
trainer.fit(module, datamodule=datamodule)

---

[lyingflatDDD]

sorry, It is my fault. now i solve this problem

[lyingflatDDD]

sorry, It is my fault. now i solve this problem