validation_step causes deadlock?

[pengzhenghao]

Hi there,

my model requires a little bit "unconstructed evaluation", which means that the model's return is a complex dict where some values are non-tensors such as strings. The evaluation function will turn tensor to numpy array and so some computing in CPU.

After introducing this code in my PTL model, the multi-gpu training will hang after the validation but before the first training step.

Does anyone has any idea on this? Thank!!!!

    def validation_step(self, batch, batch_idx):
        batch_pred_dicts = self(batch)
        final_pred_dicts = generate_prediction_dicts(batch_pred_dicts, rank=self.global_rank)
        result_str, result_dict = evaluation([final_pred_dicts])
        self.log_dict({f"eval/{k}": float(v) for k, v in result_dict.items()})

[pengzhenghao]

Now I understand what happens. For some reason, DDP doesn't like unstructured data and it only accepts tensor / dict of tensors / list of tensors or so.

Also, just mention here, all_gather operation has the same flavor too. It even doesn't support gather tensor with different shapes.

The conclusion is we need to avoid unstructured data.