Abnormally slow both single-gpu & DDP training, what is the problem here?

[Mollylulu]

Bug description

I just adapt my training into lightning framework for convenient ddp model training. But I got almost 10 times slower than my previous manually torch ddp training, the speed is shown below. I have no idea what is wrong here, could anyone help me figure out what may caus this problem and how to fix it?

I have set:

pl.Trainer(
    accelerator='gpu',
    devices=#GPUS,
    strategy='ddp',
    sync_batchnorm=True,
    deterministic=True,
    gradient_clip_val=$CLIP_VALUE
)

What version are you seeing the problem on?

v2.3

How to reproduce the bug

Error messages and logs

# Error messages and logs here please

Environment

Current environment

#- PyTorch Lightning Version (e.g., 2.5.0):
#- PyTorch Version (e.g., 2.5):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):

More info

No response

cc @justusschock @lantiga

[SkafteNicki]

Could you please provide a fully reproducible example for use to debug? The trainer config you have provided looks fine, you could try running with barebone=True which will disable a lot of lightning features to improve runtime. That said in general lightning should not be that much slower than native pytorch.