moving to lightning makes my code slower :(

[glupol]

Hi, I have moved my 3D u-net like model to the lightning.
I have a single GPU (i.e. K40) and was hoping for a better utilisation of it.
It looks like my code does not utilizes it’s full potential.
However, after running trainer.fit() I can see 2x slow down 😦
The whole framework looks quite complex and I wanted to ask for some tips on where should I look to get it tuned up?
Are there some unnecessary data transmissions or logging etc.
Thank you in advance for all the help.
G.

[tchaton]

Dear @glupol,

Are you using multiple workers ? You can use Trainer(profiler=simple|advanced|pytorch) to get a better insight where the time is lost.
If you could share them with us, it will help us to give you insights on how to resolve this problem.

Best,
T.C

[glupol]

Hi thank you.
I do use multiple workers in my code. My issue is that the pytorch-lightning is slower than the simple code I wrote. I was hopping for the lightning effect and got the slow down.
In the long run, I think I will have to write my own parallelization data-loader.
I am curious what parameters I need to set up to get at least the same performance in the pytorch-lightning as in my own code.
(I will update it with my code).

[carmocca]

Yeah, having both of your non-ported and ported code available would be helpful to see where are the slow-downs