GPU usage maxed at 10%

[cowwoc]

I was never able to get GPU usage to 10% for my own application so I decided to try BoringModel and ran into the same problem.

1. Take https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/bug_report/bug_report_model.py
2. Add gpus=-1 to Trainer configuration
3. Increase max_epochs to 10_000 in Trainer configuration

Environment:

* CUDA:
        - GPU:
                - NVIDIA GeForce RTX 3080
        - available:         True
        - version:           11.3
* Packages:
        - numpy:             1.21.4
        - pyTorch_debug:     False
        - pyTorch_version:   1.10.0+cu113
        - pytorch-lightning: 1.5.8
        - tqdm:              4.62.3
* System:
        - OS:                Windows
        - architecture:
                - 64bit
                - WindowsPE
        - processor:         AMD64 Family 23 Model 113 Stepping 0, AuthenticAMD
        - python:            3.9.7
        - version:           10.0.19044

Am I doing something wrong, or is there some sort of bug in pytorch or pytorch lightning triggering this?

[rohitgr7]

boring model is not very complex and won't lead to high GPU usage. Maybe try increasing the batch_size?

[rohitgr7]

maybe try decreasing the prefetch_factor:
https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader

[cowwoc]

maybe try decreasing the prefetch_factor: https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader

The default value is 2. You can't really decrease it much more than this :) Did you mean something else?

Also, why do you think that decreasing prefetch_factor would help? Did you mean increasing it instead?

[rohitgr7]

yeah sorry! I meant try increasing it.

but just read your full comment. If dataloader isn't able to process x data on time, wondering whether expecting it to processes Nx would even work 🤔

[rohitgr7]

wait! your data is already on the device?

self.data = torch.cuda.FloatTensor(length, size).normal_()

[cowwoc]

wait! your data is already on the device?

self.data = torch.cuda.FloatTensor(length, size).normal_()

Yes. I figured this would rule out one variable.

[cowwoc]

I think I kept on playing with this and got GPU usage up to ~50%. But that's as high as I could go. As far as I can tell, GPU usage can only increase if we're computing more on the GPU before attempting to load more data, but I could not load more data onto the GPU because it would surpass the onboard memory (10GB).

You should be able to run the above testcase. Is there anything else I can do to improve performance without using more GPU ram?