Configuration for an RTX 3090 #83
-
|
I got a new graphics card (yay!), an RTX 3090. I switched and installed the NVidia drivers. When I try to run my existing decode_env environment, I get the the following warning when importing torch in the DECODE training notebook: Here's what I have in my current decode environment: What should I update to make it work? From a quick googling it looks like I need Cuda 11 instead of 10.1, and a more recent pytorch (1.7.0 instead of 1.6.0)? Would it be simpler to make a new environment so that I don't risk breaking this one too badly (given I could need python 3.8)? I see the latest installation instructions (master as opposed to 0.9.4) have dropped the "cudatoolkit=10.1" command for just "cudatoolkit", so would installation according to the master version of the instructions work for my case? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 13 replies
-
|
Mmmh current install instructions with no version for cudatoolkit will install cudatoolkit 10.2 and pytorch 1.6.0, so I guess it won't work as is in a new environment. |
Beta Was this translation helpful? Give feedback.
-
|
First: I am jealous, I might need to ask my PI for an RTX 3090 in order to be able to fix this 😅 Ehm yes, I would definitely recommend going for a fresh environment. In your case it might be best if you
Let me know if that works |
Beta Was this translation helpful? Give feedback.
-
|
On a regular training (40x40 patch with 25 emitters), I bumped the threads to 8 (I have an 8-core, 16-threads CPU) and batch size to 256 for training (uses ~17 GB of the 24 GB VRAM). Speed is ~1.45 it/s with 39 steps for an epoch, CPU at around 35% and GPU strangely reporting 1% use (from the Windows Task Manager). All in all that gets something like 105-110 epochs/hour. |
Beta Was this translation helpful? Give feedback.
-
|
I tested the workers again, it didn't crash with a broken pipe error but kept printing a warning (always the same, 1 warning per worker so if I put 4 workers I got 4 times the same warning, 2 times per epoch - training and testing) with long pauses between warnings, resulting in much slower training. |
Beta Was this translation helpful? Give feedback.
-
|
Which kind of warning? I don't know how many CPU cores you have, but you could set num_workers to the number of physical cores. |
Beta Was this translation helpful? Give feedback.
-
|
Could you post your param_run.yaml (which can be found in the network output folder) as well as the warning? |
Beta Was this translation helpful? Give feedback.
-
|
Here the warning that appears 2X8 times in an epoch when I set num+workers to 8 (8 times at training, 8 times at testing): Here's the corresponding yaml file: |
Beta Was this translation helpful? Give feedback.
First: I am jealous, I might need to ask my PI for an RTX 3090 in order to be able to fix this 😅
Ehm yes, I would definitely recommend going for a fresh environment. In your case it might be best if you
git checkout v0.9.4conda env create -f environment_cuda.yamlconda activate decode_cudapython setup.py installLet me know if that works