High RAM usage during test step

[mshannon-sil]

On multiple clearml experiments over the past week, we've noticed that if experiments are run using --preprocess and --test without the --train step, RAM utilization spikes to over 200GB, causing experiments to crash. We need to investigate what in particular during the test step is causing this drastic increase in RAM usage.

[davidbaines]

This, or something similar, might occur when using --preproces and --translate without the --train step. That causes some books to fail to translate, but doesn't affect all books. Here's an example to check.

[mshannon-sil]

Hi @davidbaines, I took a look at your example. That seems to be a separate issue, where the GPU ran out of VRAM, rather than the system RAM like in this issue. I noticed that in the experiment you were using the 3.3B NLLB model. I know that doesn't fit during training on a single 40GB GPU, and it may be the case it's right on the border of what will fit during inference, which would explain why some books are translating correctly and others fail with the VRAM issue. If you need to run inference with that model, consider using the cheetah_47gb or cheetah_94gb queues that have more VRAM.