How to Clear CUDA cache for loaded models?

[atnjqt]

Is there a convenient way to clear CUDA memory when you load a model?

I can run it once, but if I rerun the command it seemingly doesn't replace the model in memory it tries to add on..

import whisper
model_size = "large-v2" # options are "tiny", "base", "small", "medium", "large-v1", "large-v2", "large"

model = whisper.load_model(model_size)

• if I rerun, it throws the following:
  OutOfMemoryError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 14.62 GiB total capacity; 13.51 GiB already allocated; 19.94 MiB free; 14.02 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

My thought was to try and empty cache but that doesn't seem to make a difference?

import torch
torch.cuda.empty_cache()

torch.cuda.mem_get_info()

• the mem_get_info shows memory is not cleared...

[baibhavKumar1]

Hey, have you tried rerunning the code, after emptying the cache? Ignoring the mem_get_info()

[atnjqt]

Thanks @Wholesomebruh, yes that is precisely my confusion is that I only get that error when I rerun the model_load.. as if it's not overwriting in CUDA memory or clearing cache, it's strange!

[baibhavKumar1]

Ok , Where are you running this model, if on your machine what are the hardware, if somewhere else, how to do it?

[phineas-pta]

why do you want to reload the model?

[atnjqt]

OH. That's a great point actually maybe that isn't something I should be doing 😅😅

[phineas-pta]

well it depends on you want, but in most cases you don't need to reload

[atnjqt]

Oh wait, of course -- the reason was that I loaded model_size=small for testing, and then wanted to pivot to model_size=large-v2 for scaling up and comparing... is there a recommendation on changing model size and reloading, other than terminating the environment (running on AWS SageMaker with accelerated compute that has GPUs)

[phineas-pta]

in that case, follow #1313

model = whisper.load_model("small")

del model
torch.cuda.empty_cache()
gc.collect()

model = whisper.load_model("large-v2")

[atnjqt]

maybe I'm just unclear on how much CUDA memory is really needed for the large + models.. I see the Required VRAM referenced here but no clear mention of CUDA https://github.com/openai/whisper#available-models-and-languages

[phineas-pta]

VRAM is the correct technical term, you may call it "CUDA memory"

[atnjqt]

Right yes, Video RAM -- duh! Thanks for explaining all of this relatively self-explanatory stuff. I was honestly just making a mistake of trying to reload the model size without deleting the model.

[atnjqt]

And running on AWS SageMaker for pay-as-you-go EC2 with GPUs is fantastic!

[atnjqt]

And running on AWS SageMaker for pay-as-you-go EC2 with GPUs is fantastic!