Load N model in multi GPUs

[eric781ctf]

Reference from the comment in #815.

I attempted to load two models on two RTX 4090 GPUs, but after trying, I realized that the time it took was longer compared to using only one model and performing it on a single GPU.

Here is part of my code.

import threading
cuda_list=[1,0]
data={}
class MyThread(threading.Thread):
   def __init__(self, num):
       threading.Thread.__init__(self)
       self.num = num
            
   def run(self):
       cuda=cuda_list.pop(0)
       model = whisper.load_model("large-v2",device='cuda:{}'.format(cuda))
       handle = pynvml.nvmlDeviceGetHandleByIndex(cuda)
       while len(file_list_1)>0:
           file=file_list_1.pop(0)
           
           start_time1=time.time()
           result=model.transcribe(file)
           
           data[file]=result
           print('cuda:',cuda,file,start_time1,time.time(),time.time()-start_time)

I'm not sure why it will be slower than it should be.