What is optimal way to load model multiple time?

[prateek9623]

I have a use case where I want to load the model 8 times(4 GPUs, 2 streams). What would be the optimal way to load the model. Until now I was using CUDNN based custom framework, where I can create graphs and load weights in the CPU initially and when required load weights in GPU. Is there anything similar in ORT? This is for inference only, for data size around [20000, 512, 512, 3].

[wschin]

It's unclear to me the metric you want to optimize toward optimal. If you have 4 GPUs, I'd just launch ORT with 4 GPUs and it will load the model and optimize it for you. Assume you want to run training with data parallel. The duplication of model across machines are necessary (some techniques for dynamically moving tensors from and to CPU memory can be found here.

[prateek9623]

This is for inference only and in C++. In my current use case, we have multiple models which need to be used one by one, and the data size is around [20000, 512, 512, 3]. And there are some preprocessing and postprocessing steps that are running on the CPU. So, to get full occupancy of GPU, I am using multiple inference instances with different streams and CPU thread to overlap CPU operation with GPU operations. And for this instead of creating graph multiple time in CPU I create it once and load it multiple time in GPU, but I don't know how to do in ORT.