Inference on slices of input buffer on GPU device

[ktzsh]

I have a yolov4 onnx model and due to 2 dynamic axis - batch and number of boxes I am unable to do batch inference while single example inference works.

Model Input/Output:

input [
  {
    data_type: TYPE_FP32
    dims: [ -1, 512, 512, 3 ]
  }
]
output [
  {
    data_type: TYPE_FP32
    dims: [ -1, -1, 5 ]
  }
]

So is there a way to copy whole input buffer to device, run inference on slices of it in loop and then copy the output so as to maximize the throughput
Eg. Array [40,512,512,3] -> CopyToGPU -> Loop(inference on [1,512,512,3]) -> CopyOutputToCPU

[Manutea]

Hey,

Maybe you can use the binding https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a9a53edebf4ef062a41b0e74f9c6763ec
Copy the data directly on GPU, give the pointer with the offset to your input tensor and run a model with a batch of 1.