What is the correct way to create an OrtValue from a pytorch tensor on CUDA?

[mattroos]

In python, if I have a numpy array, x, and want to create an OrtValue on a CUDA device I do the following:
ortvalue = onnxruntime.OrtValue.ortvalue_from_numpy(x, 'cuda', 0)

But if I have a torch tensor, y, that is already in CUDA/GPU memory, it seems I have to copy it to CPU memory before creating the OrtValue:
onnxruntime.OrtValue.ortvalue_from_numpy(y.data.cpu().numpy(), 'cuda', 0)

Is there a way to create the OrtValue directly from the the torch tensor, such that the tensor doesn't have to be copied to CPU memory and then back to GPU memory?

[hariharans29]

Hmm - I don't think we currently have a way to do this. What you want is to be able to provide a data pointer and metadata for that dat pointer: type, shape, and device information.

May I ask what your scenario is ? Do you want to feed an output from Torch as input to an ONNX graph via ORT's python interface and thereby looking to reduce the round-trip latency for the data ?

[mattroos]

Do you want to feed an output from Torch as input to an ONNX graph via ORT's python interface and thereby looking to reduce the round-trip latency for the data?

Yes, that's correct. Down the road I may be able to rewrite a lot of my code to avoid using Torch, and just use numpy and the ORT, but for now I'm sticking with the torch code. I'm not sure what the latency savings would be by going directly from the torch cuda tensor to the OrtValue, but certainly would want to do that if it were an option.

[devymex]

I want to convert a torch tensor (on GPU) to OrtValues directly without via NumPy.

For my project retrieving data from a GPU decoder, I need a way to prevent the CPU-GPU copy.

[hariharans29]

@mattroos and @devymex - This is definitely a useful feature, please feel free to open a GitHub issue for this request

[jeshels]

Opened #10217

[nguyenvanhoang7398]

On the other hand, is there a way to convert OrtValue to Pytorch tensor on the same device? I need to run some tensor operations provided by PyTorch before passing the OrtValue to the next module.

[hariharans29]

@nguyenvanhoang7398 : The OrtValue instance does provide access to its underlying data buffer:

onnxruntime/onnxruntime/python/onnxruntime_inference_collection.py

Line 560 in 69ab467

return self._ortvalue.data_ptr()

. Is it possible to wrap this buffer into a PyTorch tensor ?

[sradmard]

Has there been any progress made on this issue yet?

[sradmard]

@hariharans29 I was wondering whether there has been any progress on this issue or not. I would really appreciate it.

[farzanehnakhaee70]

Hi all,
I would also appreciated if there is any solution for this problem. I exactly have the same issue.

[jeshels]

#10217