Cuda EP behavior with pitched data

[admayber]

We recently found some discrepancies in the behavior of our models in the cuda EP, which we identified as being an issue related to the "pitch" (padding) of an input image in cuda memory. Seems that OpenCV, by default, will use pitched memory, whereas by default our traced models were not handling the padding correctly.

I was unable to find any documentation on this, so wanted to get clarification on some points related to this. Does onnx or ORT have the ability to handle inputs with padding? If so, is it possible to achieve this behavior dynamically such that input images could be handled correctly regardless of being padded or not?

[natke]

Hi @admyber, I'm moving this issue into our discussion forum and routing it to the appropriate team members

[admayber]

Hi @natke - we're starting some more investigation into this issue, any chance we could get a comment from the team?

[hariharans29]

Hi @admayber,

Not familiar with the specifics of OpenCV's pitched memory, but let me take a shot at this question:

Does ONNX/ORT support padded images? - Yes.

Is it possible to achieve this behavior dynamically such that input images could be handled correctly regardless of being padded or not? - Yes, to an extent. Image based ONNX ops have an attribute called auto_pad that can be set appropriately to achieve some "dynamic/implicit padding" during the operation itself. Please take a look at - https://github.com/onnx/onnx/blob/main/docs/Operators.md#Conv. But, generally speaking, it is recommended that such implicit padding using auto_pad be avoided (by setting auto_pad to NOTSET) and the input image be explicitly padded by the user (this does mean higher peak memory usage though). But the user needs to figure out the padding they would have to apply to the input image based on the padding used while training the target model.

Hopefully this clarifies. If it doesn't, can you give us a more concrete example of your use-case? If possible, sharing an ONNX model and centering the discussion around that will definitely help.

[natke]

Hi @admayber, does this answer your question?

[admayber]

Thanks @hariharans29 @natke - this is helpful, we are doing some investigation right now and we should be able to provide a specific model and some sample data that triggers the issue soon.

One clarifying question, why is it recommended that auto_pad not be used? Seems like that would be the exact solution for the issue we're encountering.

[hariharans29]

If I recall correctly, there was a time when auto_pad introduced some ambiguity when exporting models from different frameworks and there was some chatter about this in the ONNX community and therefore it was recommended that explicit pads be provided (where it can be and thus reduce the ambiguity around the implicit padding computation that would arise from using the auto_pad attribute). If you observe carefully in some ONNX ops' documentation, the auto_pad attribute is marked deprecated. I believe the ORT should still honor these attributes though and so by all means, try using it if you think that is what you would require.