Why does onnx model give floating point exception on GPU and not on CPU?

[Avenge-PRC777]

I have an onnx model created out of a pytorch based model. When I run it on a black box system using:

output = onnxruntime.InferenceSession.run() #method to indicate what I use

I create InferenceSession using CUDAExecutionProvider

I am seeing the following results:

If the black box system is CPU based (32GB memory), the code works fine
If the black box system is GPU based (V100/T4; 16GB memory), the code fails at certain inputs with floating point exception at the above line
What is even more curious to me is if I put a high level try except block, it does not capture the floating point exception and the code fails.

I use the following code to convert model to onnx:

torch.onnx.export(model, args = (dummy_input,), f = f"model_v1p1.onnx", export_params=True, opset_version=13, do_constant_folding=True, input_names = ["input_ids", "attention_mask","task"], output_names = ['output'], dynamic_axes={'input_ids' : {0 : 'batch_size'},'attention_mask':{0:'batch_size'}, 'output' : {0 : 'batch_size'}})
I cant reproduce the error with inputs for black box on a local V100 based (32GB memory) machine. I use onnxruntime-gpu 1.14.1

How can I debug this or handle the FPE error?

[pranavsharma]

Please file this as an issue and follow the issue template when doing so.