what is the right way to run inference with GPU(cuda) accelaration?

[hunterchenghx]

I use a NGC pytorch docker container with onnxruntime-gpu installed.
My container environment is like: ubuntu18.04, python3.6.10, onnx1.7.0, onnxruntime1.8.1, cuda11.0.3, cudnn8.0.4

my code is like this

import onnxruntime
print("using device: " + onnxruntime.get_device())   #shows GPU here
providers = [
    ('CUDAExecutionProvider', {
        'device_id': 0, 
        'arena_extend_strategy': 'kNextPowerOfTwo',
        'gpu_mem_limit': 24 * 1024 * 1024 * 1024, ## gpu memory in bytes
        'cudnn_conv_algo_search': 'EXHAUSTIVE',
        'do_copy_in_default_stream': True,
    }),
    'CPUExecutionProvider',
    ]
w = "v5_s_1280.onnx"
session = onnxruntime.InferenceSession(w, providers=providers)

img = ######### image loading procedure
pred = torch.tensor(session.run([session.get_outputs()[0].name], {session.get_inputs()[0].name: img}))

#post process

But when I did the inference, only CPU is working. My nvidia-smi didn't show GPU running.
Could anyone help me? Is my code wrong or sth. happened with my onnxruntime installation?

[pranavsharma]

1. Make sure you have only the onnxruntime-gpu pkg installed. If you installed onnxruntime (CPU-only pkg) before, delete both and re-install onnxruntime-gpu pkg.
2. Set the severity level to VERBOSE as shown here https://github.com/microsoft/onnxruntime/blob/master/docs/FAQ.md#how-do-i-change-the-severity-level-of-the-default-logger-to-something-other-than-the-default-warning.
3. Look for the string "Node placements" in the logs. It should tell you which nodes in the graph were assigned to cuda and CPU. Paste the logs here.
4. Can you share the model?

[hunterchenghx]

Thank you so much for the reply. After that, I tried again. The device ID in the provider is different from the IDs from nvidia-smi, so I missed. The GPU was working. Although the speed boost up was not such apparently.
Sorry that I cannot share my model due to some security issue.