Error when quantizing onnx model

[de1star]

Hi,
This error occurred when I tried to quantize my onnx model.

Traceback (most recent call last):
  File "quant.py", line 4, in <module>
    quantize(
  File "/usr/local/lib/python3.8/dist-packages/modelopt/onnx/quantization/quantize.py", line 207, in quantize
    onnx_model = quantize_func(
  File "/usr/local/lib/python3.8/dist-packages/modelopt/onnx/quantization/int8.py", line 186, in quantize
    quantize_static(
  File "/usr/local/lib/python3.8/dist-packages/onnxruntime/quantization/quantize.py", line 513, in quantize_static
    calibrator.collect_data(calibration_data_reader)
  File "/usr/local/lib/python3.8/dist-packages/modelopt/onnx/quantization/ort_patching.py", line 271, in _collect_data_histogram_calibrator
    calibrator.intermediate_outputs.append(calibrator.infer_session.run(None, inputs))
  File "/usr/local/lib/python3.8/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 220, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : CopyTensorAsync is not implemented

I have installed onnxruntime-gpu for cuda 12.x by https://onnxruntime.ai/docs/install/.
Could you help with that?

[i-riyad]

Please share the input ONNX model and the command to reproduce the error.

[DefTruth]

same error, any one can help me? microsoft/onnxruntime#21690

[DefTruth]

@riyadshairi979 some warning

2024-08-09 18:31:40.646326213 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 680 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message

[DefTruth]

2024-08-09 10:50:50.360780: W external/xla/xla/service/gpu/nvptx_compiler.cc:718] The NVIDIA driver's CUDA version is 12.4 which is older than the ptxas CUDA version (12.5.40). Because the driver is older than the ptxas version, XLA is disabling parallel compilation, which may slow down compilation. You should update your NVIDIA driver or use the NVIDIA-provided CUDA forward compatibility packages.
INFO:root:Model vit_mlp_encoder/vit_mlp.onnx with opset_version 17 is loaded.
INFO:root:Quantization Mode: fp8
INFO:root:Quantizable op types in the model: ['MatMul', 'Conv']
INFO:root:Found unpaddable conv for FP8: /vision_model/embeddings/patch_embedding/Conv
INFO:root:Total number of nodes: 7544
INFO:root:Skipped node count: 1
WARNING:root:Please consider to run pre-processing before quantization. Refer to example: https://github.com/microsoft/onnxruntime-inference-examples/blob/main/quantization/image_classification/cpu/ReadMe.md

[DefTruth]

using modelopt_examples:latest

[i-riyad]

What is your onnxruntime version? Try using this docker.

[Roxbili]

In my case, running the pre-processing step as described in the following document resolved the error.

WARNING:root:Please consider to run pre-processing before quantization. Refer to example: https://github.com/microsoft/onnxruntime-inference-examples/blob/main/quantization/image_classification/cpu/ReadMe.md

[i-riyad]

We’ve re-encountered the issue and identified that mixed precision ONNX models (e.g., FP32 + FP16 ONNX exported with torch.autocast enabled) fail when multiple execution providers are passed to the ONNX Runtime inference session. ModelOpt uses autocast by default during INT8/FP8 ONNX export from PyTorch models.

To avoid the CopyTensorAsync issue, users should specify a single execution provider when calling the quantization API—for example: calibration_eps = ["cpu"] or calibration_eps = ["cuda:0"].