Custom distillation model using DinoVol3 does not achieve the required accuracy after int8 quantization

[potato-creator]

I customized my own model according to the tutorial and trained it using distillation with dinov3/vits16 as the teacher. However, after exporting the model as ONNX and performing int8 quantization, the model accuracy dropped significantly.
This issue does not occur with models that have not undergone distillation.

[stegmuel]

Hi,
Would it be possible to share more information on the model you distilled into?

[potato-creator]

I created a custom CNN-Transformer model, which is a 4-stage architecture similar to PVT.
Perhaps you could try exporting a distilled model and performing int8 quantization to see if this issue arises.

[stegmuel]

Thanks for the additional information.
ONNX and TensorRT is something we are currently looking into.
By default, distillation uses bf16, you might want to try training with fp32 precision.
Otherwise my recommendation would be to try adding fake quantization noise to your model during training, but we did not try that yet.