FP16 optimization on CPU

[jeyblu]

Are FP16 code optimized with AVX or other vector instructions for CPU? Thanks.

MlasHalfGemmKernel<MLAS_HALF_GEMM_KERNEL_DEFAULT> doesn't seem to have any vector instructions.

[chenfucn]

your understanding is correct. it is not optimized for avx. We have optimized kernel for ARM neon chips and in the process of refining that:
https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/mlas/lib/halfgemm_kernel_neon.cpp

[jeyblu]

Are customers asking for FP16 support on ARM neon chips and not on Intel/AMD chips? Thanks

[deischi]

The OpenVINO execution provider seems to have FP16 support for CPU (CPU_FP16)
https://onnxruntime.ai/docs/execution-providers/OpenVINO-ExecutionProvider.html#summary-of-options

But honestly I do not know details about that.