Performance difference for Quantized Matmul between v2.6.3 and v3.x

[SriAlavandar]

I am performing benchmarking analysis using benchdnn between OneDNN v2.6.3 and OneDNN v3.10.2. I observed that v2.6.3 performs 10-12% better than v3.10.2 when running with a single thread (OMP_NUM_THREADS=1).
Data type combinations tested: u8:s8:u8 and u8:s8:f32

Here are the cmd I am using for this exp:
v2.6.3:
numactl --physcpubind=0 --interleave=0 tests/benchdnn/benchdnn --matmul --mode=P --cfg=u8s8u8 --bia_dt=f32 --stag=ab --wtag=any --dtag=ab --fix-times-per-prb=10000 --attr-zero-points=src:common:1+dst:common:1 --attr-oscale=per_oc:2.5 --attr-post-ops='eltwise_relu' --batch=input_relu_u8.txt
v3.10.2:
numactl --physcpubind=0 --interleave=0 tests/benchdnn/benchdnn --matmul --mode=P --dt=u8:s8:u8 --bia-dt=f32 --stag=ab --wtag=any --dtag=ab --fix-times-per-prb=10000 --attr-zero-points=src:common:1+dst:common:1 --attr-scales=src:common:1.5+wei:per_oc+dst:common:2.5 --attr-post-ops='eltwise_relu' --batch=input_relu_u8.txt

Please find some sample combinations and their behavior

M	K	N	Dtype	OneDNN v2.6.3	OneDNN v3.10.2	v2.6.3 / v3.10.2
200	13	512	u8:s8:u8	0.011	0.012	0.92
200	512	256	u8:s8:u8	0.054	0.056	0.96
200	256	128	u8:s8:f32	0.014	0.015	0.93
300	13	512	u8:s8:u8	0.016	0.018	0.89
300	512	256	u8:s8:u8	0.081	0.083	0.98
300	256	128	u8:s8:f32	0.021	0.022	0.95
400	13	512	u8:s8:u8	0.021	0.024	0.88

Here are the sample logs of these two experiments for reference:
v2.6.3:
onednn_verbose,exec,cpu,matmul,brg:avx512_core_vnni,undef,src_u8::blocked:ab:f0 wei_s8:p:blocked:BA16a64b4a:f8:zpm2 bia_f32::blocked:ab:f0_mask2 dst_u8::blocked:ab:f0,attr-oscale:2 attr-zero-points:src:0:1+dst:0:1 attr-post-ops:eltwise_relu ,,300x13:13x512:300x512,0.0158691

v3.10.2:
onednn_verbose,v1,primitive,exec,cpu,matmul,brg_matmul:avx512_core_vnni,undef,src:u8::blocked:ab::f0 wei:s8:ap:blocked:BA16a64b4a::f8:zpm2 bia:f32:a:blocked:ab::f0_mask2 dst:u8::blocked:ab::f0,attr-scales:src0:0:f32+dst:0:f32+wei:2:f32 attr-zero-points:src0:0:s32+dst:0:s32 attr-post-ops:eltwise_relu,,300x13:13x512,0.0180664

Questions:
a. Is this expected behavior due to internal changes between v2.6 and v3.x?

[vpirogov]

@SriAlavandar,

It looks like the difference in performance is related to the new quantization scheme introduced in v3.0. You can test this by removing --attr-zero-points and --attr-scales flags. One of the contributing factors is use of per-channel scales in v3.10.2 benchmark (wei:2:f32), while v2.6.3 uses a single per-tensor output scale. This change requires additional computations and additional buffer for scales.

Hope this helps!

[SriAlavandar]

Thanks for response @vpirogov

I understand that v3.x introduces changes to the quantization scheme (e.g., bias quantization is no longer required, 
and quantization parameters are passed separately). However, these additional computations appear to contribute to the observed performance gap.

Question: Is there a plan to address or narrow this gap?

Here are few observations with FP32 and BF16:
Here are FP32 matmul observations between v2.6.3 and v3.11 

M	K	N	Bias	Post op	v2.6.3 (ms)	v3.11(ms)	v2.6.3/v3.11
49152	1024	30522	FALSE	-	148.31	176.57	0.84
49152	1024	512	TRUE	-	2.41	3.00	0.80
49152	4096	1024	TRUE	-	21.15	24.36	0.87
49152	512	1024	TRUE	-	2.48	3.03	0.82
49152	512	50272	TRUE	-	116.14	130.47	0.89
49152	1024	1024	TRUE	gelu_erf	5.12	5.91	0.87
49152	1024	4096	TRUE	gelu_erf	20.39	22.85	0.89
49152	1024	4096	TRUE	relu	19.80	22.69	0.87
49152	1024	1024	TRUE		4.89	5.76	0.85
384	768	3072	TRUE	gelu_erf	0.10	0.25	0.41
384	768	768	TRUE	gelu_erf	0.05	0.05	0.95
384	768	768	TRUE	-	0.04	0.05	0.92
384	3072	768	TRUE	-	0.15	0.21	0.72

I am using Blocked weight Tag for F32

Observations:

• 128 cores: Average regression of ~15%, with certain configurations showing up to ~50% regression.
• 64 cores: The gap narrows significantly — several configurations perform on par between the two versions.

Question: Is this a known scalability limitation in the FP32 kernels?

Here are BF16 matmul observations between v2.6.3 and v3.11

M	K	N	Bias	Post op	v2.6.3 (ms)	v3.11(ms)	v2.6.3/v3.11
49152	1024	30522	FALSE	-	91.07	95.41	0.95
49152	1024	512	TRUE	-	1.59	1.65	0.97
49152	4096	1024	TRUE	-	11.22	11.74	0.96
49152	1024	4096	TRUE	relu	12.10	12.64	0.96
384	768	3072	TRUE	gelu_erf	0.17	0.18	0.95
384	768	768	TRUE	gelu_erf	0.05	0.06	0.84
384	768	30522	FALSE	-	0.81	0.86	0.95

⁠Note: I am using plain weight Tag(ab) for BF16

Observations:

• 128 cores: Average regression of ~5%, with certain configurations showing up to ~15% regression.

Question: Is this an expected behavior for FP32 and BF16 between v2.6.3 and v3.x?

[vpirogov]

I understand that v3.x introduces changes to the quantization scheme (e.g., bias quantization is no longer required,
and quantization parameters are passed separately). However, these additional computations appear to contribute to the observed performance gap.
Question: Is there a plan to address or narrow this gap?

One thing you can do to narrow the gap is avoid using src and dst scales with v3.10. Here's what I see on Intel Xeon Gold 6342 CPU:

# v2.6
$ numactl --physcpubind=3 --interleave=0 ./build-v2.6/tests/benchdnn/benchdnn --matmul --mode=P --cfg=u8s8u8 --bia_dt=f32 --stag=ab --wtag=any --dtag=ab --attr-zero-points=src:common:1+dst:common:1 --attr-oscale=per_oc:2.5 --attr-post-ops='eltwise_relu' 400x13:13x512
Output template: perf,%engine%,%impl%,%name%,%prb%,%Gops%,%Gfreq%,%-time%,%-Gflops%,%0time%,%0Gflops%
perf,cpu,brg:avx512_core_vnni,,--matmul --mode=P --cfg=u8s8u8 --stag=ab --dtag=ab --bia_dt=f32 --attr-oscale=per_oc:2.5 --attr-zero-points=dst:common:1+src:common:1 --attr-post-ops=relu 400x13:13x512,0.0053248,0,0.0354004,150.416,0.0401685,132.562
tests:1 passed:1 skipped:0 mistrusted:0 unimplemented:0 failed:0 listed:0
total perf: min(ms):0.0354004 avg(ms):0.0401685

# v3.10 with src/dst scales
$ numactl --physcpubind=3 --interleave=0 ./build-v3.10/tests/benchdnn/benchdnn --matmul --mode=P --dt=u8:s8:u8 --bia-dt=f32 --stag=ab --wtag=any --dtag=ab --attr-zero-points=src:common:1+dst:common:1 --attr-scales=src:common:1.5+wei:per_oc+dst:common:2.5 --attr-post-ops='eltwise_relu' 400x13:13x512
Output template: perf,%engine%,%impl%,%name%,%prb%,%Gops%,%+ctime%,%-time%,%-Gflops%,%0time%,%0Gflops%
perf,cpu,brg_matmul:avx512_core_vnni,,--mode=P --matmul --dt=u8:s8:u8 --stag=ab --dtag=ab --bia-dt=f32 --attr-scales=src:common:1.5+dst:common:2.5+wei:per_oc --attr-zero-points=src:common:1+dst:common:1 --attr-post-ops=relu 400x13:13x512,0.0053248,2.54565,0.041748,127.546,0.0472628,112.664
============================================================
= Implementation statistics (--summary=no-impl to disable) =
============================================================
| brg_matmul:avx512_core_vnni : 1 (100%)                   |
============================================================
tests:1 passed:1 skipped:0 mistrusted:0 unimplemented:0 invalid_arguments:0 failed:0 listed:0
total perf: min(ms):0.041748 avg(ms):0.0472628
total: 3.01s; create_pd: 0.00s (0%); create_prim: 0.00s (0%); fill: 0.00s (0%); execute: 0.00s (0%);

# v3.10 without src/dst scales
$ numactl --physcpubind=3 --interleave=0 ./build-v3.10/tests/benchdnn/benchdnn --matmul --mode=P --dt=u8:s8:u8 --bia-dt=f32 --stag=ab --wtag=any --dtag=ab --attr-zero-points=src:common:1+dst:common:1 --attr-scales=wei:per_oc --attr-post-ops='eltwise_relu' 400x13:13x512
Output template: perf,%engine%,%impl%,%name%,%prb%,%Gops%,%+ctime%,%-time%,%-Gflops%,%0time%,%0Gflops%
perf,cpu,brg_matmul:avx512_core_vnni,,--mode=P --matmul --dt=u8:s8:u8 --stag=ab --dtag=ab --bia-dt=f32 --attr-scales=wei:per_oc --attr-zero-points=src:common:1+dst:common:1 --attr-post-ops=relu 400x13:13x512,0.0053248,2.05347,0.0380859,139.81,0.0426024,124.988
============================================================
= Implementation statistics (--summary=no-impl to disable) =
============================================================
| brg_matmul:avx512_core_vnni : 1 (100%)                   |
============================================================
tests:1 passed:1 skipped:0 mistrusted:0 unimplemented:0 invalid_arguments:0 failed:0 listed:0
total perf: min(ms):0.0380859 avg(ms):0.0426024
total: 3.01s; create_pd: 0.00s (0%); create_prim: 0.00s (0%); fill: 0.00s (0%); execute: 0.00s (0%);

A few notes on benchmarking:

• --physcpubind=0 may lead to instability in performance results on Linux distributions, I would suggest using other cores
• --fix-times-per-prb=10000 for tiny matmul shapes like in this benchmark may lead to unstable results due to CPU power management features. It would take time to get from low power state affecting the performance results.

So dropping src and dst scales results in significant reduction of the gap. Remaining gap is largely the price to pay for changing quantization from static scales in v2.6 to runtime scales v3.x. There may still be a room for improvement there, but we currently do not have plans to investigate it. Also note, that performance in benchdnn does not necessarily represent performance in AI model execution. With changes like this it's important to evaluate performance at model level.

Question: Is this a known scalability limitation in the FP32 kernels?
Question: Is this an expected behavior for FP32 and BF16 between v2.6.3 and v3.x?

It's hard to tell what's going on here without additional details. In particular scaling experiment can be affected by NUMA architecture of you system. Could you please share:

• Specific benchdnn commands used to collect the data
• Output of lscpu command that shows CPU model and NUMA topology