~0.5% performance boost with mitigations=off?

[magikRUKKOLA]

Just tried to put mitigations=off into the grub and compared the performance.

With mitigations:

  Retbleed:                  Mitigation; untrained return thunk; SMT enabled with STIBP protection
  Spec rstack overflow:      Mitigation; Safe RET
  Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:                Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected

sweep-bench:

....................................................................................................
llama_new_context_with_model: n_ctx      = 131072
llama_new_context_with_model: n_batch    = 4096
llama_new_context_with_model: n_ubatch   = 2048
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: mla_attn   = 3
llama_new_context_with_model: attn_max_b = 512
llama_new_context_with_model: fused_moe  = 1
llama_new_context_with_model: ser        = -1, 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init:      CUDA0 KV buffer size =  2218.51 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =  2448.02 MiB
llama_new_context_with_model: KV self size  = 4666.50 MiB, c^KV (q8_0): 4666.50 MiB, kv^T: not used
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.49 MiB
llama_new_context_with_model: pipeline parallelism enabled (n_copies=1)
llama_new_context_with_model:      CUDA0 compute buffer size = 12564.01 MiB
llama_new_context_with_model:      CUDA1 compute buffer size = 11624.02 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =  1080.02 MiB
llama_new_context_with_model: graph nodes  = 24379
llama_new_context_with_model: graph splits = 166

main: n_kv_max = 131072, n_batch = 4096, n_ubatch = 2048, flash_attn = 1, n_gpu_layers = 99, n_threads = 64, n_threads_batch = 64

|    PP |     TG |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
|  2048 |    512 |      0 |   19.361 |   105.78 |   51.550 |     9.93 |
|  2048 |    512 |   2048 |   19.701 |   103.95 |   52.262 |     9.80 |

without mitigations:

  Retbleed:                  Vulnerable
  Spec rstack overflow:      Vulnerable
  Spec store bypass:         Vulnerable
  Spectre v1:                Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
  Spectre v2:                Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Not affected; BHI: Not affected

|    PP |     TG |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
|  2048 |    512 |      0 |   19.256 |   106.36 |   51.501 |     9.94 |
|  2048 |    512 |   2048 |   19.597 |   104.50 |   51.989 |     9.85 |

as can be seen, there is a slight performance boost in prefill:

106.36 vs 105.78
104.5 vs 103.95

Did anyone observed similar results?

  Model name:                AMD Ryzen Threadripper PRO 3995WX 64-Cores

microcode       : 0x830107d

[magikRUKKOLA]

What else can be done to improve the performance? Any ideas?