[clang] Optimisation regression in trunk for x86-64 AVX2 shuffle

[desal]

Looks like there's an optimisation regression between clang 21.1.0 and trunk in handling AVX2 shuffle operations. It seems to only manifest when shuffling the result of a comparison, (e.g. _mm256_cmpeq_epi32). clang now prefers to do the shuffling in packed 128 bit xmm registers and then reconstruct the 256 bit ymm output.

Test case (Compiler Explorer: https://gcc.godbolt.org/z/xGGs8hW5P)
Compiled with -O2 -mavx2

#include <immintrin.h>

__m256i foo(__m256i a, __m256i b) {
    __m256i x = _mm256_cmpeq_epi32(a, b);
    return _mm256_shuffle_epi32(x, 0b11101111); 
}

Clang trunk clang version 22.0.0git (https://github.com/llvm/llvm-project.git 03e66aeb96928592ee6cd51913bf72a6e21066fc)

foo(long long vector[4], long long vector[4]):
  vpcmpeqd ymm0, ymm0, ymm1
  vextracti128 xmm1, ymm0, 1
  vpackssdw xmm0, xmm0, xmm1
  vpshuflw xmm0, xmm0, 239
  vpshufhw xmm0, xmm0, 239
  vpmovsxwd ymm0, xmm0
  ret

Clang 21.1.0:

foo(long long vector[4], long long vector[4]):
  vpcmpeqd ymm0, ymm0, ymm1
  vpshufd ymm0, ymm0, 239
  ret

[llvmbot]

@llvm/issue-subscribers-backend-x86

Author: None (desal)

Looks like there's an optimisation regression between clang 21.1.0 and trunk in handling AVX2 shuffle operations. It seems to only manifest when shuffling the result of a comparison, (e.g. `_mm256_cmpeq_epi32`). clang now prefers to do the shuffling in packed 128bit xmm registers and then reconstruct the 128bit ymm output.

Test case (Compiler Explorer: https://gcc.godbolt.org/z/xGGs8hW5P)
Compiled with -O2 -mavx2

#include <immintrin.h>

__m256i foo(__m256i a, __m256i b) {
    __m256i x = _mm256_cmpeq_epi32(a, b);
    return _mm256_shuffle_epi32(x, 0b11101111); 
}

Clang trunk clang version 22.0.0git (https://github.com/llvm/llvm-project.git 03e66aeb96928592ee6cd51913bf72a6e21066fc)

foo(long long vector[4], long long vector[4]):
  vpcmpeqd ymm0, ymm0, ymm1
  vextracti128 xmm1, ymm0, 1
  vpackssdw xmm0, xmm0, xmm1
  vpshuflw xmm0, xmm0, 239
  vpshufhw xmm0, xmm0, 239
  vpmovsxwd ymm0, xmm0
  ret

Clang 21.1.0:

foo(long long vector[4], long long vector[4]):
  vpcmpeqd ymm0, ymm0, ymm1
  vpshufd ymm0, ymm0, 239
  ret

[RKSimon]

VectorCombine is folding from:

define <4 x i64> @foo(<4 x i64> noundef %a, <4 x i64> noundef %b) local_unnamed_addr {
entry:
  %0 = bitcast <4 x i64> %a to <8 x i32>
  %1 = bitcast <4 x i64> %b to <8 x i32>
  %cmp.i = icmp eq <8 x i32> %0, %1
  %sext.i = sext <8 x i1> %cmp.i to <8 x i32>
  %permil = shufflevector <8 x i32> %sext.i, <8 x i32> poison, <8 x i32> <i32 3, i32 3, i32 2, i32 3, i32 7, i32 7, i32 6, i32 7>
  %2 = bitcast <8 x i32> %permil to <4 x i64>
  ret <4 x i64> %2
}

to

define <4 x i64> @foo(<4 x i64> noundef %a, <4 x i64> noundef %b) local_unnamed_addr {
entry:
  %0 = bitcast <4 x i64> %a to <8 x i32>
  %1 = bitcast <4 x i64> %b to <8 x i32>
  %cmp.i = icmp eq <8 x i32> %0, %1
  %2 = shufflevector <8 x i1> %cmp.i, <8 x i1> poison, <8 x i32> <i32 3, i32 3, i32 2, i32 3, i32 7, i32 7, i32 6, i32 7>
  %permil = sext <8 x i1> %2 to <8 x i32>
  %3 = bitcast <8 x i32> %permil to <4 x i64>
  ret <4 x i64> %3
}

So we're probably looking at a cost table issue - either not recognising that the original sext is free or that the repeated, in-lane <8 x i32> shuffle is very cheap