Poor vector code generation for blend operation

[binji]

Bugzilla Link	50305
Version	trunk
OS	Linux
CC	@RKSimon,@zygoloid,@tlively

Extended Description

See https://godbolt.org/z/Ms9E4nPhM

Given the following code:

typedef uint8_t u8;
typedef uint16_t u16;
typedef u8 u8x16 attribute((vector_size(16)));
typedef u16 u16x8 attribute((vector_size(16)));

typedef struct {
u8x16 counter, shift;
} A;

void bad(A* a) {
u8x16 active = a->counter == 0;
a->counter -= 1 & ~active;
a->shift = ((a->shift << 1) & active) | (a->shift & ~active);
}

Clang seems to prefer to generate a variable shift in the LLVM IR (see %11), which then cannot be lowered efficiently in x86 SSE3/Wasm:

define dso_local void @​_Z3badP1A(%struct.A* nocapture %0) local_unnamed_addr #​0 !dbg !​267 {
call void @​llvm.dbg.value(metadata %struct.A* %0, metadata !​285, metadata !DIExpression()), !dbg !​287
%2 = getelementptr inbounds %struct.A, %struct.A* %0, i64 0, i32 0, !dbg !​288
%3 = load <16 x i8>, <16 x i8>* %2, align 16, !dbg !​288, !tbaa !​289
%4 = icmp ne <16 x i8> %3, zeroinitializer, !dbg !​292
call void @​llvm.dbg.value(metadata <16 x i8> undef, metadata !​286, metadata !DIExpression()), !dbg !​287
%5 = sext <16 x i1> %4 to <16 x i8>, !dbg !​293
%6 = add <16 x i8> %3, %5, !dbg !​294
store <16 x i8> %6, <16 x i8>* %2, align 16, !dbg !​294, !tbaa !​289
%7 = getelementptr inbounds %struct.A, %struct.A* %0, i64 0, i32 1, !dbg !​295
%8 = load <16 x i8>, <16 x i8>* %7, align 16, !dbg !​295, !tbaa !​289
%9 = xor <16 x i1> %4, <i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true, i1 true>, !dbg !​296
%10 = zext <16 x i1> %9 to <16 x i8>, !dbg !​296
%11 = shl <16 x i8> %8, %10, !dbg !​296
store <16 x i8> %11, <16 x i8>* %7, align 16, !dbg !​297, !tbaa !​289
ret void, !dbg !​298
}

Using the platform-specific vector intrinsics seems to avoid this issue.

[RCoder01]

I'm not entirely sure if the following is the same issue, but it also misses a blend optimization: https://godbolt.org/z/sarboz6Ef

const N: usize = 4;

// if a[i] == b[i], it doesn't matter which we select so mask is irrelevant
pub fn blend_good(a: [i32; N], b: [i32; N], mask: [u32; N]) -> [i32; N] {
    array::from_fn(|i| if a[i] == b[i] || mask[i] == 0 {a[i]} else {b[i]})
}

pub fn blend_bad(a: [i32; N], b: [i32; N], mask: [u32; N]) -> [i32; N] {
    array::from_fn(|i| if /*a[i] == b[i] ||*/ mask[i] == 0 {a[i]} else {b[i]})
}

Playing with this code a bit, the difference I've observed is whether the condition is related to one of the data being blended. Looking at the opt pipeline report, the main difference seems to be in SLPVectorizerPass.

Update: I was able to get the version without the redundant check optimized the way I expected with this strange workaround: https://godbolt.org/z/sa947xeMo

use std::array;

const N: usize = 4;

pub fn blend_1(a: [i32; N], b: [i32; N], mask: [u32; N]) -> [i32; N] {
    array::from_fn(|i| if a[i] == b[i] || mask[i] == 0 {a[i]} else {b[i]})
}

pub fn blend_2(a: [i32; N], b: [i32; N], mask: [u32; N]) -> [i32; N] {
    array::from_fn(|i| {let (ai, bi) = load(a, b, i); if mask[i] == 0 {ai} else {bi}})
}

// inlining this function "by hand" does not lead to the code being optimized
#[inline(always)] // but even marking it #[inline(always)] leads to it being optimized
fn load(a: [i32; N], b: [i32; N], i: usize) -> (i32, i32) {
    (a[i], b[i])
}

The difference seems to be the IR before the SLPVectorizerPass runs. In the case of blend_bad, the IR selects between the address of a[i] and b[i] based on mask[i], then loads the address it selected. For blend_2, it loads both a[i] and b[i], then selects the value based on mask[i]. The address loads are not able to be combined into one vector instruction, whereas the value select is.

Not sure how this could be update in LLVM to find this kind of issue, but leaving this comment up in case anyone stumbles upon this.