Skip to content

[IR] Add new CreateVectorInterleave interface #150931

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
david-arm merged 3 commits into from
Jul 29, 2025
Merged

[IR] Add new CreateVectorInterleave interface #150931

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5cbdfe8
[IR] Add new CreateVectorInterleave interface and constant fold
david-arm Jul 28, 2025
b2091fb
Remove constant folds
david-arm Jul 28, 2025
2b20cb8
Merge branch 'main' into interleave_helper
david-arm Jul 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions llvm/include/llvm/Analysis/InstSimplifyFolder.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,10 @@ class LLVM_ABI InstSimplifyFolder final : public IRBuilderFolder {
dyn_cast_if_present<CallBase>(FMFSource));
}

Value *FoldVectorInterleave(ArrayRef<Value *> Ops) const override {
return ConstFolder.FoldVectorInterleave(Ops);
}

//===--------------------------------------------------------------------===//
// Cast/Conversion Operators
//===--------------------------------------------------------------------===//
Expand Down
18 changes: 18 additions & 0 deletions llvm/include/llvm/Analysis/TargetFolder.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,24 @@ class LLVM_ABI TargetFolder final : public IRBuilderFolder {
return nullptr;
}

Value *FoldVectorInterleave(ArrayRef<Value *> Ops) const override {
Copy link
Contributor

@nikic nikic Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code should not be in IRBuilder folders, but rather part of the generic intrinsic ConstantFolding support.

Copy link
Contributor Author

@david-arm david-arm Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand. Do you mean that I shouldn't be changing any of the folders whatsoever, or that I should be doing something like this in TargetFolder:

  Value *FoldBinaryIntrinsic(Intrinsic::ID ID, Value *LHS, Value *RHS, Type *Ty,
                             Instruction *FMFSource) const override {
    auto *C1 = dyn_cast<Constant>(LHS);
    auto *C2 = dyn_cast<Constant>(RHS);
    if (C1 && C2)
      return ConstantFoldBinaryIntrinsic(ID, C1, C2, Ty, FMFSource);
    return nullptr;
  }

I thought the standard practice in IRBuilder was to apply folds using the folder, i.e. like we do here:

Value *IRBuilderBase::CreateBinaryIntrinsic(Intrinsic::ID ID, Value *LHS,
                                            Value *RHS, FMFSource FMFSource,
                                            const Twine &Name) {
  Module *M = BB->getModule();
  Function *Fn = Intrinsic::getOrInsertDeclaration(M, ID, {LHS->getType()});
  if (Value *V = Folder.FoldBinaryIntrinsic(ID, LHS, RHS, Fn->getReturnType(),
                                            /*FMFSource=*/nullptr))

Copy link
Contributor Author

@david-arm david-arm Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't use the pre-existing FoldBinaryIntrinsic or FoldUnaryIntrinsic functions either because the number of operands is variable, although I'm happy to add a FoldUnaryIntrinsic.

Copy link
Contributor

@nikic nikic Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You probably want to replace FoldBinaryIntrinsic with FoldIntrinsic -- we shouldn't specialize to binary. This is going to need a small bit of refactoring in ConstantFolding to expose a ConstantFoldIntrinsic function instead of ConstantFoldBinaryIntrinsic.

Copy link
Contributor Author

@david-arm david-arm Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK thanks for the pointer, I'll take a look!

Copy link
Contributor Author

@david-arm david-arm Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The places where I'd like to see these constant folds happening are where we're using a default version of IRBuilder, which uses ConstantFolder. However, the ConstantFolder.h version of FoldBinaryIntrinsic says the caller should be using TargetFolder or InstSimplifyFolder instead. So I could change FoldBinaryIntrinsic in llvm/Analysis/ConstantFolding.h (used by TargetFolder.h), but then I wouldn't see any improvements in generated code for IRBuilder instances using the default version.

To be honest, I'm not really sure the best way to proceed now. Would it be acceptable to add a variant of FoldIntrinsic to ConstantFolder.h as well, or is the preferred solution to just change all the IRBuilder instances I care about to use the TargetFolder instead?

Copy link
Contributor

@nikic nikic Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using TargetFolder is generally preferred. (The whole TargetFolder vs ConstantFolder is an annoying legacy split, that's unfortunately not entirely straightforward to remove.)

Copy link
Contributor

@lukel97 lukel97 Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we need a llvm::ConstantFoldBinaryIntrinsic in ConstantFolding.cpp, and call that from both ConstantFolder.h and InstructionSimplify.cpp's ConstantFoldInstOperandsImpl? That would bring it inline with the other the other instruction types.

And we probably need to rename that to just ConstantFoldIntrinsic?

Copy link
Contributor Author

@david-arm david-arm Jul 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we can do that because llvm/include/llvm/IR/ConstantFolder.h only includes llvm/IR/ConstantFold.h, whereas I think what you're talking about is trying to reuse an intrinsic folding function in the Analysis directory.

I'll abandon the constant folds for now, because if I only add support in the TargetFolder the only way I could test it is via unit tests, since both the loop vectoriser and the complex interleaving pass use the default IRBuilder, which uses ConstantFolder. I care more about adding the CreateVectorInterleave interface - I just saw an opportunity to tidy up the IR but it turns out to be far too complicated to do all in one patch.

// Check to see if all operands are the same.
for (unsigned I = 1; I < Ops.size(); I++) {
if (Ops[I] != Ops[0])
return nullptr;
}

// Is this just a large splat?
if (auto *C = dyn_cast<Constant>(Ops[0])) {
if (auto *V = C->getSplatValue()) {
auto *SubvecTy = cast<VectorType>(Ops[0]->getType());
return ConstantVector::getSplat(
SubvecTy->getElementCount() * Ops.size(), V);
}
}
return nullptr;
}

Value *FoldBinaryIntrinsic(Intrinsic::ID ID, Value *LHS, Value *RHS, Type *Ty,
Instruction *FMFSource) const override {
auto *C1 = dyn_cast<Constant>(LHS);
Expand Down
6 changes: 0 additions & 6 deletions llvm/include/llvm/Analysis/VectorUtils.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,12 +177,6 @@ LLVM_ABI bool isVectorIntrinsicWithStructReturnOverloadAtField(
LLVM_ABI Intrinsic::ID
getVectorIntrinsicIDForCall(const CallInst *CI, const TargetLibraryInfo *TLI);

/// Returns the corresponding llvm.vector.interleaveN intrinsic for factor N.
LLVM_ABI Intrinsic::ID getInterleaveIntrinsicID(unsigned Factor);

/// Returns the corresponding llvm.vector.deinterleaveN intrinsic for factor N.
LLVM_ABI Intrinsic::ID getDeinterleaveIntrinsicID(unsigned Factor);

/// Returns the corresponding factor of llvm.vector.interleaveN intrinsics.
LLVM_ABI unsigned getInterleaveIntrinsicFactor(Intrinsic::ID ID);

Expand Down
18 changes: 18 additions & 0 deletions llvm/include/llvm/IR/ConstantFolder.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,24 @@ class LLVM_ABI ConstantFolder final : public IRBuilderFolder {
return nullptr;
}

Value *FoldVectorInterleave(ArrayRef<Value *> Ops) const override {
// Check to see if all operands are the same.
for (unsigned I = 1; I < Ops.size(); I++) {
if (Ops[I] != Ops[0])
return nullptr;
}

// Is this just a large splat?
if (auto *C = dyn_cast<Constant>(Ops[0])) {
if (auto *V = C->getSplatValue()) {
auto *SubvecTy = cast<VectorType>(Ops[0]->getType());
return ConstantVector::getSplat(
SubvecTy->getElementCount() * Ops.size(), V);
}
}
return nullptr;
}

Value *FoldBinaryIntrinsic(Intrinsic::ID ID, Value *LHS, Value *RHS, Type *Ty,
Instruction *FMFSource) const override {
// Use TargetFolder or InstSimplifyFolder instead.
Expand Down
2 changes: 2 additions & 0 deletions llvm/include/llvm/IR/IRBuilder.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2614,6 +2614,8 @@ class IRBuilderBase {
return CreateShuffleVector(V, PoisonValue::get(V->getType()), Mask, Name);
}

Value *CreateVectorInterleave(ArrayRef<Value *> Ops, const Twine &Name = "");

Value *CreateExtractValue(Value *Agg, ArrayRef<unsigned> Idxs,
const Twine &Name = "") {
if (auto *V = Folder.FoldExtractValue(Agg, Idxs))
Expand Down
2 changes: 2 additions & 0 deletions llvm/include/llvm/IR/IRBuilderFolder.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,8 @@ class LLVM_ABI IRBuilderFolder {
virtual Value *FoldCast(Instruction::CastOps Op, Value *V,
Type *DestTy) const = 0;

virtual Value *FoldVectorInterleave(ArrayRef<Value *> Ops) const = 0;

virtual Value *
FoldBinaryIntrinsic(Intrinsic::ID ID, Value *LHS, Value *RHS, Type *Ty,
Instruction *FMFSource = nullptr) const = 0;
Expand Down
11 changes: 9 additions & 2 deletions llvm/include/llvm/IR/Intrinsics.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,8 +283,15 @@ namespace Intrinsic {
// or of the wrong kind will be renamed by adding ".renamed" to the name.
LLVM_ABI std::optional<Function *> remangleIntrinsicFunction(Function *F);

} // End Intrinsic namespace
/// Returns the corresponding llvm.vector.interleaveN intrinsic for factor N.
LLVM_ABI Intrinsic::ID getInterleaveIntrinsicID(unsigned Factor);

} // End llvm namespace
/// Returns the corresponding llvm.vector.deinterleaveN intrinsic for factor
/// N.
LLVM_ABI Intrinsic::ID getDeinterleaveIntrinsicID(unsigned Factor);

} // namespace Intrinsic

} // namespace llvm

#endif
4 changes: 4 additions & 0 deletions llvm/include/llvm/IR/NoFolder.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,10 @@ class LLVM_ABI NoFolder final : public IRBuilderFolder {
return nullptr;
}

Value *FoldVectorInterleave(ArrayRef<Value *> Ops) const override {
return nullptr;
}

//===--------------------------------------------------------------------===//
// Cast/Conversion Operators
//===--------------------------------------------------------------------===//
Expand Down
12 changes: 12 additions & 0 deletions llvm/lib/Analysis/ConstantFolding.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1641,7 +1641,19 @@ bool llvm::canConstantFoldCallTo(const CallBase *Call, const Function *F) {
case Intrinsic::vector_extract:
case Intrinsic::vector_insert:
case Intrinsic::vector_interleave2:
case Intrinsic::vector_interleave3:
case Intrinsic::vector_interleave4:
case Intrinsic::vector_interleave5:
case Intrinsic::vector_interleave6:
case Intrinsic::vector_interleave7:
case Intrinsic::vector_interleave8:
case Intrinsic::vector_deinterleave2:
case Intrinsic::vector_deinterleave3:
case Intrinsic::vector_deinterleave4:
case Intrinsic::vector_deinterleave5:
case Intrinsic::vector_deinterleave6:
case Intrinsic::vector_deinterleave7:
case Intrinsic::vector_deinterleave8:
// Target intrinsics
case Intrinsic::amdgcn_perm:
case Intrinsic::amdgcn_wave_reduce_umin:
Expand Down
24 changes: 0 additions & 24 deletions llvm/lib/Analysis/VectorUtils.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,30 +240,6 @@ Intrinsic::ID llvm::getVectorIntrinsicIDForCall(const CallInst *CI,
return Intrinsic::not_intrinsic;
}

struct InterleaveIntrinsic {
Intrinsic::ID Interleave, Deinterleave;
};

static InterleaveIntrinsic InterleaveIntrinsics[] = {
{Intrinsic::vector_interleave2, Intrinsic::vector_deinterleave2},
{Intrinsic::vector_interleave3, Intrinsic::vector_deinterleave3},
{Intrinsic::vector_interleave4, Intrinsic::vector_deinterleave4},
{Intrinsic::vector_interleave5, Intrinsic::vector_deinterleave5},
{Intrinsic::vector_interleave6, Intrinsic::vector_deinterleave6},
{Intrinsic::vector_interleave7, Intrinsic::vector_deinterleave7},
{Intrinsic::vector_interleave8, Intrinsic::vector_deinterleave8},
};

Intrinsic::ID llvm::getInterleaveIntrinsicID(unsigned Factor) {
assert(Factor >= 2 && Factor <= 8 && "Unexpected factor");
return InterleaveIntrinsics[Factor - 2].Interleave;
}

Intrinsic::ID llvm::getDeinterleaveIntrinsicID(unsigned Factor) {
assert(Factor >= 2 && Factor <= 8 && "Unexpected factor");
return InterleaveIntrinsics[Factor - 2].Deinterleave;
}

unsigned llvm::getInterleaveIntrinsicFactor(Intrinsic::ID ID) {
switch (ID) {
case Intrinsic::vector_interleave2:
Expand Down
17 changes: 7 additions & 10 deletions llvm/lib/CodeGen/ComplexDeinterleavingPass.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2194,11 +2194,10 @@ Value *ComplexDeinterleavingGraph::replaceNode(IRBuilderBase &Builder,
// Splats that are not constant are interleaved where they are located
Instruction *InsertPoint = (I->comesBefore(R) ? R : I)->getNextNode();
IRBuilder<> IRB(InsertPoint);
ReplacementNode = IRB.CreateIntrinsic(Intrinsic::vector_interleave2,
NewTy, {Node->Real, Node->Imag});
Copy link
Contributor

@fhahn fhahn Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like NewTy & NewMaskTy are unused now.

Copy link
Contributor Author

@david-arm david-arm Jul 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. #151100

ReplacementNode = IRB.CreateVectorInterleave({Node->Real, Node->Imag});
} else {
ReplacementNode = Builder.CreateIntrinsic(
Intrinsic::vector_interleave2, NewTy, {Node->Real, Node->Imag});
ReplacementNode =
Builder.CreateVectorInterleave({Node->Real, Node->Imag});
}
break;
}
Expand Down Expand Up @@ -2228,8 +2227,7 @@ Value *ComplexDeinterleavingGraph::replaceNode(IRBuilderBase &Builder,
auto *B = replaceNode(Builder, Node->Operands[1]);
auto *NewMaskTy = VectorType::getDoubleElementsVectorType(
cast<VectorType>(MaskReal->getType()));
auto *NewMask = Builder.CreateIntrinsic(Intrinsic::vector_interleave2,
NewMaskTy, {MaskReal, MaskImag});
auto *NewMask = Builder.CreateVectorInterleave({MaskReal, MaskImag});
ReplacementNode = Builder.CreateSelect(NewMask, A, B);
break;
}
Expand Down Expand Up @@ -2260,8 +2258,8 @@ void ComplexDeinterleavingGraph::processReductionSingle(
}

if (!NewInit)
NewInit = Builder.CreateIntrinsic(Intrinsic::vector_interleave2, NewVTy,
{Init, Constant::getNullValue(VTy)});
NewInit =
Builder.CreateVectorInterleave({Init, Constant::getNullValue(VTy)});

NewPHI->addIncoming(NewInit, Incoming);
NewPHI->addIncoming(OperationReplacement, BackEdge);
Expand Down Expand Up @@ -2289,8 +2287,7 @@ void ComplexDeinterleavingGraph::processReductionOperation(
Value *InitImag = OldPHIImag->getIncomingValueForBlock(Incoming);

IRBuilder<> Builder(Incoming->getTerminator());
auto *NewInit = Builder.CreateIntrinsic(Intrinsic::vector_interleave2, NewVTy,
{InitReal, InitImag});
auto *NewInit = Builder.CreateVectorInterleave({InitReal, InitImag});

NewPHI->addIncoming(NewInit, Incoming);
NewPHI->addIncoming(OperationReplacement, BackEdge);
Expand Down
33 changes: 30 additions & 3 deletions llvm/lib/IR/IRBuilder.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

#include "llvm/IR/IRBuilder.h"
#include "llvm/ADT/ArrayRef.h"
#include "llvm/Analysis/VectorUtils.h"
#include "llvm/IR/Constant.h"
#include "llvm/IR/Constants.h"
#include "llvm/IR/DerivedTypes.h"
Expand Down Expand Up @@ -1144,9 +1145,35 @@ Value *IRBuilderBase::CreateVectorSplat(ElementCount EC, Value *V,
return CreateShuffleVector(V, Zeros, Name + ".splat");
}

Value *IRBuilderBase::CreatePreserveArrayAccessIndex(
Type *ElTy, Value *Base, unsigned Dimension, unsigned LastIndex,
MDNode *DbgInfo) {
Value *IRBuilderBase::CreateVectorInterleave(ArrayRef<Value *> Ops,
const Twine &Name) {
assert(Ops.size() >= 2 && Ops.size() <= 8 &&
"Unexpected number of operands to interleave");

// Make sure all operands are the same type.
assert(isa<VectorType>(Ops[0]->getType()) && "Unexpected type");

#ifndef NDEBUG
for (unsigned I = 1; I < Ops.size(); I++) {
assert(Ops[I]->getType() == Ops[0]->getType() &&
"Vector interleave expects matching operand types!");
}
#endif

if (auto *V = Folder.FoldVectorInterleave(Ops))
return V;

unsigned IID = Intrinsic::getInterleaveIntrinsicID(Ops.size());
auto *SubvecTy = cast<VectorType>(Ops[0]->getType());
Type *DestTy = VectorType::get(SubvecTy->getElementType(),
SubvecTy->getElementCount() * Ops.size());
return CreateIntrinsic(IID, {DestTy}, Ops, {}, Name);
}

Value *IRBuilderBase::CreatePreserveArrayAccessIndex(Type *ElTy, Value *Base,
unsigned Dimension,
unsigned LastIndex,
MDNode *DbgInfo) {
auto *BaseType = Base->getType();
assert(isa<PointerType>(BaseType) &&
"Invalid Base ptr type for preserve.array.access.index.");
Expand Down
24 changes: 24 additions & 0 deletions llvm/lib/IR/Intrinsics.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1133,3 +1133,27 @@ std::optional<Function *> Intrinsic::remangleIntrinsicFunction(Function *F) {
"Shouldn't change the signature");
return NewDecl;
}

struct InterleaveIntrinsic {
Intrinsic::ID Interleave, Deinterleave;
};

static InterleaveIntrinsic InterleaveIntrinsics[] = {
{Intrinsic::vector_interleave2, Intrinsic::vector_deinterleave2},
{Intrinsic::vector_interleave3, Intrinsic::vector_deinterleave3},
{Intrinsic::vector_interleave4, Intrinsic::vector_deinterleave4},
{Intrinsic::vector_interleave5, Intrinsic::vector_deinterleave5},
{Intrinsic::vector_interleave6, Intrinsic::vector_deinterleave6},
{Intrinsic::vector_interleave7, Intrinsic::vector_deinterleave7},
{Intrinsic::vector_interleave8, Intrinsic::vector_deinterleave8},
};

Intrinsic::ID Intrinsic::getInterleaveIntrinsicID(unsigned Factor) {
assert(Factor >= 2 && Factor <= 8 && "Unexpected factor");
return InterleaveIntrinsics[Factor - 2].Interleave;
}

Intrinsic::ID Intrinsic::getDeinterleaveIntrinsicID(unsigned Factor) {
assert(Factor >= 2 && Factor <= 8 && "Unexpected factor");
return InterleaveIntrinsics[Factor - 2].Deinterleave;
}
11 changes: 3 additions & 8 deletions llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3391,12 +3391,7 @@ static Value *interleaveVectors(IRBuilderBase &Builder, ArrayRef<Value *> Vals,
// must use intrinsics to interleave.
if (VecTy->isScalableTy()) {
assert(Factor <= 8 && "Unsupported interleave factor for scalable vectors");
VectorType *InterleaveTy =
VectorType::get(VecTy->getElementType(),
VecTy->getElementCount().multiplyCoefficientBy(Factor));
return Builder.CreateIntrinsic(InterleaveTy,
getInterleaveIntrinsicID(Factor), Vals,
/*FMFSource=*/nullptr, Name);
return Builder.CreateVectorInterleave(Vals, Name);
}

// Fixed length. Start by concatenating all vectors into a wide vector.
Expand Down Expand Up @@ -3503,8 +3498,8 @@ void VPInterleaveRecipe::execute(VPTransformState &State) {
assert(InterleaveFactor <= 8 &&
"Unsupported deinterleave factor for scalable vectors");
NewLoad = State.Builder.CreateIntrinsic(
getDeinterleaveIntrinsicID(InterleaveFactor), NewLoad->getType(),
NewLoad,
Intrinsic::getDeinterleaveIntrinsicID(InterleaveFactor),
NewLoad->getType(), NewLoad,
/*FMFSource=*/nullptr, "strided.vec");
}

Expand Down
6 changes: 2 additions & 4 deletions llvm/test/CodeGen/AArch64/complex-deinterleaving-opt-crash.ll
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,9 @@ define void @reprocessing_crash() #0 {
; CHECK-LABEL: define void @reprocessing_crash(
; CHECK-SAME: ) #[[ATTR0:[0-9]+]] {
; CHECK-NEXT: [[ENTRY:.*]]:
; CHECK-NEXT: [[TMP0:%.*]] = call <vscale x 4 x double> @llvm.vector.interleave2.nxv4f64(<vscale x 2 x double> zeroinitializer, <vscale x 2 x double> zeroinitializer)
; CHECK-NEXT: br label %[[VECTOR_BODY:.*]]
; CHECK: [[VECTOR_BODY]]:
; CHECK-NEXT: [[TMP1:%.*]] = phi <vscale x 4 x double> [ [[TMP0]], %[[ENTRY]] ], [ [[TMP2:%.*]], %[[VECTOR_BODY]] ]
; CHECK-NEXT: [[TMP1:%.*]] = phi <vscale x 4 x double> [ zeroinitializer, %[[ENTRY]] ], [ [[TMP2:%.*]], %[[VECTOR_BODY]] ]
; CHECK-NEXT: [[TMP2]] = fsub <vscale x 4 x double> [[TMP1]], zeroinitializer
; CHECK-NEXT: br i1 false, label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]]
; CHECK: [[MIDDLE_BLOCK]]:
Expand Down Expand Up @@ -48,11 +47,10 @@ define double @test_fp_single_reduction(i1 %c) #2 {
; CHECK-LABEL: define double @test_fp_single_reduction(
; CHECK-SAME: i1 [[C:%.*]]) #[[ATTR1:[0-9]+]] {
; CHECK-NEXT: [[ENTRY:.*]]:
; CHECK-NEXT: [[TMP0:%.*]] = call <8 x double> @llvm.vector.interleave2.v8f64(<4 x double> zeroinitializer, <4 x double> zeroinitializer)
; CHECK-NEXT: br label %[[VECTOR_BODY:.*]]
; CHECK: [[VECTOR_BODY]]:
; CHECK-NEXT: [[VEC_PHI218:%.*]] = phi <4 x double> [ zeroinitializer, %[[ENTRY]] ], [ [[TMP2:%.*]], %[[VECTOR_BODY]] ]
; CHECK-NEXT: [[TMP1:%.*]] = phi <8 x double> [ [[TMP0]], %[[ENTRY]] ], [ [[TMP3:%.*]], %[[VECTOR_BODY]] ]
; CHECK-NEXT: [[TMP1:%.*]] = phi <8 x double> [ zeroinitializer, %[[ENTRY]] ], [ [[TMP3:%.*]], %[[VECTOR_BODY]] ]
; CHECK-NEXT: [[STRIDED_VEC:%.*]] = shufflevector <8 x double> zeroinitializer, <8 x double> zeroinitializer, <4 x i32> <i32 0, i32 2, i32 4, i32 6>
; CHECK-NEXT: [[TMP2]] = fadd <4 x double> [[VEC_PHI218]], [[STRIDED_VEC]]
; CHECK-NEXT: [[TMP3]] = fadd <8 x double> [[TMP1]], zeroinitializer
Expand Down
Loading