Skip to content

[LV] Create in-loop sub reductions #147026

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Aug 12, 2025
Merged

[LV] Create in-loop sub reductions #147026

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
6171fd6
[LV] Create in-loop sub reductions
SamTebbs33 Jul 1, 2025
93dd762
Use empty flags for negation
SamTebbs33 Jul 16, 2025
52a8138
Improve test comment
SamTebbs33 Jul 16, 2025
2f7a77f
Add RecurKind::Sub
SamTebbs33 Jul 16, 2025
b38b1cb
Address review
SamTebbs33 Jul 18, 2025
75baa8e
Fix inloop reductions
SamTebbs33 Jul 21, 2025
0669c78
Remove CurrentLinkI opcode check
SamTebbs33 Jul 21, 2025
e855872
Fix interleaving case
SamTebbs33 Jul 22, 2025
35b6b9d
format
SamTebbs33 Jul 22, 2025
c659696
Improve codegen
SamTebbs33 Jul 29, 2025
2b8e239
Address review
SamTebbs33 Jul 30, 2025
bdf2604
format
SamTebbs33 Jul 30, 2025
83f029c
Improve mixed recurrence kind cases
SamTebbs33 Jul 31, 2025
6978e95
Remove check in isCorrectOpcode
SamTebbs33 Jul 31, 2025
f53accf
Update test comment and set start value to 3
SamTebbs33 Aug 1, 2025
0033606
Add Sub cases to SLPVectorizer
SamTebbs33 Aug 1, 2025
3ae3e67
Add test for mixed inloop reductions
SamTebbs33 Aug 4, 2025
0f5cb6d
Format and remove check from isCorrectOpcode
SamTebbs33 Aug 5, 2025
bd5c474
Make RecurKind::Sub unreachable in SLPVectorizer.cpp
SamTebbs33 Aug 6, 2025
6a6bb86
Add RecurKind::AddChainWithSubs
SamTebbs33 Aug 7, 2025
c32b0e1
Simplify isRecurrenceInstr, getReductionOpChain and remove a case in SLP
SamTebbs33 Aug 8, 2025
340003b
Revert partial-reduce-chained.ll changes
SamTebbs33 Aug 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions llvm/include/llvm/Analysis/IVDescriptors.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ enum class RecurKind {
// clang-format off
None, ///< Not a recurrence.
Add, ///< Sum of integers.
Sub, ///< Subtraction of integers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You will need to update a number of places with switches over recurrence kind, in SLPVectorizer and possibly others

Probably also should have a SLP test to make sure its handled correctly.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've added RecurKind::Sub cases to the SLP vectorizer and added a test. I don't think adding proper sub reduction support to SLP is in scope of this PR so haven't added full support.

Mul, ///< Product of integers.
Or, ///< Bitwise or logical OR of integers.
And, ///< Bitwise or logical AND of integers.
Expand Down
31 changes: 29 additions & 2 deletions llvm/lib/Analysis/IVDescriptors.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ bool RecurrenceDescriptor::isIntegerRecurrenceKind(RecurKind Kind) {
switch (Kind) {
default:
break;
case RecurKind::Sub:
case RecurKind::Add:
case RecurKind::Mul:
case RecurKind::Or:
Expand Down Expand Up @@ -897,8 +898,23 @@ RecurrenceDescriptor::InstDesc RecurrenceDescriptor::isRecurrenceInstr(
case Instruction::PHI:
return InstDesc(I, Prev.getRecKind(), Prev.getExactFPMathInst());
case Instruction::Sub:
if (Prev.getRecKind() == RecurKind::Add && Kind == RecurKind::Add)
return InstDesc(I, Prev.getRecKind());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is to support chains of instructions ,where the first one is an add, but the backedge value is a sub, right?

Do we have a test with in-loop reductions for that to make sure we handle this correctly?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that's correct. We have partial-reduce-chained.ll which tests for the first one being an add and the backedge value being a sub, but those weren't creating inloop reductions so I have added a more explicit test to reduction-inloop.ll and re-added some checks that are necessary for mixed in-loop reductions. Thank you.

else if (Kind == RecurKind::Sub)
return InstDesc(I, Kind);
else
return InstDesc(false, I);
case Instruction::Add:
return InstDesc(Kind == RecurKind::Add, I);
// Loops with a sub reduction followed by an add reduction will have the sub
// input negated. It needs to be recorded as RecurKind::Add for that to
// happen since the loop vectorizer considers the last found RecurKind for
// the reduction phi's kind.
if (Prev.getRecKind() == RecurKind::Sub && Kind == RecurKind::Sub)
return InstDesc(I, RecurKind::Add);
else if (Kind == RecurKind::Add)
return InstDesc(I, Kind);
else
return InstDesc(false, I);
case Instruction::Mul:
return InstDesc(Kind == RecurKind::Mul, I);
case Instruction::And:
Expand All @@ -917,7 +933,8 @@ RecurrenceDescriptor::InstDesc RecurrenceDescriptor::isRecurrenceInstr(
I->hasAllowReassoc() ? nullptr : I);
case Instruction::Select:
if (Kind == RecurKind::FAdd || Kind == RecurKind::FMul ||
Kind == RecurKind::Add || Kind == RecurKind::Mul)
Kind == RecurKind::Add || Kind == RecurKind::Mul ||
Kind == RecurKind::Sub)
return isConditionalRdxPattern(I);
if (isFindIVRecurrenceKind(Kind) && SE)
return isFindIVPattern(Kind, L, OrigPhi, I, *SE);
Expand Down Expand Up @@ -1003,6 +1020,11 @@ bool RecurrenceDescriptor::isReductionPHI(PHINode *Phi, Loop *TheLoop,
LLVM_DEBUG(dbgs() << "Found an ADD reduction PHI." << *Phi << "\n");
return true;
}
if (AddReductionVar(Phi, RecurKind::Sub, TheLoop, FMF, RedDes, DB, AC, DT,
SE)) {
LLVM_DEBUG(dbgs() << "Found a SUB reduction PHI." << *Phi << "\n");
return true;
}
if (AddReductionVar(Phi, RecurKind::Mul, TheLoop, FMF, RedDes, DB, AC, DT,
SE)) {
LLVM_DEBUG(dbgs() << "Found a MUL reduction PHI." << *Phi << "\n");
Expand Down Expand Up @@ -1201,6 +1223,8 @@ bool RecurrenceDescriptor::isFixedOrderRecurrence(PHINode *Phi, Loop *TheLoop,

unsigned RecurrenceDescriptor::getOpcode(RecurKind Kind) {
switch (Kind) {
case RecurKind::Sub:
return Instruction::Sub;
case RecurKind::Add:
return Instruction::Add;
case RecurKind::Mul:
Expand Down Expand Up @@ -1288,6 +1312,9 @@ RecurrenceDescriptor::getReductionOpChain(PHINode *Phi, Loop *L) const {
if (isFMulAddIntrinsic(Cur))
return true;

if (Cur->getOpcode() == Instruction::Sub && getOpcode() == Instruction::Add)
return true;

return Cur->getOpcode() == getOpcode();
};

Expand Down
1 change: 1 addition & 0 deletions llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5151,6 +5151,7 @@ bool AArch64TTIImpl::isLegalToVectorizeReduction(
return false;

switch (RdxDesc.getRecurrenceKind()) {
case RecurKind::Sub:
case RecurKind::Add:
case RecurKind::FAdd:
case RecurKind::And:
Expand Down
2 changes: 2 additions & 0 deletions llvm/lib/Transforms/Utils/LoopUtils.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -914,6 +914,7 @@ constexpr Intrinsic::ID llvm::getReductionIntrinsicID(RecurKind RK) {
switch (RK) {
default:
llvm_unreachable("Unexpected recurrence kind");
case RecurKind::Sub:
case RecurKind::Add:
return Intrinsic::vector_reduce_add;
case RecurKind::Mul:
Expand Down Expand Up @@ -1301,6 +1302,7 @@ Value *llvm::createSimpleReduction(IRBuilderBase &Builder, Value *Src,
Builder.getFastMathFlags());
};
switch (RdxKind) {
case RecurKind::Sub:
case RecurKind::Add:
case RecurKind::Mul:
case RecurKind::And:
Expand Down
10 changes: 10 additions & 0 deletions llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9118,6 +9118,16 @@ void LoopVectorizationPlanner::adjustRecipesForReductions(
CurrentLinkI->getFastMathFlags());
LinkVPBB->insert(FMulRecipe, CurrentLink->getIterator());
VecOp = FMulRecipe;
} else if (PhiR->isInLoop() && Kind == RecurKind::Add &&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason this cannot be classified as Sub reduction? Would be good to update the RecurrenceDescriptor to clarify that RecurKind::Add can be a sub-reduction.

Would be great if we could avoid having RecurKind meanings depend on whether it's in loop or not,

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there is a reason so I'll experiment with adding RecurKind::Sub 👍 .

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Let me know what you think.

CurrentLinkI->getOpcode() == Instruction::Sub) {
Type *PhiTy = PhiR->getUnderlyingValue()->getType();
auto *Zero = Plan->getOrAddLiveIn(ConstantInt::get(PhiTy, 0));
VPWidenRecipe *Sub = new VPWidenRecipe(
Instruction::Sub, {Zero, CurrentLink->getOperand(1)}, {},
VPIRMetadata(), CurrentLinkI->getDebugLoc());
Sub->setUnderlyingValue(CurrentLinkI);
LinkVPBB->insert(Sub, CurrentLink->getIterator());
VecOp = Sub;
} else {
if (RecurrenceDescriptor::isMinMaxRecurrenceKind(Kind)) {
if (isa<VPWidenRecipe>(CurrentLink)) {
Expand Down
23 changes: 17 additions & 6 deletions llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22344,6 +22344,7 @@ class HorizontalReduction {
return Builder.CreateBinOp((Instruction::BinaryOps)RdxOpcode, LHS, RHS,
Name);
}
case RecurKind::Sub:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not so sure this is the right approach, because it suggests that Sub is somehow supported, which it isn't (yet).
My suggestion would be to do the following:

  • if a switch-statement has no default case, then we should add a llvm_unreachable to make it clear that RecurKind::Sub is unexpected and unsupported.
  • if the switch-statement has a default case with an llvm_unreachable to catch unimplemented RecurKind's, then strictly speaking no action is required, although a specific llvm_unreachable might still be helpful.

From looking at the SLPVectorizer code, it never tries to match a RecurKind::Sub so in other places it should never encounter one either, and so a llvm_unreachable would be fine.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it suggested sub is supported when it really isn't. Fixed now, thanks!

case RecurKind::Add:
case RecurKind::Mul:
case RecurKind::Xor:
Expand Down Expand Up @@ -23505,6 +23506,7 @@ class HorizontalReduction {
// vector with full register use).
bool DoesRequireReductionOp = !AllConsts && VectorValuesAndScales.empty();
switch (RdxKind) {
case RecurKind::Sub:
case RecurKind::Add:
case RecurKind::Mul:
case RecurKind::Or:
Expand Down Expand Up @@ -23641,6 +23643,7 @@ class HorizontalReduction {
if (Cnt > 1) {
ElementCount EC = cast<VectorType>(Vec->getType())->getElementCount();
switch (RdxKind) {
case RecurKind::Sub:
case RecurKind::Add: {
if (ScalarTy == Builder.getInt1Ty() && ScalarTy != DestTy) {
unsigned VF = getNumElements(Vec->getType());
Expand All @@ -23661,8 +23664,9 @@ class HorizontalReduction {
IsSigned);
Value *Scale = ConstantVector::getSplat(
EC, ConstantInt::get(DestTy->getScalarType(), Cnt));
LLVM_DEBUG(dbgs() << "SLP: Add (to-mul) " << Cnt << "of " << Vec
<< ". (HorRdx)\n");
LLVM_DEBUG(dbgs()
<< "SLP: " << (RdxKind == RecurKind::Add ? "Add" : "Sub")
<< " (to-mul) " << Cnt << "of " << Vec << ". (HorRdx)\n");
++NumVectorInstructions;
Vec = Builder.CreateMul(Vec, Scale);
break;
Expand Down Expand Up @@ -23802,11 +23806,14 @@ class HorizontalReduction {
if (Cnt == 1)
return VectorizedValue;
switch (RdxKind) {
case RecurKind::Sub:
case RecurKind::Add: {
// res = mul vv, n
Value *Scale = ConstantInt::get(VectorizedValue->getType(), Cnt);
LLVM_DEBUG(dbgs() << "SLP: Add (to-mul) " << Cnt << "of "
<< VectorizedValue << ". (HorRdx)\n");
LLVM_DEBUG(dbgs() << "SLP: "
<< (RdxKind == RecurKind::Add ? "Add" : "Sub")
<< " (to-mul) " << Cnt << "of " << VectorizedValue
<< ". (HorRdx)\n");
return Builder.CreateMul(VectorizedValue, Scale);
}
case RecurKind::Xor: {
Expand Down Expand Up @@ -23872,6 +23879,7 @@ class HorizontalReduction {
R.isSignedMinBitwidthRootNode());
}
switch (RdxKind) {
case RecurKind::Sub:
case RecurKind::Add: {
// root = mul prev_root, <1, 1, n, 1>
SmallVector<Constant *> Vals;
Expand All @@ -23880,8 +23888,10 @@ class HorizontalReduction {
Vals.push_back(ConstantInt::get(V->getType(), Cnt, /*IsSigned=*/false));
}
auto *Scale = ConstantVector::get(Vals);
LLVM_DEBUG(dbgs() << "SLP: Add (to-mul) " << Scale << "of "
<< VectorizedValue << ". (HorRdx)\n");
LLVM_DEBUG(dbgs() << "SLP: "
<< (RdxKind == RecurKind::Add ? "Add" : "Sub")
<< " (to-mul) " << Scale << "of " << VectorizedValue
<< ". (HorRdx)\n");
return Builder.CreateMul(VectorizedValue, Scale);
}
case RecurKind::And:
Expand Down Expand Up @@ -24331,6 +24341,7 @@ bool SLPVectorizerPass::tryToVectorize(Instruction *I, BoUpSLP &R) {
TTI.getInstructionCost(Inst, CostKind);
InstructionCost RedCost;
switch (::getRdxKind(Inst)) {
case RecurKind::Sub:
case RecurKind::Add:
case RecurKind::Mul:
case RecurKind::Or:
Expand Down
16 changes: 12 additions & 4 deletions llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -812,10 +812,18 @@ Value *VPInstruction::generate(VPTransformState &State) {
Value *RdxPart = RdxParts[Part];
if (RecurrenceDescriptor::isMinMaxRecurrenceKind(RK))
ReducedPartRdx = createMinMaxOp(Builder, RK, ReducedPartRdx, RdxPart);
else
ReducedPartRdx = Builder.CreateBinOp(
(Instruction::BinaryOps)RecurrenceDescriptor::getOpcode(RK),
RdxPart, ReducedPartRdx, "bin.rdx");
else {
Instruction::BinaryOps Opcode;
// For sub-recurrences, each UF's reduction variable is already
// negative, we need to do: reduce.add(-acc_uf0 + -acc_uf1)
if (RK == RecurKind::Sub)
Opcode = Instruction::Add;
else
Opcode =
(Instruction::BinaryOps)RecurrenceDescriptor::getOpcode(RK);
ReducedPartRdx =
Builder.CreateBinOp(Opcode, RdxPart, ReducedPartRdx, "bin.rdx");
}
}
}

Expand Down
2 changes: 1 addition & 1 deletion llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1742,7 +1742,7 @@ void VPlanTransforms::clearReductionWrapFlags(VPlan &Plan) {
if (!PhiR)
continue;
RecurKind RK = PhiR->getRecurrenceKind();
if (RK != RecurKind::Add && RK != RecurKind::Mul)
if (RK != RecurKind::Add && RK != RecurKind::Mul && RK != RecurKind::Sub)
continue;

for (VPUser *U : collectUsersRecursively(PhiR))
Expand Down
Loading
Loading