[VPlan] Move initial skeleton construction earlier (NFC). #150848

fhahn · 2025-07-27T19:20:29Z

Split up the not clearly named prepareForVectorization transform into addInitialSkeleton, which adds the vector preheader, middle and scalar preheader blocks, as well as the canonical induction recipes and sets the trip count. The new transform is run directly after building the plain CFG VPlan initially.

The remaining code handling early exits and adding the branch in the middle block is renamed to handleEarlyExitsAndAddMiddleCheck and still runs at the original position.

With the code movement, we only have to add the skeleton once to the initial VPlan, and cloning will take care of the rest. It will also enable moving other construction steps to work directly on VPlan0, like adding resume phis.

Split up the not clearly named prepareForVectorization transform into addInitialSkeleton, which adds the vector preheader, middle and scalar preheader blocks, as well as the canonical induction recipes and sets the trip count. The new transform is run directly after building the plain CFG VPlan initially. The remaining code handling early exits and adding the branch in the middle block is renamed to handleEarlyExitsAndAddMiddleCheck and still runs at the original position. With the code movement, we only have to add the skeleton once to the initial VPlan, and cloning will take care of the rest. It will also enable moving other construction steps to work directly on VPlan0, like adding resume phis.

llvmbot · 2025-07-27T19:21:00Z

@llvm/pr-subscribers-vectorizers

@llvm/pr-subscribers-llvm-transforms

Author: Florian Hahn (fhahn)

Changes

Split up the not clearly named prepareForVectorization transform into addInitialSkeleton, which adds the vector preheader, middle and scalar preheader blocks, as well as the canonical induction recipes and sets the trip count. The new transform is run directly after building the plain CFG VPlan initially.

The remaining code handling early exits and adding the branch in the middle block is renamed to handleEarlyExitsAndAddMiddleCheck and still runs at the original position.

With the code movement, we only have to add the skeleton once to the initial VPlan, and cloning will take care of the rest. It will also enable moving other construction steps to work directly on VPlan0, like adding resume phis.

Full diff: https://github.com/llvm/llvm-project/pull/150848.diff

4 Files Affected:

(modified) llvm/lib/Transforms/Vectorize/LoopVectorize.cpp (+20-15)
(modified) llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp (+42-34)
(modified) llvm/lib/Transforms/Vectorize/VPlanTransforms.h (+14-10)
(modified) llvm/unittests/Transforms/Vectorize/VPlanTestBase.h (+5-2)

diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 6616e61f9bb84..59b276b5096fd 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -8373,8 +8373,18 @@ void LoopVectorizationPlanner::buildVPlansWithVPRecipes(ElementCount MinVF,
     LVer.prepareNoAliasMetadata();
   }
 
-  auto MaxVFTimes2 = MaxVF * 2;
+  // Create initial VPlan skeleton, having a basic block for the pre-header
+  // which contains SCEV expansions that need to happen before the CFG is
+  // modified; a basic block for the vector pre-header, followed by a region for
+  // the vector loop, followed by the middle basic block, connecting to the
+  // scalar preheader and exit blcoks.
   auto VPlan0 = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);
+  VPlanTransforms::addInitialSkeleton(
+      *VPlan0, Legal->getWidestInductionType(),
+      getDebugLocFromInstOrOperands(Legal->getPrimaryInduction()), PSE,
+      OrigLoop);
+
+  auto MaxVFTimes2 = MaxVF * 2;
   for (ElementCount VF = MinVF; ElementCount::isKnownLT(VF, MaxVFTimes2);) {
     VFRange SubRange = {VF, MaxVFTimes2};
     if (auto Plan = tryToBuildVPlanWithVPRecipes(
@@ -8615,22 +8625,14 @@ VPlanPtr LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(
   // visit each basic block after having visited its predecessor basic blocks.
   // ---------------------------------------------------------------------------
 
-  // Create initial VPlan skeleton, having a basic block for the pre-header
-  // which contains SCEV expansions that need to happen before the CFG is
-  // modified; a basic block for the vector pre-header, followed by a region for
-  // the vector loop, followed by the middle basic block. The skeleton vector
-  // loop region contains a header and latch basic blocks.
-
   bool RequiresScalarEpilogueCheck =
       LoopVectorizationPlanner::getDecisionAndClampRange(
           [this](ElementCount VF) {
             return !CM.requiresScalarEpilogue(VF.isVector());
           },
           Range);
-  VPlanTransforms::prepareForVectorization(
-      *Plan, Legal->getWidestInductionType(), PSE, RequiresScalarEpilogueCheck,
-      CM.foldTailByMasking(), OrigLoop,
-      getDebugLocFromInstOrOperands(Legal->getPrimaryInduction()),
+  VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(
+      *Plan, RequiresScalarEpilogueCheck, CM.foldTailByMasking(),
       Legal->hasUncountableEarlyExit(), Range);
   VPlanTransforms::createLoopRegions(*Plan);
   VPlanTransforms::createExtractsForLiveOuts(*Plan);
@@ -8918,10 +8920,13 @@ VPlanPtr LoopVectorizationPlanner::tryToBuildVPlan(VFRange &Range) {
   assert(EnableVPlanNativePath && "VPlan-native path is not enabled.");
 
   auto Plan = VPlanTransforms::buildPlainCFG(OrigLoop, *LI);
-  VPlanTransforms::prepareForVectorization(
-      *Plan, Legal->getWidestInductionType(), PSE, true, false, OrigLoop,
-      getDebugLocFromInstOrOperands(Legal->getPrimaryInduction()), false,
-      Range);
+
+  VPlanTransforms::addInitialSkeleton(
+      *Plan, Legal->getWidestInductionType(),
+      getDebugLocFromInstOrOperands(Legal->getPrimaryInduction()), PSE,
+      OrigLoop);
+  VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(*Plan, true, false, false,
+                                                     Range);
   VPlanTransforms::createLoopRegions(*Plan);
 
   for (ElementCount VF : Range)
diff --git a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
index 6c1f53b4eaa24..92b2d017d7a11 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
@@ -459,10 +459,10 @@ static void addCanonicalIVRecipes(VPlan &Plan, VPBasicBlock *HeaderVPBB,
                        LatchDL);
 }
 
-void VPlanTransforms::prepareForVectorization(
-    VPlan &Plan, Type *InductionTy, PredicatedScalarEvolution &PSE,
-    bool RequiresScalarEpilogueCheck, bool TailFolded, Loop *TheLoop,
-    DebugLoc IVDL, bool HasUncountableEarlyExit, VFRange &Range) {
+void VPlanTransforms::addInitialSkeleton(VPlan &Plan, Type *InductionTy,
+                                         DebugLoc IVDL,
+                                         PredicatedScalarEvolution &PSE,
+                                         Loop *TheLoop) {
   VPDominatorTree VPDT;
   VPDT.recalculate(Plan);
 
@@ -488,12 +488,46 @@ void VPlanTransforms::prepareForVectorization(
 
   addCanonicalIVRecipes(Plan, HeaderVPBB, LatchVPBB, InductionTy, IVDL);
 
-  [[maybe_unused]] bool HandledUncountableEarlyExit = false;
+  // Create SCEV and VPValue for the trip count.
+  // We use the symbolic max backedge-taken-count, which works also when
+  // vectorizing loops with uncountable early exits.
+  const SCEV *BackedgeTakenCountSCEV = PSE.getSymbolicMaxBackedgeTakenCount();
+  assert(!isa<SCEVCouldNotCompute>(BackedgeTakenCountSCEV) &&
+         "Invalid loop count");
+  ScalarEvolution &SE = *PSE.getSE();
+  const SCEV *TripCount = SE.getTripCountFromExitCount(BackedgeTakenCountSCEV,
+                                                       InductionTy, TheLoop);
+  Plan.setTripCount(
+      vputils::getOrCreateVPValueForSCEVExpr(Plan, TripCount, SE));
+
+  VPBasicBlock *ScalarPH = Plan.createVPBasicBlock("scalar.ph");
+  VPBlockUtils::connectBlocks(ScalarPH, Plan.getScalarHeader());
+
+  // The connection order corresponds to the operands of the conditional branch,
+  // with the middle block already connected to the exit block.
+  VPBlockUtils::connectBlocks(MiddleVPBB, ScalarPH);
+  // Also connect the entry block to the scalar preheader.
+  // TODO: Also introduce a branch recipe together with the minimum trip count
+  // check.
+  VPBlockUtils::connectBlocks(Plan.getEntry(), ScalarPH);
+  Plan.getEntry()->swapSuccessors();
+}
+
+void VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(
+    VPlan &Plan, bool RequiresScalarEpilogueCheck, bool TailFolded,
+    bool HasUncountableEarlyExit, VFRange &Range) {
+  auto *MiddleVPBB = cast<VPBasicBlock>(
+      Plan.getScalarHeader()->getSinglePredecessor()->getPredecessors()[0]);
+  VPBlockBase *HeaderVPB =
+      Plan.getEntry()->getSuccessors()[1]->getSingleSuccessor();
+  auto *LatchVPBB = cast<VPBasicBlock>(HeaderVPB->getPredecessors()[1]);
+
   // Disconnect all early exits from the loop leaving it with a single exit from
   // the latch. Early exits that are countable are left for a scalar epilog. The
   // condition of uncountable early exits (currently at most one is supported)
   // is fused into the latch exit, and used to branch from middle block to the
   // early exit destination.
+  [[maybe_unused]] bool HandledUncountableEarlyExit = false;
   for (VPIRBasicBlock *EB : Plan.getExitBlocks()) {
     for (VPBlockBase *Pred : to_vector(EB->getPredecessors())) {
       if (Pred == MiddleVPBB)
@@ -502,7 +536,8 @@ void VPlanTransforms::prepareForVectorization(
         assert(!HandledUncountableEarlyExit &&
                "can handle exactly one uncountable early exit");
         handleUncountableEarlyExit(cast<VPBasicBlock>(Pred), EB, Plan,
-                                   HeaderVPBB, LatchVPBB, Range);
+                                   cast<VPBasicBlock>(HeaderVPB), LatchVPBB,
+                                   Range);
         HandledUncountableEarlyExit = true;
       } else {
         for (VPRecipeBase &R : EB->phis())
@@ -516,38 +551,11 @@ void VPlanTransforms::prepareForVectorization(
   assert((!HasUncountableEarlyExit || HandledUncountableEarlyExit) &&
          "missed an uncountable exit that must be handled");
 
-  // Create SCEV and VPValue for the trip count.
-  // We use the symbolic max backedge-taken-count, which works also when
-  // vectorizing loops with uncountable early exits.
-  const SCEV *BackedgeTakenCountSCEV = PSE.getSymbolicMaxBackedgeTakenCount();
-  assert(!isa<SCEVCouldNotCompute>(BackedgeTakenCountSCEV) &&
-         "Invalid loop count");
-  ScalarEvolution &SE = *PSE.getSE();
-  const SCEV *TripCount = SE.getTripCountFromExitCount(BackedgeTakenCountSCEV,
-                                                       InductionTy, TheLoop);
-  Plan.setTripCount(
-      vputils::getOrCreateVPValueForSCEVExpr(Plan, TripCount, SE));
-
-  VPBasicBlock *ScalarPH = Plan.createVPBasicBlock("scalar.ph");
-  VPBlockUtils::connectBlocks(ScalarPH, Plan.getScalarHeader());
-
-  // The connection order corresponds to the operands of the conditional branch,
-  // with the middle block already connected to the exit block.
-  VPBlockUtils::connectBlocks(MiddleVPBB, ScalarPH);
-  // Also connect the entry block to the scalar preheader.
-  // TODO: Also introduce a branch recipe together with the minimum trip count
-  // check.
-  VPBlockUtils::connectBlocks(Plan.getEntry(), ScalarPH);
-  Plan.getEntry()->swapSuccessors();
-
   // If MiddleVPBB has a single successor then the original loop does not exit
   // via the latch and the single successor must be the scalar preheader.
   // There's no need to add a runtime check to MiddleVPBB.
-  if (MiddleVPBB->getNumSuccessors() == 1) {
-    assert(MiddleVPBB->getSingleSuccessor() == ScalarPH &&
-           "must have ScalarPH as single successor");
+  if (MiddleVPBB->getNumSuccessors() == 1)
     return;
-  }
 
   assert(MiddleVPBB->getNumSuccessors() == 2 && "must have 2 successors");
 
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
index d5af6cd73a4a0..0b24cb00ff73c 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
@@ -58,17 +58,21 @@ struct VPlanTransforms {
                                                                 LoopInfo &LI);
 
   /// Prepare the plan for vectorization. It will introduce a dedicated
-  /// VPBasicBlock for the vector pre-header as well as a VPBasicBlock as exit
-  /// block of the main vector loop (middle.block). If a check is needed to
+  /// VPBasicBlock for the vector pre-header, a VPBasicBlock as exit
+  /// block of the main vector loop (middle.block) and a VPBaiscBlock for the
+  /// scalar preheader. It also adds a canonical IV and its increment, using \p
+  /// InductionTy and \p IVDL, and creates a VPValue expression for the original
+  /// trip count.
+  LLVM_ABI_FOR_TEST static void
+  addInitialSkeleton(VPlan &Plan, Type *InductionTy, DebugLoc IVDL,
+                     PredicatedScalarEvolution &PSE, Loop *TheLoop);
+
+  /// Update \p Plan to account for all early exits. If a check is needed to
   /// guard executing the scalar epilogue loop, it will be added to the middle
-  /// block, together with VPBasicBlocks for the scalar preheader and exit
-  /// blocks. \p InductionTy is the type of the canonical induction and used for
-  /// related values, like the trip count expression.  It also creates a VPValue
-  /// expression for the original trip count.
-  LLVM_ABI_FOR_TEST static void prepareForVectorization(
-      VPlan &Plan, Type *InductionTy, PredicatedScalarEvolution &PSE,
-      bool RequiresScalarEpilogueCheck, bool TailFolded, Loop *TheLoop,
-      DebugLoc IVDL, bool HasUncountableExit, VFRange &Range);
+  /// block
+  LLVM_ABI_FOR_TEST static void handleEarlyExitsAndAddMiddleCheck(
+      VPlan &Plan, bool RequiresScalarEpilogueCheck, bool TailFolded,
+      bool HasUncountableExit, VFRange &Range);
 
   /// Replace loops in \p Plan's flat CFG with VPRegionBlocks, turning \p Plan's
   /// flat CFG into a hierarchical CFG.
diff --git a/llvm/unittests/Transforms/Vectorize/VPlanTestBase.h b/llvm/unittests/Transforms/Vectorize/VPlanTestBase.h
index 7dfd11a48b595..877394cc22ba2 100644
--- a/llvm/unittests/Transforms/Vectorize/VPlanTestBase.h
+++ b/llvm/unittests/Transforms/Vectorize/VPlanTestBase.h
@@ -74,8 +74,11 @@ class VPlanTestIRBase : public testing::Test {
     PredicatedScalarEvolution PSE(*SE, *L);
     auto Plan = VPlanTransforms::buildPlainCFG(L, *LI);
     VFRange R(ElementCount::getFixed(1), ElementCount::getFixed(2));
-    VPlanTransforms::prepareForVectorization(*Plan, IntegerType::get(*Ctx, 64),
-                                             PSE, true, false, L, {}, false, R);
+    VPlanTransforms::addInitialSkeleton(*Plan, IntegerType::get(*Ctx, 64), {},
+                                        PSE, L);
+
+    VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(*Plan, true, false,
+                                                       false, R);
     VPlanTransforms::createLoopRegions(*Plan);
     return Plan;
   }

artagnon

Looks good overall; just a few suggestions/clarifications.

llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp

artagnon · 2025-07-30T13:57:59Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+  VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(*Plan, true, false, false,
+                                                     Range);


Inline comments annotating the true/false args?

artagnon · 2025-07-30T14:02:04Z

llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp

-    assert(MiddleVPBB->getSingleSuccessor() == ScalarPH &&
-           "must have ScalarPH as single successor");


Why is this assert gone?

It's easy to retrieve the scalar preheader here, added back, thanks

artagnon · 2025-07-30T14:02:30Z

llvm/lib/Transforms/Vectorize/VPlanTransforms.h

-      VPlan &Plan, Type *InductionTy, PredicatedScalarEvolution &PSE,
-      bool RequiresScalarEpilogueCheck, bool TailFolded, Loop *TheLoop,
-      DebugLoc IVDL, bool HasUncountableExit, VFRange &Range);
+  /// block


Suggested change

/// block

/// block.

done thanks

…rvectorization

artagnon

LGTM, thanks!

ayalz

Post approval review.

ayalz · 2025-07-31T06:08:18Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+  // Create initial VPlan skeleton, having a basic block for the pre-header
+  // which contains SCEV expansions that need to happen before the CFG is
+  // modified; a basic block for the vector pre-header, followed by a region for
+  // the vector loop, followed by the middle basic block, connecting to the
+  // scalar preheader and exit blcoks.


Better to place the above documentation next to the definition of addInitialSkeleton()?

Here it would be good to explain that a "base" VPlan0 is built to serve as a common generic starting point for all candidates built later for specific VF ranges.

Updated, thanks

ayalz · 2025-07-31T06:15:15Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+  VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(
+      *Plan, RequiresScalarEpilogueCheck, CM.foldTailByMasking(),


Potential follow-up: would be good to complete the common skeleton with a canonical representation of early exits and middle check, to be specialized per each VF range according to its RequiresScalarEpilogueCheck (fold tail is currently also common?).

Yep taht sounds good!

ayalz · 2025-07-31T06:22:42Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

+      *Plan, Legal->getWidestInductionType(),
+      getDebugLocFromInstOrOperands(Legal->getPrimaryInduction()), PSE,
+      OrigLoop);
+  VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(*Plan, true, false, false,


+1

Suggested change

VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(*Plan, true, false, false,

VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(*Plan, true /* requires scalar epilog check */, false /*fold tail by masking */, false /*has uncountable exit */,

ayalz · 2025-07-31T06:26:08Z

llvm/unittests/Transforms/Vectorize/VPlanTestBase.h

+    VPlanTransforms::addInitialSkeleton(*Plan, IntegerType::get(*Ctx, 64), {},
+                                        PSE, L);


addInitialSkeleton() complements buildPlainCFG() in constructing the common VF-agnostic VPlan0 base, so best appear together, if not fused:

Suggested change

VPlanTransforms::addInitialSkeleton(*Plan, IntegerType::get(*Ctx, 64), {},

PSE, L);

VPlanTransforms::addInitialSkeleton(*Plan, IntegerType::get(*Ctx, 64), {},

PSE, L);

Range R(ElementCount::getFixed(1), ElementCount::getFixed(2));

done thanks

ayalz · 2025-07-31T06:28:47Z

llvm/lib/Transforms/Vectorize/VPlanTransforms.h

-      bool RequiresScalarEpilogueCheck, bool TailFolded, Loop *TheLoop,
-      DebugLoc IVDL, bool HasUncountableExit, VFRange &Range);
+  /// block
+  LLVM_ABI_FOR_TEST static void handleEarlyExitsAndAddMiddleCheck(


LLVM_ABI_FOR_TEST?

this has been a recent addition, and now all symbols not to be exported to the public LLVM API but needed for unit tests need to be marked as such. It looks like this was for better supporting shared libraries with symbols hidden by default: 4e2efa5

ayalz · 2025-07-31T19:09:24Z

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

-      *Plan, Legal->getWidestInductionType(), PSE, true, false, OrigLoop,
-      getDebugLocFromInstOrOperands(Legal->getPrimaryInduction()), false,
-      Range);
+


Suggested change

Should buildPlainCFG() be called buildPlainLoopCFG() or buildPlainCFGForLoop(), as it focuses on building the basic blocks of the loop along with its preheader and exit. buildInitialSkeleton()also builds plain CFG, can be calledconnectPlainLoopCFG(), as it adds new basic blocks before and after the loop connecting it(?). Should a new buildSkeletalPlan()callbuildPlainCFG()followed byaddInitialSkeleton()`? Together they build an initial CFG-based skeleton of VPlan.

combined into buildVPlan0, wdyt?

I tried to combine the comments as well as possibe

ayalz · 2025-08-03T16:13:52Z

llvm/lib/Transforms/Vectorize/VPlanTransforms.h

-  /// VPBasicBlock for the vector pre-header as well as a VPBasicBlock as exit
-  /// block of the main vector loop (middle.block). If a check is needed to
+  /// VPBasicBlock for the vector pre-header, a VPBasicBlock as exit
+  /// block of the main vector loop (middle.block) and a VPBaiscBlock for the


Suggested change

/// block of the main vector loop (middle.block) and a VPBaiscBlock for the

/// block of the main vector loop (middle.block) and a VPBasicBlock for the

That should be fixed in the latest version, thanks

ayalz · 2025-08-03T16:16:09Z

llvm/lib/Transforms/Vectorize/VPlanTransforms.h

+  /// scalar preheader. It also adds a canonical IV and its increment, using \p
+  /// InductionTy and \p IVDL, and creates a VPValue expression for the original
+  /// trip count.


Should this "also" part that handles values and recipes rather than CFG blocks and edges be handled by a separate function?

ayalz · 2025-08-03T16:24:24Z

llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp

+  auto *MiddleVPBB = cast<VPBasicBlock>(
+      Plan.getScalarHeader()->getSinglePredecessor()->getPredecessors()[0]);
+  VPBlockBase *HeaderVPB =
+      Plan.getEntry()->getSuccessors()[1]->getSingleSuccessor();


These seem somewhat obscure and fragile. Perhaps better fuse into a common canonical skeleton form.

Agreed, next step would be to move this forward also.

ayalz · 2025-08-03T16:26:23Z

llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp

+  Plan.getEntry()->swapSuccessors();
+}
+
+void VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(


Splitting middle block is one part of handling certain early exits.

Suggested change

void VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(

void VPlanTransforms::handleEarlyExits(

This currently also handled connecting the exit block/scalar ph.

Once the PR lands we could default to creating the branch and then replacing it with kown-true/false depending on tail-folding/requireScalarEpilogue.

Better split into separate handleEarlyExits() and addMiddleBranch()?
(Former is unneeded for native, although perhaps better to keep aligned.)

Split up thanks.

…rvectorization

ayalz

LGTM, thanks! Adding some minor suggestions.

ayalz · 2025-08-05T08:00:14Z

llvm/lib/Transforms/Vectorize/VPlanTransforms.h

+  /// \p TheLoop being directly translated to VPBasicBlocks with VPInstruction
+  /// corresponding to the input IR.
+  ///
+  /// The created loop is wrapped into an initial skeleton to facilitate


Suggested change

/// The created loop is wrapped into an initial skeleton to facilitate

/// The created loop is wrapped in an initial skeleton to facilitate

done thanks

ayalz · 2025-08-05T08:00:25Z

llvm/lib/Transforms/Vectorize/VPlanTransforms.h

+  /// corresponding to the input IR.
+  ///
+  /// The created loop is wrapped into an initial skeleton to facilitate
+  /// vectorization, consisting of a vector pre-header, a exit block for the


Suggested change

/// vectorization, consisting of a vector pre-header, a exit block for the

/// vectorization, consisting of a vector pre-header, an exit block for the

done thanks

ayalz · 2025-08-05T08:19:39Z

llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp

+  Plan.getEntry()->swapSuccessors();
+}
+
+void VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(


Better split into separate handleEarlyExits() and addMiddleBranch()?
(Former is unneeded for native, although perhaps better to keep aligned.)

ayalz · 2025-08-05T08:27:11Z

llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp

-  // check.
-  VPBlockUtils::connectBlocks(Plan.getEntry(), ScalarPH);
-  Plan.getEntry()->swapSuccessors();
-
  // If MiddleVPBB has a single successor then the original loop does not exit
  // via the latch and the single successor must be the scalar preheader.


Independent/follow-up: this refers to how LV handles loops whose latch does not exit, i.e., by requiring a scalar epilogue. Better handle all such cases consistently - either by wiring middle block to scalar preheader only as here, or by case 1 below.

Yep, this should be taken care of by consistently emitting a check if VectorTC == TC. This is the most general, the others are optimizations of cases where we know the check is either always false or true.

ayalz · 2025-08-05T08:32:16Z

llvm/lib/Transforms/Vectorize/VPlanTransforms.h

-      bool RequiresScalarEpilogueCheck, bool TailFolded, Loop *TheLoop,
-      DebugLoc IVDL, bool HasUncountableExit, VFRange &Range);
+  /// block
+  LLVM_ABI_FOR_TEST static void handleEarlyExitsAndAddMiddleCheck(


…rvectorization

…(#150848) Split up the not clearly named prepareForVectorization transform into buildVPlan0, which adds the vector preheader, middle and scalar preheader blocks, as well as the canonical induction recipes and sets the trip count. The new transform is run directly after building the plain CFG VPlan initially. The remaining code handling early exits and adding the branch in the middle block is renamed to handleEarlyExitsAndAddMiddleCheck and still runs at the original position. With the code movement, we only have to add the skeleton once to the initial VPlan, and cloning will take care of the rest. It will also enable moving other construction steps to work directly on VPlan0, like adding resume phis. PR: llvm/llvm-project#150848

fhahn requested review from rengolin, ayalz and aniragil July 27, 2025 19:20

llvmbot added vectorizers llvm:transforms labels Jul 27, 2025

artagnon reviewed Jul 30, 2025

View reviewed changes

fhahn added 2 commits July 31, 2025 11:57

Merge remote-tracking branch 'origin/main' into vplan-split-preparefo…

f835229

…rvectorization

!fixup address comments, thanks

9558817

artagnon approved these changes Jul 31, 2025

View reviewed changes

ayalz reviewed Aug 3, 2025

View reviewed changes

fhahn added 2 commits August 4, 2025 20:09

Merge remote-tracking branch 'origin/main' into vplan-split-preparefo…

3b51594

…rvectorization

!fixup address comments, thanks

d2a7fce

ayalz approved these changes Aug 5, 2025

View reviewed changes

fhahn added 3 commits August 5, 2025 14:05

Merge remote-tracking branch 'origin/main' into vplan-split-preparefo…

7a79e1c

…rvectorization

!fixup address comments, thanks

b28bb32

Merge branch 'main' into vplan-split-prepareforvectorization

032d726

fhahn merged commit 06fd0f9 into llvm:main Aug 9, 2025
9 checks passed

		VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(*Plan, true, false, false,
		Range);

		assert(MiddleVPBB->getSingleSuccessor() == ScalarPH &&
		"must have ScalarPH as single successor");

		VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(
		*Plan, RequiresScalarEpilogueCheck, CM.foldTailByMasking(),

		VPlanTransforms::addInitialSkeleton(Plan, IntegerType::get(Ctx, 64), {},
		PSE, L);

	/// block of the main vector loop (middle.block) and a VPBaiscBlock for the
	/// block of the main vector loop (middle.block) and a VPBasicBlock for the

	void VPlanTransforms::handleEarlyExitsAndAddMiddleCheck(
	void VPlanTransforms::handleEarlyExits(

	/// The created loop is wrapped into an initial skeleton to facilitate
	/// The created loop is wrapped in an initial skeleton to facilitate

	/// vectorization, consisting of a vector pre-header, a exit block for the
	/// vectorization, consisting of a vector pre-header, an exit block for the

[VPlan] Move initial skeleton construction earlier (NFC). #150848

[VPlan] Move initial skeleton construction earlier (NFC). #150848

Conversation

fhahn commented Jul 27, 2025

Uh oh!

llvmbot commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

artagnon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

artagnon left a comment

Choose a reason for hiding this comment

Uh oh!

ayalz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ayalz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

llvmbot commented Jul 27, 2025 •

edited

Loading