Skip to content

Commit e172110

Browse files
authored
[LV] Don't calculate scalar costs for scalable VFs in setVectorizedCallDecision (#152713)
In setVectorizedCallDecision we attempt to calculate the scalar costs for vectorisation calls, even for scalable VFs where we already know the answer is Invalid. We can avoid doing unnecessary work by skipping this completely for scalable vectors.
1 parent 694a488 commit e172110

File tree

1 file changed

+15
-7
lines changed

1 file changed

+15
-7
lines changed

llvm/lib/Transforms/Vectorize/LoopVectorize.cpp

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5832,14 +5832,22 @@ void LoopVectorizationCostModel::setVectorizedCallDecision(ElementCount VF) {
58325832
// assumed to be vectors, so we need to extract individual elements from
58335833
// there, execute VF scalar calls, and then gather the result into the
58345834
// vector return value.
5835-
InstructionCost ScalarCallCost =
5836-
TTI.getCallInstrCost(ScalarFunc, ScalarRetTy, ScalarTys, CostKind);
5837-
5838-
// Compute costs of unpacking argument values for the scalar calls and
5839-
// packing the return values to a vector.
5840-
InstructionCost ScalarizationCost = getScalarizationOverhead(CI, VF);
5835+
if (VF.isFixed()) {
5836+
InstructionCost ScalarCallCost =
5837+
TTI.getCallInstrCost(ScalarFunc, ScalarRetTy, ScalarTys, CostKind);
5838+
5839+
// Compute costs of unpacking argument values for the scalar calls and
5840+
// packing the return values to a vector.
5841+
InstructionCost ScalarizationCost = getScalarizationOverhead(CI, VF);
5842+
ScalarCost = ScalarCallCost * VF.getKnownMinValue() + ScalarizationCost;
5843+
} else {
5844+
// There is no point attempting to calculate the scalar cost for a
5845+
// scalable VF as we know it will be Invalid.
5846+
assert(!getScalarizationOverhead(CI, VF).isValid() &&
5847+
"Unexpected valid cost for scalarizing scalable vectors");
5848+
ScalarCost = InstructionCost::getInvalid();
5849+
}
58415850

5842-
ScalarCost = ScalarCallCost * VF.getKnownMinValue() + ScalarizationCost;
58435851
// Honor ForcedScalars and UniformAfterVectorization decisions.
58445852
// TODO: For calls, it might still be more profitable to widen. Use
58455853
// VPlan-based cost model to compare different options.

0 commit comments

Comments
 (0)