Skip to content

Commit ed52bdd

Browse files
authored
[VPlan] Get Addr computation cost with scalar type if it is uniform for gather/scatter. (NFC) (#150371)
This patch query `getAddressComputationCost()` with scalar type if the address is uniform. This can help the cost for gather/scatter more accurate. In current LV, non consecutive VPWidenMemoryRecipe (gather/scatter) will account the cost of address computation. But there are some cases that the address is uniform across all lanes, that makes the address can be calculated with scalar type and broadcast. I have a followup optimization that tries to convert gather/scatter with uniform memory access to scalar load/store + broadcast (and select if needed). With this optimization, we can remove this temporary change. This patch is preparation for #149955 to prevent regressions.
1 parent 9db7e8d commit ed52bdd

File tree

1 file changed

+9
-2
lines changed

1 file changed

+9
-2
lines changed

llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3200,10 +3200,17 @@ InstructionCost VPWidenMemoryRecipe::computeCost(ElementCount VF,
32003200
// TODO: Using the original IR may not be accurate.
32013201
// Currently, ARM will use the underlying IR to calculate gather/scatter
32023202
// instruction cost.
3203-
const Value *Ptr = getLoadStorePointerOperand(&Ingredient);
3204-
Type *PtrTy = toVectorTy(Ptr->getType(), VF);
32053203
assert(!Reverse &&
32063204
"Inconsecutive memory access should not have the order.");
3205+
3206+
const Value *Ptr = getLoadStorePointerOperand(&Ingredient);
3207+
Type *PtrTy = Ptr->getType();
3208+
3209+
// If the address value is uniform across all lanes, then the address can be
3210+
// calculated with scalar type and broadcast.
3211+
if (!vputils::isSingleScalar(getAddr()))
3212+
PtrTy = toVectorTy(PtrTy, VF);
3213+
32073214
return Ctx.TTI.getAddressComputationCost(PtrTy, nullptr, nullptr,
32083215
Ctx.CostKind) +
32093216
Ctx.TTI.getGatherScatterOpCost(Opcode, Ty, Ptr, IsMasked, Alignment,

0 commit comments

Comments
 (0)