Skip to content

[LV] Consider EVL legality for TTI tail folding preference #144790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 19, 2025
Merged

[LV] Consider EVL legality for TTI tail folding preference #144790

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fe476ab
[LV] Consider EVL legality for TTI tail folding preference
preames Jun 18, 2025
6c24b06
Merge branch 'main' into pr-lv-evl-check-after-tti
preames Jun 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 20 additions & 23 deletions llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1364,16 +1364,14 @@ class LoopVectorizationCostModel {
return;
}

if (!ForceTailFoldingStyle.getNumOccurrences()) {
ChosenTailFoldingStyle = {
TTI.getPreferredTailFoldingStyle(/*IVUpdateMayOverflow=*/true),
TTI.getPreferredTailFoldingStyle(/*IVUpdateMayOverflow=*/false)};
return;
}
// Default to TTI preference, but allow command line override.
ChosenTailFoldingStyle = {
TTI.getPreferredTailFoldingStyle(/*IVUpdateMayOverflow=*/true),
TTI.getPreferredTailFoldingStyle(/*IVUpdateMayOverflow=*/false)};
Comment on lines +1368 to +1370
Copy link
Member

@alexey-bataev alexey-bataev Jun 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need a test for this change, I assume

Copy link
Collaborator Author

@preames preames Jun 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no way to test this with in tree backends.

if (ForceTailFoldingStyle.getNumOccurrences())
ChosenTailFoldingStyle = {ForceTailFoldingStyle.getValue(),
ForceTailFoldingStyle.getValue()};

// Set styles when forced.
ChosenTailFoldingStyle = {ForceTailFoldingStyle.getValue(),
ForceTailFoldingStyle.getValue()};
if (ForceTailFoldingStyle != TailFoldingStyle::DataWithEVL)
return;
// Override forced styles if needed.
Expand All @@ -1382,20 +1380,19 @@ class LoopVectorizationCostModel {
bool EVLIsLegal = UserIC <= 1 && IsScalableVF &&
TTI.hasActiveVectorLength(0, nullptr, Align()) &&
!EnableVPlanNativePath;
if (!EVLIsLegal) {
// If for some reason EVL mode is unsupported, fallback to
// DataWithoutLaneMask to try to vectorize the loop with folded tail
// in a generic way.
ChosenTailFoldingStyle = {TailFoldingStyle::DataWithoutLaneMask,
TailFoldingStyle::DataWithoutLaneMask};
LLVM_DEBUG(
dbgs()
<< "LV: Preference for VP intrinsics indicated. Will "
"not try to generate VP Intrinsics "
<< (UserIC > 1
? "since interleave count specified is greater than 1.\n"
: "due to non-interleaving reasons.\n"));
}
if (EVLIsLegal)
return;
// If for some reason EVL mode is unsupported, fallback to
// DataWithoutLaneMask to try to vectorize the loop with folded tail
// in a generic way.
ChosenTailFoldingStyle = {TailFoldingStyle::DataWithoutLaneMask,
TailFoldingStyle::DataWithoutLaneMask};
LLVM_DEBUG(
dbgs() << "LV: Preference for VP intrinsics indicated. Will "
"not try to generate VP Intrinsics "
<< (UserIC > 1
? "since interleave count specified is greater than 1.\n"
: "due to non-interleaving reasons.\n"));
}

/// Returns true if all loop blocks should be masked to fold tail loop.
Expand Down