Skip to content

[NVPTX] Reland mad.wide combine under (default off) CLI option #160214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 25, 2025
Merged

[NVPTX] Reland mad.wide combine under (default off) CLI option #160214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7f9deca
Initial commit
justinfargnoli Sep 22, 2025
8e64c00
clang-format
justinfargnoli Sep 22, 2025
1b1e6ef
Update comment
justinfargnoli Sep 22, 2025
b32c6ee
Update test
justinfargnoli Sep 23, 2025
204513f
Add FIXME
justinfargnoli Sep 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions llvm/lib/Target/NVPTX/NVPTXISelDAGToDAG.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,13 @@ static cl::opt<bool>
EnableRsqrtOpt("nvptx-rsqrt-approx-opt", cl::init(true), cl::Hidden,
cl::desc("Enable reciprocal sqrt optimization"));

// FIXME: This is a WAR to recover lost performance from #155024.
// We still need to investigate the regression and find a more permanent
// solution.
static cl::opt<bool> EnableMADWide("nvptx-mad-wide-opt", cl::init(false),
cl::Hidden,
cl::desc("Enable MAD wide optimization"));

/// createNVPTXISelDag - This pass converts a legalized DAG into a
/// NVPTX-specific DAG, ready for instruction scheduling.
FunctionPass *llvm::createNVPTXISelDag(NVPTXTargetMachine &TM,
Expand Down Expand Up @@ -84,6 +91,8 @@ bool NVPTXDAGToDAGISel::allowFMA() const {

bool NVPTXDAGToDAGISel::doRsqrtOpt() const { return EnableRsqrtOpt; }

bool NVPTXDAGToDAGISel::doMADWideOpt() const { return EnableMADWide; }

/// Select - Select instructions not customized! Used for
/// expanded, promoted and normal instructions.
void NVPTXDAGToDAGISel::Select(SDNode *N) {
Expand Down
1 change: 1 addition & 0 deletions llvm/lib/Target/NVPTX/NVPTXISelDAGToDAG.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ class LLVM_LIBRARY_VISIBILITY NVPTXDAGToDAGISel : public SelectionDAGISel {
bool useF32FTZ() const;
bool allowFMA() const;
bool doRsqrtOpt() const;
bool doMADWideOpt() const;

NVPTXScopes Scopes{};

Expand Down
10 changes: 9 additions & 1 deletion llvm/lib/Target/NVPTX/NVPTXInstrInfo.td
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ def hasArchAccelFeatures : Predicate<"Subtarget->hasArchAccelFeatures()">;
def doF32FTZ : Predicate<"useF32FTZ()">;
def doNoF32FTZ : Predicate<"!useF32FTZ()">;
def doRsqrtOpt : Predicate<"doRsqrtOpt()">;
def doMADWideOpt : Predicate<"doMADWideOpt()">;

def hasHWROT32 : Predicate<"Subtarget->hasHWROT32()">;
def noHWROT32 : Predicate<"!Subtarget->hasHWROT32()">;
Expand Down Expand Up @@ -899,8 +900,15 @@ let Predicates = [hasOptEnabled] in {
defm MAD_LO_S32 : MADInst<"lo.s32", mul, I32RT, I32RT>;
defm MAD_LO_S64 : MADInst<"lo.s64", mul, I64RT, I64RT>;

// Generating mad.wide causes a regression:
// Generating mad.wide causes a regression in some cases:
// https://github.com/llvm/llvm-project/pull/150477#issuecomment-3191367837
// Only do so when the user requests it.
let Predicates = [doMADWideOpt] in {
defm MAD_WIDE_U16 : MADInst<"wide.u16", umul_wide, I32RT, I16RT>;
defm MAD_WIDE_S16 : MADInst<"wide.s16", smul_wide, I32RT, I16RT>;
defm MAD_WIDE_U32 : MADInst<"wide.u32", umul_wide, I64RT, I32RT>;
defm MAD_WIDE_S32 : MADInst<"wide.s32", smul_wide, I64RT, I32RT>;
}
}

//-----------------------------------
Expand Down
Loading