[FPEnv][SDAG] Implement FNEARBYINT with optional chain #163081
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change replaces the DAG node STRICT_FNEARBYINT with a modified FNEARBYINT, which uses optional chain property. The modified node can be used in both strictfp and default environments. This approach is based on the assumption that a floating-point operation is fundamentally the same in strictfp and default environments and is therefore lowered in almost identical manner. Indeed, all targets but one lower STRICT_FNEARBYINT using the same action as for FNEARBYINT. The only exception is PowerPC: it lowers STRICT_FNEARBYINT for vector types using `Expand`, even though FNEARBYINT for v4f32 and v2f64 is legal. This change implements the lowering uniformly, treating these vector types as legal for PowePC target. The change demonstrate the transition from using separate nodes for strictfp and default environments to using a single node with an optional chain. It also modifies some methods of DAG functions required to support such nodes.
spavloff
requested review from
andykaylor,
arsenm,
ecnelises,
efriedma-quic,
jcranmer-intel,
kpneal,
nemanjai and
uweigand
llvmbot
added
backend:AArch64
backend:RISC-V
backend:PowerPC
backend:SystemZ
backend:X86
llvm:codegen
llvm:SelectionDAG
|
@llvm/pr-subscribers-llvm-selectiondag @llvm/pr-subscribers-backend-risc-v Author: Serge Pavlov (spavloff) ChangesThis change replaces the DAG node STRICT_FNEARBYINT with a modified FNEARBYINT, which uses optional chain property. The modified node can be used in both strictfp and default environments. This approach is based on the assumption that a floating-point operation is fundamentally the same in strictfp and default environments and is therefore lowered in almost identical manner. Indeed, all targets but one lower STRICT_FNEARBYINT using the same action as for FNEARBYINT. The only exception is PowerPC: it lowers STRICT_FNEARBYINT for vector types using The change demonstrate the transition from using separate nodes for strictfp and default environments to using a single node with an optional chain. It also modifies some methods of DAG functions required to support such nodes. Patch is 64.53 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/163081.diff 28 Files Affected:
diff --git a/llvm/include/llvm/CodeGen/ISDOpcodes.h b/llvm/include/llvm/CodeGen/ISDOpcodes.h
index c76c83d84b3c7..e05a67ae0655f 100644
--- a/llvm/include/llvm/CodeGen/ISDOpcodes.h
+++ b/llvm/include/llvm/CodeGen/ISDOpcodes.h
@@ -448,7 +448,6 @@ enum NodeType {
STRICT_FLOG10,
STRICT_FLOG2,
STRICT_FRINT,
- STRICT_FNEARBYINT,
STRICT_FMAXNUM,
STRICT_FMINNUM,
STRICT_FCEIL,
diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index 73f2c55a71125..0d140ac745f67 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -1341,6 +1341,8 @@ class LLVM_ABI TargetLoweringBase {
unsigned EqOpc;
switch (Op) {
default: llvm_unreachable("Unexpected FP pseudo-opcode");
+#define FP_OPERATION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
+ case ISD::DAGN: EqOpc = ISD::DAGN; break;
#define DAG_INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
case ISD::STRICT_##DAGN: EqOpc = ISD::DAGN; break;
#define CMP_INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
diff --git a/llvm/include/llvm/IR/ConstrainedOps.def b/llvm/include/llvm/IR/ConstrainedOps.def
index 30a82bf633d57..76bed326a3e50 100644
--- a/llvm/include/llvm/IR/ConstrainedOps.def
+++ b/llvm/include/llvm/IR/ConstrainedOps.def
@@ -39,6 +39,12 @@
#define CMP_INSTRUCTION(N,A,R,I,D) DAG_INSTRUCTION(N,A,R,I,D)
#endif
+// FP_OPERATION is same as DAG_FUNCTION, but in DAG it is represented by the
+// same node, as non-constrained function.
+#ifndef FP_OPERATION
+#define FP_OPERATION(N,A,R,I,D) DAG_FUNCTION(N,A,R,I,D)
+#endif
+
// Arguments of the entries are:
// - instruction or intrinsic function name.
// - Number of original instruction/intrinsic arguments.
@@ -91,7 +97,7 @@ DAG_FUNCTION(maxnum, 2, 0, experimental_constrained_maxnum, FMAXNUM
DAG_FUNCTION(minnum, 2, 0, experimental_constrained_minnum, FMINNUM)
DAG_FUNCTION(maximum, 2, 0, experimental_constrained_maximum, FMAXIMUM)
DAG_FUNCTION(minimum, 2, 0, experimental_constrained_minimum, FMINIMUM)
-DAG_FUNCTION(nearbyint, 1, 1, experimental_constrained_nearbyint, FNEARBYINT)
+FP_OPERATION(nearbyint, 1, 1, experimental_constrained_nearbyint, FNEARBYINT)
DAG_FUNCTION(pow, 2, 1, experimental_constrained_pow, FPOW)
DAG_FUNCTION(powi, 2, 1, experimental_constrained_powi, FPOWI)
DAG_FUNCTION(ldexp, 2, 1, experimental_constrained_ldexp, FLDEXP)
@@ -114,3 +120,4 @@ FUNCTION(fmuladd, 3, 1, experimental_constrained_fmuladd)
#undef CMP_INSTRUCTION
#undef DAG_INSTRUCTION
#undef DAG_FUNCTION
+#undef FP_OPERATION
diff --git a/llvm/include/llvm/Target/TargetSelectionDAG.td b/llvm/include/llvm/Target/TargetSelectionDAG.td
index 632be7ad9e350..8edfb59455402 100644
--- a/llvm/include/llvm/Target/TargetSelectionDAG.td
+++ b/llvm/include/llvm/Target/TargetSelectionDAG.td
@@ -575,7 +575,7 @@ def frint : SDNode<"ISD::FRINT" , SDTFPUnaryOp>;
def ftrunc : SDNode<"ISD::FTRUNC" , SDTFPUnaryOp>;
def fceil : SDNode<"ISD::FCEIL" , SDTFPUnaryOp>;
def ffloor : SDNode<"ISD::FFLOOR" , SDTFPUnaryOp>;
-def fnearbyint : SDNode<"ISD::FNEARBYINT" , SDTFPUnaryOp>;
+def fnearbyint : SDNode<"ISD::FNEARBYINT" , SDTFPUnaryOp, [SDNPMayHaveChain]>;
def fround : SDNode<"ISD::FROUND" , SDTFPUnaryOp>;
def froundeven : SDNode<"ISD::FROUNDEVEN" , SDTFPUnaryOp>;
@@ -653,8 +653,6 @@ def strict_lrint : SDNode<"ISD::STRICT_LRINT",
SDTFPToIntOp, [SDNPHasChain]>;
def strict_llrint : SDNode<"ISD::STRICT_LLRINT",
SDTFPToIntOp, [SDNPHasChain]>;
-def strict_fnearbyint : SDNode<"ISD::STRICT_FNEARBYINT",
- SDTFPUnaryOp, [SDNPHasChain]>;
def strict_fceil : SDNode<"ISD::STRICT_FCEIL",
SDTFPUnaryOp, [SDNPHasChain]>;
def strict_ffloor : SDNode<"ISD::STRICT_FFLOOR",
@@ -1704,9 +1702,6 @@ def any_lrint : PatFrags<(ops node:$src),
def any_llrint : PatFrags<(ops node:$src),
[(strict_llrint node:$src),
(llrint node:$src)]>;
-def any_fnearbyint : PatFrags<(ops node:$src),
- [(strict_fnearbyint node:$src),
- (fnearbyint node:$src)]>;
def any_fceil : PatFrags<(ops node:$src),
[(strict_fceil node:$src),
(fceil node:$src)]>;
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
index 5fb7e63cfb605..72fc5a7570ce7 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp
@@ -2196,7 +2196,7 @@ void SelectionDAGLegalize::ExpandFPLibCall(SDNode* Node,
if (LC == RTLIB::UNKNOWN_LIBCALL)
llvm_unreachable("Can't create an unknown libcall!");
- if (Node->isStrictFPOpcode()) {
+ if (Node->isStrictFPOpcode() || (Node->hasChain() && Node->isFPOperation())) {
EVT RetVT = Node->getValueType(0);
SmallVector<SDValue, 4> Ops(drop_begin(Node->ops()));
TargetLowering::MakeLibCallOptions CallOptions;
@@ -4791,7 +4791,6 @@ void SelectionDAGLegalize::ConvertNodeToLibcall(SDNode *Node) {
RTLIB::RINT_PPCF128, Results);
break;
case ISD::FNEARBYINT:
- case ISD::STRICT_FNEARBYINT:
ExpandFPLibCall(Node, RTLIB::NEARBYINT_F32,
RTLIB::NEARBYINT_F64,
RTLIB::NEARBYINT_F80,
@@ -5760,7 +5759,6 @@ void SelectionDAGLegalize::PromoteNode(SDNode *Node) {
case ISD::FFLOOR:
case ISD::FCEIL:
case ISD::FRINT:
- case ISD::FNEARBYINT:
case ISD::FROUND:
case ISD::FROUNDEVEN:
case ISD::FTRUNC:
@@ -5792,7 +5790,6 @@ void SelectionDAGLegalize::PromoteNode(SDNode *Node) {
case ISD::STRICT_FFLOOR:
case ISD::STRICT_FCEIL:
case ISD::STRICT_FRINT:
- case ISD::STRICT_FNEARBYINT:
case ISD::STRICT_FROUND:
case ISD::STRICT_FROUNDEVEN:
case ISD::STRICT_FTRUNC:
@@ -5821,6 +5818,25 @@ void SelectionDAGLegalize::PromoteNode(SDNode *Node) {
Results.push_back(Tmp3);
Results.push_back(Tmp3.getValue(1));
break;
+ case ISD::FNEARBYINT:
+ if (Node->hasChain()) {
+ Tmp1 = DAG.getNode(ISD::STRICT_FP_EXTEND, dl, {NVT, MVT::Other},
+ {Node->getOperand(0), Node->getOperand(1)});
+ Tmp2 = DAG.getNode(Node->getOpcode(), dl, {NVT, MVT::Other},
+ {Tmp1.getValue(1), Tmp1});
+ Tmp3 = DAG.getNode(ISD::STRICT_FP_ROUND, dl, {OVT, MVT::Other},
+ {Tmp2.getValue(1), Tmp2,
+ DAG.getIntPtrConstant(0, dl, /*isTarget=*/true)});
+ Results.push_back(Tmp3);
+ Results.push_back(Tmp3.getValue(1));
+ } else {
+ Tmp1 = DAG.getNode(ISD::FP_EXTEND, dl, NVT, Node->getOperand(0));
+ Tmp2 = DAG.getNode(Node->getOpcode(), dl, NVT, Tmp1);
+ Results.push_back(
+ DAG.getNode(ISD::FP_ROUND, dl, OVT, Tmp2,
+ DAG.getIntPtrConstant(0, dl, /*isTarget=*/true)));
+ }
+ break;
case ISD::BUILD_VECTOR: {
MVT EltVT = OVT.getVectorElementType();
MVT NewEltVT = NVT.getVectorElementType();
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
index 437d0f4654096..33d133ff9cba5 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
@@ -115,7 +115,6 @@ void DAGTypeLegalizer::SoftenFloatResult(SDNode *N, unsigned ResNo) {
case ISD::FMA: R = SoftenFloatRes_FMA(N); break;
case ISD::STRICT_FMUL:
case ISD::FMUL: R = SoftenFloatRes_FMUL(N); break;
- case ISD::STRICT_FNEARBYINT:
case ISD::FNEARBYINT: R = SoftenFloatRes_FNEARBYINT(N); break;
case ISD::FNEG: R = SoftenFloatRes_FNEG(N); break;
case ISD::STRICT_FP_EXTEND:
@@ -227,6 +226,32 @@ SDValue DAGTypeLegalizer::SoftenFloatRes_Binary(SDNode *N, RTLIB::Libcall LC) {
return Tmp.first;
}
+SDValue DAGTypeLegalizer::SoftenFloatRes_FPOperation(SDNode *N,
+ RTLIB::Libcall LC) {
+ bool HasChain = N->hasChain();
+ assert(N->getNumValues() == 1 + HasChain &&
+ "multiple result is not supported yet");
+ SDValue Chain = HasChain ? N->getOperand(0) : SDValue();
+ SmallVector<SDValue, 4> Ops;
+ SmallVector<EVT, 4> OpsVT;
+
+ for (unsigned i = HasChain, e = N->getNumOperands(); i != e; ++i) {
+ SDValue Op = N->getOperand(i);
+ OpsVT.push_back(Op.getValueType());
+ Op = GetSoftenedFloat(Op);
+ Ops.push_back(Op);
+ }
+
+ EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+ TargetLowering::MakeLibCallOptions CallOptions;
+ CallOptions.setTypeListBeforeSoften(OpsVT, N->getValueType(0));
+ std::pair<SDValue, SDValue> Tmp =
+ TLI.makeLibCall(DAG, LC, NVT, Ops, CallOptions, SDLoc(N), Chain);
+ if (HasChain)
+ ReplaceValueWith(SDValue(N, 1), Tmp.second);
+ return Tmp.first;
+}
+
SDValue DAGTypeLegalizer::SoftenFloatRes_BITCAST(SDNode *N) {
return BitConvertToInteger(N->getOperand(0));
}
@@ -582,7 +607,7 @@ SDValue DAGTypeLegalizer::SoftenFloatRes_FMUL(SDNode *N) {
}
SDValue DAGTypeLegalizer::SoftenFloatRes_FNEARBYINT(SDNode *N) {
- return SoftenFloatRes_Unary(N, GetFPLibCall(N->getValueType(0),
+ return SoftenFloatRes_FPOperation(N, GetFPLibCall(N->getValueType(0),
RTLIB::NEARBYINT_F32,
RTLIB::NEARBYINT_F64,
RTLIB::NEARBYINT_F80,
@@ -1596,7 +1621,6 @@ void DAGTypeLegalizer::ExpandFloatResult(SDNode *N, unsigned ResNo) {
case ISD::FMA: ExpandFloatRes_FMA(N, Lo, Hi); break;
case ISD::STRICT_FMUL:
case ISD::FMUL: ExpandFloatRes_FMUL(N, Lo, Hi); break;
- case ISD::STRICT_FNEARBYINT:
case ISD::FNEARBYINT: ExpandFloatRes_FNEARBYINT(N, Lo, Hi); break;
case ISD::FNEG: ExpandFloatRes_FNEG(N, Lo, Hi); break;
case ISD::STRICT_FP_EXTEND:
@@ -1688,6 +1712,21 @@ void DAGTypeLegalizer::ExpandFloatRes_Binary(SDNode *N, RTLIB::Libcall LC,
GetPairElements(Tmp.first, Lo, Hi);
}
+void DAGTypeLegalizer::ExpandFloatRes_FPOperation(SDNode *N, RTLIB::Libcall LC,
+ SDValue &Lo, SDValue &Hi) {
+ bool HasChain = N->hasChain();
+ SDValue Chain = HasChain ? N->getOperand(0) : SDValue();
+ assert(N->getNumValues() == 1 + HasChain &&
+ "multiple result is not supported yet");
+ SmallVector<SDValue, 4> Ops(HasChain ? llvm::drop_begin(N->ops()) : N->ops());
+ TargetLowering::MakeLibCallOptions CallOptions;
+ std::pair<SDValue, SDValue> Tmp = TLI.makeLibCall(
+ DAG, LC, N->getValueType(0), Ops, CallOptions, SDLoc(N), Chain);
+ if (HasChain)
+ ReplaceValueWith(SDValue(N, 1), Tmp.second);
+ GetPairElements(Tmp.first, Lo, Hi);
+}
+
void DAGTypeLegalizer::ExpandFloatRes_FMODF(SDNode *N) {
ExpandFloatRes_UnaryWithTwoFPResults(N, RTLIB::getMODF(N->getValueType(0)),
/*CallRetResNo=*/0);
@@ -1951,7 +1990,7 @@ void DAGTypeLegalizer::ExpandFloatRes_FMUL(SDNode *N, SDValue &Lo,
void DAGTypeLegalizer::ExpandFloatRes_FNEARBYINT(SDNode *N,
SDValue &Lo, SDValue &Hi) {
- ExpandFloatRes_Unary(N, GetFPLibCall(N->getValueType(0),
+ ExpandFloatRes_FPOperation(N, GetFPLibCall(N->getValueType(0),
RTLIB::NEARBYINT_F32,
RTLIB::NEARBYINT_F64,
RTLIB::NEARBYINT_F80,
@@ -2827,6 +2866,11 @@ void DAGTypeLegalizer::PromoteFloatResult(SDNode *N, unsigned ResNo) {
R = PromoteFloatRes_EXTRACT_VECTOR_ELT(N); break;
case ISD::FCOPYSIGN: R = PromoteFloatRes_FCOPYSIGN(N); break;
+ // Floating-point operations with optional chain.
+ case ISD::FNEARBYINT:
+ R = PromoteFloatRes_FPOperation(N);
+ break;
+
// Unary FP Operations
case ISD::FABS:
case ISD::FACOS:
@@ -2843,7 +2887,6 @@ void DAGTypeLegalizer::PromoteFloatResult(SDNode *N, unsigned ResNo) {
case ISD::FLOG:
case ISD::FLOG2:
case ISD::FLOG10:
- case ISD::FNEARBYINT:
case ISD::FNEG:
case ISD::FRINT:
case ISD::FROUND:
@@ -3071,6 +3114,29 @@ SDValue DAGTypeLegalizer::PromoteFloatRes_BinOp(SDNode *N) {
return DAG.getNode(N->getOpcode(), SDLoc(N), NVT, Op0, Op1, N->getFlags());
}
+SDValue DAGTypeLegalizer::PromoteFloatRes_FPOperation(SDNode *N) {
+ bool HasChain = N->hasChain();
+ SDValue Chain = HasChain ? N->getOperand(0) : SDValue();
+ assert(N->getNumValues() == 1 + HasChain &&
+ "multiple result is not supported yet");
+ SmallVector<SDValue, 4> Ops;
+
+ if (HasChain)
+ Ops.push_back(Chain);
+ for (unsigned i = HasChain, e = N->getNumOperands(); i != e; ++i) {
+ SDValue Op = N->getOperand(i);
+ // FIXME Use strict conversions for strict operations.
+ Op = GetPromotedFloat(Op);
+ Ops.push_back(Op);
+ }
+
+ EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
+ SDValue Res = DAG.getNode(N->getOpcode(), SDLoc(N), NVT, Ops);
+ if (HasChain)
+ ReplaceValueWith(SDValue(N, 1), Res.getValue(1));
+ return Res;
+}
+
SDValue DAGTypeLegalizer::PromoteFloatRes_FMAD(SDNode *N) {
EVT VT = N->getValueType(0);
EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), VT);
@@ -3312,6 +3378,11 @@ void DAGTypeLegalizer::SoftPromoteHalfResult(SDNode *N, unsigned ResNo) {
case ISD::STRICT_FP_ROUND:
case ISD::FP_ROUND: R = SoftPromoteHalfRes_FP_ROUND(N); break;
+ // Floating-point operations with optional chain.
+ case ISD::FNEARBYINT:
+ R = SoftPromoteHalfRes_FPOperation(N);
+ break;
+
// Unary FP Operations
case ISD::FACOS:
case ISD::FASIN:
@@ -3327,7 +3398,6 @@ void DAGTypeLegalizer::SoftPromoteHalfResult(SDNode *N, unsigned ResNo) {
case ISD::FLOG:
case ISD::FLOG2:
case ISD::FLOG10:
- case ISD::FNEARBYINT:
case ISD::FREEZE:
case ISD::FRINT:
case ISD::FROUND:
@@ -3714,6 +3784,38 @@ SDValue DAGTypeLegalizer::SoftPromoteHalfRes_BinOp(SDNode *N) {
return DAG.getNode(GetPromotionOpcode(NVT, OVT), dl, MVT::i16, Res);
}
+SDValue DAGTypeLegalizer::SoftPromoteHalfRes_FPOperation(SDNode *N) {
+ SDLoc dl(N);
+ bool HasChain = N->hasChain();
+ assert(N->getNumValues() == 1 + HasChain &&
+ "multiple result is not supported yet");
+ SDValue Chain = HasChain ? N->getOperand(0) : SDValue();
+ EVT OVT = N->getValueType(0);
+ EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), OVT);
+ auto PromotionOpcode = GetPromotionOpcode(OVT, NVT);
+
+ SmallVector<SDValue, 4> Ops;
+ if (HasChain)
+ Ops.push_back(Chain);
+ for (unsigned i = HasChain, e = N->getNumOperands(); i != e; ++i) {
+ SDValue Op = GetSoftPromotedHalf(N->getOperand(i));
+ // FIXME Use strict conversions for strict operations.
+ Op = DAG.getNode(PromotionOpcode, dl, NVT, Op);
+ Ops.push_back(Op);
+ }
+
+ SDValue Res = DAG.getNode(N->getOpcode(), SDLoc(N), NVT, Ops);
+ if (HasChain)
+ Chain = Res.getValue(1);
+
+ // Convert back to FP16 as an integer.
+ Res = DAG.getNode(GetPromotionOpcode(NVT, OVT), dl, MVT::i16, Res);
+
+ if (HasChain)
+ ReplaceValueWith(SDValue(N, 1), Chain);
+ return Res;
+}
+
SDValue DAGTypeLegalizer::SoftPromoteHalfRes_VECREDUCE(SDNode *N) {
// Expand and soften recursively.
ReplaceValueWith(SDValue(N, 0), TLI.expandVecReduce(N, DAG));
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h b/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
index 603dc34ce72a7..541977c7dad03 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h
@@ -573,6 +573,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
bool SoftenFloatRes_UnaryWithTwoFPResults(
SDNode *N, RTLIB::Libcall LC, std::optional<unsigned> CallRetResNo = {});
SDValue SoftenFloatRes_Binary(SDNode *N, RTLIB::Libcall LC);
+ SDValue SoftenFloatRes_FPOperation(SDNode *N, RTLIB::Libcall LC);
SDValue SoftenFloatRes_MERGE_VALUES(SDNode *N, unsigned ResNo);
SDValue SoftenFloatRes_ARITH_FENCE(SDNode *N);
SDValue SoftenFloatRes_BITCAST(SDNode *N);
@@ -681,6 +682,8 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue &Lo, SDValue &Hi);
void ExpandFloatRes_UnaryWithTwoFPResults(
SDNode *N, RTLIB::Libcall LC, std::optional<unsigned> CallRetResNo = {});
+ void ExpandFloatRes_FPOperation(SDNode *N, RTLIB::Libcall LC, SDValue &Lo,
+ SDValue &Hi);
// clang-format off
void ExpandFloatRes_AssertNoFPClass(SDNode *N, SDValue &Lo, SDValue &Hi);
@@ -788,6 +791,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue PromoteFloatRes_XINT_TO_FP(SDNode *N);
SDValue PromoteFloatRes_VECREDUCE(SDNode *N);
SDValue PromoteFloatRes_VECREDUCE_SEQ(SDNode *N);
+ SDValue PromoteFloatRes_FPOperation(SDNode *N);
bool PromoteFloatOperand(SDNode *N, unsigned OpNo);
SDValue PromoteFloatOp_BITCAST(SDNode *N, unsigned OpNo);
@@ -839,6 +843,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue SoftPromoteHalfRes_UNDEF(SDNode *N);
SDValue SoftPromoteHalfRes_VECREDUCE(SDNode *N);
SDValue SoftPromoteHalfRes_VECREDUCE_SEQ(SDNode *N);
+ SDValue SoftPromoteHalfRes_FPOperation(SDNode *N);
bool SoftPromoteHalfOperand(SDNode *N, unsigned OpNo);
SDValue SoftPromoteHalfOp_BITCAST(SDNode *N);
@@ -881,6 +886,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
SDValue ScalarizeVecRes_OverflowOp(SDNode *N, unsigned ResNo);
SDValue ScalarizeVecRes_InregOp(SDNode *N);
SDValue ScalarizeVecRes_VecInregOp(SDNode *N);
+ SDValue ScalarizeVecRes_FPOperation(SDNode *N);
SDValue ScalarizeVecRes_ADDRSPACECAST(SDNode *N);
SDValue ScalarizeVecRes_BITCAST(SDNode *N);
@@ -965,6 +971,7 @@ class LLVM_LIBRARY_VISIBILITY DAGTypeLegalizer {
void SplitVecRes_StrictFPOp(SDNode *N, SDValue &Lo, SDValue &Hi);
void SplitVecRes_OverflowOp(SDNode *N, unsigned ResNo,
SDValue &Lo, SDValue &Hi);
+ void SplitVecRes_FPOperation(SDNode *N, SDValue &Lo, SDValue &Hi);
void SplitVecRes_FIX(SDNode *N, SDValue &Lo, SDValue &Hi);
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp
index 8e423c4f83b38..3abc8594f3d17 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp
@@ -319,9 +319,14 @@ SDValue VectorLegalizer::LegalizeOp(SDValue Op) {
if (Action == TargetLowering::Legal)
Action = TargetLowering::Expand;
break;
+#define FP_OPERATION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) case ISD::DAGN:
#define DAG_INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
case ISD::STRICT_##DAGN:
#include "llvm/IR/ConstrainedOps.def"
+ if (!Node->hasChain()) {
+ Action = TLI.getOperationAction(Node->getOpcode(), Node->getValueType(0));
+ break;
+ }
ValVT = Node->getValueType(0);
if (Op.getOpcode() == ISD::STRICT_SINT_TO_FP ||
Op.getOpcode() == ISD::STRICT_UINT_TO_FP)
@@ -435,7 +440,6 @@ SDValue VectorLegalizer::LegalizeOp(SDValue Op) {
case ISD::FCEIL:
case ISD::FTRUNC:
case ISD::FRINT:
- case ISD::FNEARBYINT:
case ISD::FROUND:
case ISD::FROUNDEVEN:
case ISD::FFLOOR:
@@ -1218,11 +1222,20 @@ void VectorLegalizer::Expand(SDNode *Node, SmallVectorImpl<SDValue> &Results) {
case ISD::SDIVFIXSAT:
case ISD::UDIVFIXSAT:
break;
+#define FP_OPERATION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
+ case ISD::DAGN:
#define DAG_INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
case ISD::STRICT_##DAGN:
#include "llvm/IR/ConstrainedOps.def"
- ExpandStrictFPOp(Node, Results);
- return;
+ if (Node->hasChain()) {
+ ExpandStrictFPOp(Node, Results);
+ return;
+ }
+ if (SDValue Expanded = TLI.expandVectorNaryOpBySplitting(Node, DAG)) {
+ Results.push_back(Expanded);
+ return;
+ }
+ break;
case ISD::VECREDUCE_ADD:
...
[truncated]
|
You can test this locally with the following command:git-clang-format --diff origin/main HEAD --extensions h,cpp -- llvm/include/llvm/CodeGen/ISDOpcodes.h llvm/include/llvm/CodeGen/TargetLowering.h llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp llvm/lib/CodeGen/SelectionDAG/LegalizeTypes.h llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp llvm/lib/CodeGen/SelectionDAG/SelectionDAG.cpp llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp llvm/lib/CodeGen/SelectionDAG/SelectionDAGDumper.cpp llvm/lib/CodeGen/SelectionDAG/TargetLowering.cpp llvm/lib/CodeGen/TargetLoweringBase.cpp llvm/lib/Target/AArch64/AArch64ISelLowering.cpp llvm/lib/Target/PowerPC/PPCISelLowering.cpp llvm/lib/Target/RISCV/RISCVISelLowering.cpp llvm/lib/Target/SystemZ/SystemZISelLowering.cpp llvm/lib/Target/X86/X86ISelDAGToDAG.cpp llvm/lib/Target/X86/X86ISelLowering.cpp
View the diff from clang-format here.diff --git a/llvm/include/llvm/CodeGen/TargetLowering.h b/llvm/include/llvm/CodeGen/TargetLowering.h
index 0d140ac74..356cd434e 100644
--- a/llvm/include/llvm/CodeGen/TargetLowering.h
+++ b/llvm/include/llvm/CodeGen/TargetLowering.h
@@ -1342,7 +1342,9 @@ public:
switch (Op) {
default: llvm_unreachable("Unexpected FP pseudo-opcode");
#define FP_OPERATION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
- case ISD::DAGN: EqOpc = ISD::DAGN; break;
+ case ISD::DAGN: \
+ EqOpc = ISD::DAGN; \
+ break;
#define DAG_INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
case ISD::STRICT_##DAGN: EqOpc = ISD::DAGN; break;
#define CMP_INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
index 33d133ff9..13b7fabdd 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeFloatTypes.cpp
@@ -607,12 +607,10 @@ SDValue DAGTypeLegalizer::SoftenFloatRes_FMUL(SDNode *N) {
}
SDValue DAGTypeLegalizer::SoftenFloatRes_FNEARBYINT(SDNode *N) {
- return SoftenFloatRes_FPOperation(N, GetFPLibCall(N->getValueType(0),
- RTLIB::NEARBYINT_F32,
- RTLIB::NEARBYINT_F64,
- RTLIB::NEARBYINT_F80,
- RTLIB::NEARBYINT_F128,
- RTLIB::NEARBYINT_PPCF128));
+ return SoftenFloatRes_FPOperation(
+ N, GetFPLibCall(N->getValueType(0), RTLIB::NEARBYINT_F32,
+ RTLIB::NEARBYINT_F64, RTLIB::NEARBYINT_F80,
+ RTLIB::NEARBYINT_F128, RTLIB::NEARBYINT_PPCF128));
}
SDValue DAGTypeLegalizer::SoftenFloatRes_FNEG(SDNode *N) {
@@ -1990,12 +1988,12 @@ void DAGTypeLegalizer::ExpandFloatRes_FMUL(SDNode *N, SDValue &Lo,
void DAGTypeLegalizer::ExpandFloatRes_FNEARBYINT(SDNode *N,
SDValue &Lo, SDValue &Hi) {
- ExpandFloatRes_FPOperation(N, GetFPLibCall(N->getValueType(0),
- RTLIB::NEARBYINT_F32,
- RTLIB::NEARBYINT_F64,
- RTLIB::NEARBYINT_F80,
- RTLIB::NEARBYINT_F128,
- RTLIB::NEARBYINT_PPCF128), Lo, Hi);
+ ExpandFloatRes_FPOperation(
+ N,
+ GetFPLibCall(N->getValueType(0), RTLIB::NEARBYINT_F32,
+ RTLIB::NEARBYINT_F64, RTLIB::NEARBYINT_F80,
+ RTLIB::NEARBYINT_F128, RTLIB::NEARBYINT_PPCF128),
+ Lo, Hi);
}
void DAGTypeLegalizer::ExpandFloatRes_FNEG(SDNode *N, SDValue &Lo,
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp
index 3abc8594f..19e4f3067 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp
@@ -1222,8 +1222,7 @@ void VectorLegalizer::Expand(SDNode *Node, SmallVectorImpl<SDValue> &Results) {
case ISD::SDIVFIXSAT:
case ISD::UDIVFIXSAT:
break;
-#define FP_OPERATION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
- case ISD::DAGN:
+#define FP_OPERATION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) case ISD::DAGN:
#define DAG_INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
case ISD::STRICT_##DAGN:
#include "llvm/IR/ConstrainedOps.def"
diff --git a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
index b312396a8..56c679a87 100644
--- a/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
+++ b/llvm/lib/CodeGen/SelectionDAG/LegalizeVectorTypes.cpp
@@ -544,7 +544,7 @@ SDValue DAGTypeLegalizer::ScalarizeVecRes_FPOperation(SDNode *N) {
SDValue Result;
if (HasChain) {
Result = DAG.getNode(N->getOpcode(), DL, {DestVT, MVT::Other}, Ops,
- N->getFlags());
+ N->getFlags());
ReplaceValueWith(SDValue(N, 1), Result.getValue(1));
} else {
Result = DAG.getNode(N->getOpcode(), DL, DestVT, Ops, N->getFlags());
@@ -5071,7 +5071,6 @@ void DAGTypeLegalizer::WidenVectorResult(SDNode *N, unsigned ResNo) {
Res = WidenVecRes_BinaryWithExtraScalarOp(N);
break;
-
#define FP_OPERATION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) case ISD::DAGN:
#include "llvm/IR/ConstrainedOps.def"
if (N->hasChain())
diff --git a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
index a23c1d5d6..0442d9b67 100644
--- a/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
+++ b/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp
@@ -1407,9 +1407,8 @@ AArch64TargetLowering::AArch64TargetLowering(const TargetMachine &TM,
for (auto Op :
{ISD::FFLOOR, ISD::FNEARBYINT, ISD::FCEIL, ISD::FRINT, ISD::FTRUNC,
ISD::FROUND, ISD::FROUNDEVEN, ISD::FMAXNUM_IEEE, ISD::FMINNUM_IEEE,
- ISD::STRICT_FFLOOR, ISD::STRICT_FCEIL,
- ISD::STRICT_FRINT, ISD::STRICT_FTRUNC, ISD::STRICT_FROUND,
- ISD::STRICT_FROUNDEVEN}) {
+ ISD::STRICT_FFLOOR, ISD::STRICT_FCEIL, ISD::STRICT_FRINT,
+ ISD::STRICT_FTRUNC, ISD::STRICT_FROUND, ISD::STRICT_FROUNDEVEN}) {
for (MVT Ty : {MVT::v2f32, MVT::v4f32, MVT::v2f64})
setOperationAction(Op, Ty, Legal);
if (Subtarget->hasFullFP16())
diff --git a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
index 7057aa082..9df05219a 100644
--- a/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
+++ b/llvm/lib/Target/RISCV/RISCVISelLowering.cpp
@@ -539,9 +539,9 @@ RISCVTargetLowering::RISCVTargetLowering(const TargetMachine &TM,
// FIXME: Need to promote f16 STRICT_* to f32 libcalls, but we don't have
// complete support for all operations in LegalizeDAG.
setOperationAction({ISD::STRICT_FCEIL, ISD::STRICT_FFLOOR,
- ISD::STRICT_FRINT,
- ISD::STRICT_FROUND, ISD::STRICT_FROUNDEVEN,
- ISD::STRICT_FTRUNC, ISD::STRICT_FLDEXP},
+ ISD::STRICT_FRINT, ISD::STRICT_FROUND,
+ ISD::STRICT_FROUNDEVEN, ISD::STRICT_FTRUNC,
+ ISD::STRICT_FLDEXP},
MVT::f16, Promote);
// We need to custom promote this.
diff --git a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
index 85b12e649..546e2591a 100644
--- a/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
+++ b/llvm/lib/Target/X86/X86ISelDAGToDAG.cpp
@@ -1272,7 +1272,9 @@ void X86DAGToDAGISel::PreprocessISelDAG() {
case ISD::STRICT_FTRUNC:
case ISD::FTRUNC: Imm = 0xB; break;
case ISD::STRICT_FROUNDEVEN:
- case ISD::FROUNDEVEN: Imm = 0x8; break;
+ case ISD::FROUNDEVEN:
+ Imm = 0x8;
+ break;
case ISD::FNEARBYINT: Imm = 0xC; break;
case ISD::STRICT_FRINT:
case ISD::FRINT: Imm = 0x4; break;
diff --git a/llvm/lib/Target/X86/X86ISelLowering.cpp b/llvm/lib/Target/X86/X86ISelLowering.cpp
index 4c78891f7..daf1bcd79 100644
--- a/llvm/lib/Target/X86/X86ISelLowering.cpp
+++ b/llvm/lib/Target/X86/X86ISelLowering.cpp
@@ -1357,7 +1357,7 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
setOperationAction(ISD::STRICT_FTRUNC, RoundedTy, Legal);
setOperationAction(ISD::FRINT, RoundedTy, Legal);
setOperationAction(ISD::STRICT_FRINT, RoundedTy, Legal);
- setOperationAction(ISD::FNEARBYINT, RoundedTy, Legal);
+ setOperationAction(ISD::FNEARBYINT, RoundedTy, Legal);
setOperationAction(ISD::FROUNDEVEN, RoundedTy, Legal);
setOperationAction(ISD::STRICT_FROUNDEVEN, RoundedTy, Legal);
@@ -1457,7 +1457,7 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
setOperationAction(ISD::STRICT_FTRUNC, VT, Legal);
setOperationAction(ISD::FRINT, VT, Legal);
setOperationAction(ISD::STRICT_FRINT, VT, Legal);
- setOperationAction(ISD::FNEARBYINT, VT, Legal);
+ setOperationAction(ISD::FNEARBYINT, VT, Legal);
setOperationAction(ISD::FROUNDEVEN, VT, Legal);
setOperationAction(ISD::STRICT_FROUNDEVEN, VT, Legal);
@@ -1913,7 +1913,7 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
setOperationAction(ISD::STRICT_FTRUNC, VT, Legal);
setOperationAction(ISD::FRINT, VT, Legal);
setOperationAction(ISD::STRICT_FRINT, VT, Legal);
- setOperationAction(ISD::FNEARBYINT, VT, Legal);
+ setOperationAction(ISD::FNEARBYINT, VT, Legal);
setOperationAction(ISD::FROUNDEVEN, VT, Legal);
setOperationAction(ISD::STRICT_FROUNDEVEN, VT, Legal);
@@ -2250,7 +2250,7 @@ X86TargetLowering::X86TargetLowering(const X86TargetMachine &TM,
setOperationAction(ISD::STRICT_FTRUNC, VT, Legal);
setOperationAction(ISD::FRINT, VT, Legal);
setOperationAction(ISD::STRICT_FRINT, VT, Legal);
- setOperationAction(ISD::FNEARBYINT, VT, Legal);
+ setOperationAction(ISD::FNEARBYINT, VT, Legal);
setOperationAction(ISD::FROUNDEVEN, VT, Legal);
setOperationAction(ISD::STRICT_FROUNDEVEN, VT, Legal);
|
|
I'm a little worried that having the chain as an optional argument will end up with replacing a bunch of checks for Any thoughts on a null chain, vs. the entry chain? Is this a prerequisite for the IR refactoring you're looking at, or just a related cleanup? I don't see any obvious connection (as long as there's some way to translate from LLVM IR to SelectionDAG, it doesn't matter if it exactly matches). If this optional-chain thing works well, maybe we could also use it for #2535. |
|
The use of mandatory chain argument is discussed here: https://discourse.llvm.org/t/rfc-change-of-strict-fp-operation-representation-in-ir/85021. Using null chain is an interesting solution, because it allows detecting problems earlier. If we choose mandatory chain, it could be better than using the entry node.
No, this is independent solution for similar problem.
It is very interesting case. Solving that problem probably requires setting memory effects for sdiv/udiv, but DAG definitely has to be fixed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change replaces the DAG node STRICT_FNEARBYINT with a modified FNEARBYINT, which uses optional chain property. The modified node can be used in both strictfp and default environments.
This approach is based on the assumption that a floating-point operation is fundamentally the same in strictfp and default environments and is therefore lowered in almost identical manner. Indeed, all targets but one lower STRICT_FNEARBYINT using the same action as for FNEARBYINT. The only exception is PowerPC: it lowers STRICT_FNEARBYINT for vector types using
Expand, even though FNEARBYINT for v4f32 and v2f64 is legal. This change implements the lowering uniformly, treating these vector types as legal for PowePC target.The change demonstrate the transition from using separate nodes for strictfp and default environments to using a single node with an optional chain. It also modifies some methods of DAG functions required to support such nodes.