Incorrect fabs optimization when fdiv or fmul is involved

[aqjune]

Bugzilla Link	47943
Version	trunk
OS	All
Blocks	#47292
CC	@LebedevRI,@RKSimon,@nikic,@nunoplopes,@regehr,@rotateright

Extended Description

---

// Transforms/InstCombine/fmul.ll

define float @​fabs_squared(float %x) {
%0:
%x.fabs = fabs float %x
%mul = fmul float %x.fabs, %x.fabs
ret float %mul
}
=>
define float @​fabs_squared(float %x) {
%0:
%mul = fmul float %x, %x
ret float %mul
}
Transformation doesn't verify!
ERROR: Value mismatch

Example:
float %x = undef

Source:
float %x.fabs = NaN [based on undef value]
float %mul = NaN [based on undef value]

Target:
float %mul = #x80000000 (-0.0)
Source value: NaN
Target value: #x80000000 (-0.0)

---

// Transforms/InstCombine/fdiv.ll

define float @​fabs_same_op(float %x) {
%0:
%a = fabs float %x
%r = fdiv float %a, %a
ret float %r
}
=>
define float @​fabs_same_op(float %x) {
%0:
%r = fdiv float %x, %x
ret float %r
}
Transformation doesn't verify!
ERROR: Value mismatch

After optimization, fmul %x, %x isn't guaranteed to be non-negative because the result can be negative if %x was undef.
Similarly for the fdiv case.

[nunoplopes]

This optimization becomes correct once undef is removed, so I'm not sure it's worth fixing it, unless it triggers a real miscompilation.

[rotateright]

In the original fmul example:
%mul = fmul float %x.fabs, %x.fabs

Why is that guaranteed to be non-negative?

Assume that %x is some NaN value. Then %x.fabs is guaranteed to be some positive NaN value (fabs guarantees that the sign-bit is cleared for any input).
But the result of fmul with NaN operands is not guaranteed to preserve the sign-bit -- it only needs to return an arbitrary NaN value. From IEEE-754: "this standard does not specify the sign bit of a NaN result".

(It would be a very strange implementation if an actual target set the sign bit, but it is not forbidden by the standard or the LLVM LangRef from what I can tell.)

[aqjune]

Okay, to be more precise the wording should be that %mul in source is either NaN or non-negative; %mul cannot be negative non-NaN however (e.g., -1.0)

[rotateright]

Okay, to be more precise the wording should be that %mul in source is either
NaN or non-negative; %mul cannot be negative non-NaN however (e.g., -1.0)

Ok, that makes sense - thanks for explaining! So Alive2 is at least partially tracking the bit-fields within an FP value?

If we do want to fix this due to miscompile, then we could change this bit of ConstantFolding:

  // [any flop] undef, undef -> undef
  if (isa<UndefValue>(C1) && isa<UndefValue>(C2))
    return C1;
  // [any flop] C, undef -> NaN
  // [any flop] undef, C -> NaN

Instead of returning undef when both operands are undef, just return NaN like we do already if only one operand is undef?

[aqjune]

I think the undef folding optimization is okay, but these specific transformations seem problematic.

As Nuno mentioned, this optimization can be revived if undef value is removed - I'll make an umbrella bug report that links all such bugs.

[aqjune]

mentioned in issue #47292