[Flang][AArch64] Missed optimization opportunities with `unsafe-math-optimizations`

[yus3710-fj]

Duplicate of #165514

---

The execution time of Flang-compiled version of blts subroutine in blts.f90 from LU in NAS Parallel Benchmarks 3.4.3 is significantly slower than its GFortran-compiled one.

We measured the execution time using the perf command on Rocky Linux, running on an NVIDIA GRACE processor.
The sampling frequency for perf was set to 97, as detailed in the command below.

We used Flang built from commit 3e1d4d4144cc9d28ccd85cf49d6fc836c38ffbaa.
The GFortran version was 15.2.0.

All benchmarks were conducted by executing following command:

perf record --call-graph fp -F 97 -o data /path/to/compiled/binary

We observed that the execution time of Flang's compilation result, when compiled with the -Ofast -mcpu=neoverse-v2 option, was 167% of GFortran's execution time.
While the benchmark does not specify these options, we use them to obtain the optimal performance.

We found two factors: Flang's failure to apply adequate vectorization (#49896, #46522) and fast-math optimizations.

Specifically, Flang missed reciprocal-math optimization opportunities.
We compiled blts.f90 from LU in NAS Parallel Benchmarks with the following compile options:

-g -Ofast -mcpu=neoverse-v2 -fno-tree-vectorize -fno-fast-math -freciprocal-math

Note: Several parameters were specified to mitigate the influence of other factors on performance.

The execution time of Flang-compiled executable is 124% of GFortran's.
We checked the assembly code by toggling -freciprocal-math and found no diffs in Flang's output, whereas GFortran's output did show changes.

Additionally, while GFortran can optimize by specifying both -fno-signed-zeros and -fno-trapping-math, the latter (-fno-trapping-math) is not implemented in Flang.

[yus3710-fj]

Unfortunately, @yokamoto-fj is leaving our team, and his account will be closed. Therefore, his issue will be deleted soon. I have duplicated it.

[yus3710-fj]

By any chance, did you try compiling with -mrecip (#143418)?

I assume it is different from what GCC does for -freciprocal-math. GCC transform a/b into a*(1/b) in a few cases including if 1/b is invariant and then will put it out of the loop. or if you have a/b and c/b and will do tmp = 1/b and then a*tmp and b*tmp. This is different from recip which use the recip estimate instructions.

That is correct, though, the description of this issue is not explaining what kind of "reciprocal-math optimization opportunities" are missing in Flang comparing to Gfortran. This is why I asked if other "reiprocal" related options have been tried.

In any case, I think there are crucial details missing in the description.

@vzakhari, @pinskia Thank you for pointing it out and for the detailed information.
First of all, GFortran does not support -mrecip for AArch64, so I did not try it with either Flang or GFortran.

Next, I'd like to illustrate what GFortran acutually does for this program with -freciprocal-math. In short, CSE for divisions is promoted.

• -fno-fast-math

fdiv	d27, d27, d24 // tv(5) / tmat(5,5)
 ...
str	d27, [x4] // v(5,i,j,k)
fdiv	d25, d25, d23 // tv(4) / tmat(4,4)
 ...
fdiv	d26, d26, d4 // tv(3) / tmat(3,3)
 ...
stp	d26, d25, [x4, -16] // v(3:4,i,j,k)
 ...
fdiv	d29, d29, d6 // tv(2) / tmat(2,2)
 ...
fdiv	d31, d31, d9 // tv(1) / tmat(1,1)
stp	d31, d29, [x4, -32] // v(1:2,i,j,k)

• -fno-fast-math -freciprocal-math

fdiv	d27, d27, d21 // tv(5) / tmat(5,5)
 ...
str	d27, [x4] // v(5,i,j,k)
fmul	d25, d25, d19 // tv(4) * (1.0/tmat(4,4))
 ..
fmul	d28, d28, d20 // tv(3) * (1.0/tmat(3,3))
 ...
stp	d28, d25, [x4, -16] // v(3:4,i,j,k)
 ...
fmul	d30, d30, d17 // tv(2) * (1.0/tmat(2,2))
 ...
fmul	d31, d31, d16 // tv(1) * (1.0/tmat(1,1))
stp	d31, d30, [x4, -32] // v(1:2,i,j,k)

GFortran, when -freciprocal-math is specified, attempts to reuse the already calculated 1.0 / tmat(1,1) in the computation of tv(1) / tmat(1,1). Conversely, Flang does not perform such an optimization, even with -freciprocal-math enabled.

fdiv	d20, d20, d23 // tv(4) / tmat(4,4)
 ...
fdiv	d1, d1, d22 // tv(3) / tmat(3,3)
 ...
stp	d1, d20, [x13, #24] // v(3:4,i,j,k)
 ...
fdiv	d21, d21, d27 // tv(2) / tmat(2,2)
 ...
fdiv	d0, d0, d16 // tv(1) / tmat(1,1)
 ...
stp	d0, d21, [x13, #8] // v(1:2,i,j,k)
 ...
fdiv	d1, d1, d2 // tv(5) / tmat(5,5)
str	d1, [x13], #40 // v(5,i,j,k)

[vzakhari]

@yus3710-fj thanks for the details, but I think I am missing something. I do not see common subexpressions in this code (the one you quoted for -fno-fast-math and Gfortran):

fdiv	d27, d27, d24 // tv(5) / tmat(5,5)
 ...
str	d27, [x4] // v(5,i,j,k)
fdiv	d25, d25, d23 // tv(4) / tmat(4,4)
 ...
fdiv	d26, d26, d4 // tv(3) / tmat(3,3)
 ...
stp	d26, d25, [x4, -16] // v(3:4,i,j,k)
 ...
fdiv	d29, d29, d6 // tv(2) / tmat(2,2)
 ...
fdiv	d31, d31, d9 // tv(1) / tmat(1,1)
stp	d31, d29, [x4, -32] // v(1:2,i,j,k)

All the dvisions are using different elements of tv and tmat arrays. Does the redundancy come from these divisions being inside a loop? If so, can you please confirm that the loads tma(5,5), tv(5), etc. are hoisted out of the loop?

A source reproducer will be also welcome.

[yus3710-fj]

My apologies. I omitted some information. tmat is not loop-invariant. Actually, tmat is reciprocated before the code above, as shown here:

fmov	d2, 1.0e+0
 ...
fdiv	d25, d2, d9 // 1.0 / tmat(1,1)
 ...
fdiv	d3, d2, d6 // 1.0 / tmat(2,2)
 ...
fdiv	d15, d2, d4 // 1.0 / tmat(3,3)
 ...
fdiv	d31, d2, d23 // 1.0 / tmat(4,4)

I've created a small reproducer: https://godbolt.org/z/79jPcoazT
I hope this helps.

[yus3710-fj]

The arcp fast-math flag (triggered by -freciprocal-math, assuming Flang hooks it up) should trigger a/b -> a * (1.0/b) and a/(b/c) -> a*(c/b) transformations. arcp by itself doesn't trigger any reassociation, which can reduce optimization opportunities.

the latter (-fno-trapping-math) is not implemented in Flang.

The Flang driver is a little outside my knowledge base, but in general, LLVM defaults to -fno-trapping-math, so it's more accurate to say that -ftrapping-math is not implted in Flang.

@jcranmer-intel Thank you for your comment.
I checked that the behavior of GFortran remains unchanged with both -fno-fast-math -fassociative-math and -fno-fast-math options. Therefore, I assume that, at least in this case, the issue is not related to reassociation.

[vzakhari]

Thank you for the reproducer, @yus3710-fj

It looks like the nofpexcept MachineInstruction flag matters for optimizing this case. It is set under -ffast-math, and not set when you use -fno-fast-math. I am not sure if Flang has a separate option to set nofpexcept.

This seems to affect x86 as well.