FPE in SITL

[andyp1per]

Bug report

Getting the following FPE in SITL quite regularly:

Apologies for the screenshot, can't seem to cut and paste

Issue details

Crash in SITL

Version
master

Platform
[ ] All
[ ] AntennaTracker
[X] Copter
[ ] Plane
[ ] Rover
[ ] Submarine

[andyp1per]

FYI @tpwrules @peterbarker

[andyp1per]

Caused by EK3_GSF_RUN_MASK being non-zero and some combinatino of SITL parameters.

[IamPete1]

While flying? There is one that the rover skid steer frame used to hit because it would yaw at a huge rate if not configured correctly. In that case we fixed the simulator to not allow such high rates.

[andyp1per]

While flying? There is one that the rover skid steer frame used to hit because it would yaw at a huge rate if not configured correctly. In that case we fixed the simulator to not allow such high rates.

Yeah while flying. It seems this is caused by SIM_SPEEDUP not being set to 1 when running realflight. I can kind of get my head round why that might be a problem, but an FPE still seems a little lame.

[peterbarker]

You can reliably reproduce this one using the disabled "CompassMot" test :-)

[luckys00]

@peterbarker
I followed your suggestion to reproduce this.

I enabled the disabled CompassMot test in Tools/autotest/arducopter.py and ran it locally.

I can confirm that it causes the SITL instance to crash reliably. The test script loses connection immediately after COMPASSMOT_STATUS, showing an EOF on TCP socket followed by Connection refused errors (as the process terminates).

output:
AT-0002.6: Received (COMPASSMOT_STATUS ...)
EOF on TCP socket
Attempting reconnect
[Errno 111] Connection refused sleeping

I am happy to try debugging this further with GDB if you think that's the right next step for me.

[luckys00]

@peterbarker
I successfully debugged the crash using GDB.

I handled the SIGTSTP/SIGCONT signals and caught the FPE. It crashes in EKFGSF_yaw::updateRotMat at line 623 (libraries/AP_NavEKF/EKFGSF_yaw.cpp).

Backtrace:

Program received signal SIGFPE, Arithmetic exception.
0x00005d92da55b563 in EKFGSF_yaw::updateRotMat (this=0x1, R=..., g=...)
at ../../libraries/AP_NavEKF/EKFGSF_yaw.cpp:623
623 rowLengthSq = ret[r][0] * ret[r][0] + ret[r][1] * ret[r][1] + ret[r][2] * ret[r][2];
This seems to be an arithmetic exception during the matrix row length calculation

[peterbarker]

@peterbarker I successfully debugged the crash using GDB.

I handled the SIGTSTP/SIGCONT signals and caught the FPE. It crashes in EKFGSF_yaw::updateRotMat at line 623 (libraries/AP_NavEKF/EKFGSF_yaw.cpp).

Backtrace:

Program received signal SIGFPE, Arithmetic exception. 0x00005d92da55b563 in EKFGSF_yaw::updateRotMat (this=0x1, R=..., g=...) at ../../libraries/AP_NavEKF/EKFGSF_yaw.cpp:623 623 rowLengthSq = ret[r][0] * ret[r][0] + ret[r][1] * ret[r][1] + ret[r][2] * ret[r][2]; This seems to be an arithmetic exception during the matrix row length calculation

Yep! Glad you got this into gdb.

So then the question is.... what's causing the state to get out-of-shape in there, and how do we stop that happening? Chasing that stuff down is why this test is still disabled in CI ;-)

[luckys00]

@peterbarker
To understand "what's causing the state to get out-of-shape", I inspected the matrix values at the time of the crash.

It seems the EKF state is diverging to infinity. The ret matrix contains values as large as 3.82e+304.

GDB Output:

(gdb) p ret
$2 = {a = {x = 4.8617558231713149e+290, ... },
c = {x = 3.8280303518771961e+304, ...}}
This confirms that a floating-point overflow occurs when these values are squared at line 623. I am currently investigating why the filter diverges to these values, but I wanted to share this trace as it directly answers the question about the state values.

[luckys00]

@peterbarker @andyp1per
Hi !! sir please review whenever you got chance .

[peterbarker]

@peterbarker @andyp1per Hi !! sir please review whenever you got chance .

Nothing really to add, I'm afraid. Yes, the values are getting crazy - it's on you to work out how!