Offboard goes to manual in midair immediately after automatic takeoff by MRS

[MJavadZallaghi]

I have an aerial robot, and have installed MRS UAV System version 1.5 (ROS Noetic) on it. I'm using mrs_point_lio as my state estimator as my SLAM.

MRS is able to take off my robot autonomously, but immediately after the takeoff, the robot performs an emergency landing (eland). I checked my console logs, and I see there is an error in the control manager after the takeoff that states:

[ERROR]: [ControlManager]: we fell out of  OFFBOARD in mid air, disabling output

To check the issue more deeply, I removed my props and made a bench test, where I moved the robot to the takeoff setpoint, and in this case, the issue did not exist! There was only one difference between my real flight and the bench test: My ground control station was connected to the Pixhawk over USB. To confirm this, I removed the USB and repeated the bench test, and saw that the issue was happening. Also, I was able to make a full autonomous flight from takeoff to landing with MSR in offboard mode, but with a connected USB cable to the ground control station (my laptop).

I have these things on my aerial robot:

Frame and motors: Holybro X650
Flight computer: Pixhawk 6x
Companion computer: Jetson Orin NX 16 GB
Firmware: PX4 1.16.0
My ground control station (GCS): QGroundControl on Windows

I have a good Wi-Fi UDP connection between GCS and Pixhawk 6x, not interrupted even during flights. The strange point is that in offboard mode, the GCS has nothing to do with the robot except monitoring; So what is really the link between my problem and the non-existence of USB between GCS and Pixhawk 6x?

Also, I check my PX4 logs (available here). Indeed, the error that appears after the takeoff by MRS is a PX4 failsafe that states offboard signal lost. In the MRS ROS core log, it states OFFBOARD has deactivated, so it had to stop sending output.

It's really strange, but what it is, is related to the USB between Pixhawk and GCS.

I have attached my main MRS terminal logs.
4_HwApi.log
5_Status.log
7_Core.log
8_AutoStart.log
14_EstimDiag.log
15_kernel_log.log
16_roscore.log

I would appreciate any kind of ideas and fixes for solving this problem!

[MJavadZallaghi]

@klaxalk Hey Tomas. Would be appreciated to hear your opinion and feedbacks on this problem.

@migsdigs Did you use mrs_point_lio? Did you have the problem of mine? However, I'm sure it's not related to the mrs_point_lio as the state estimator is working perfectly.

[jvitku]

Hi @MJavadZallaghi , we were playing with MRS recently, couple of notes on this from my experience. If I understand it correctly, the MRS is basically telling you the FC switched from OFFBOARD itself. We have seen this due to the following reasones:

• right after the takeoff, the companion computer frozen briefly (high CPU load), did not manage to send the control signal fast enough and the FC decided to failsafe and switch the flight mode
• also if the takeoff is too abrupt, it can decrease your battery voltage temporarily, potentially causing the FC to switch to a predefined failsafe (offboard PX4 failsafe conditions described here https://docs.px4.io/main/en/flight_modes/offboard#description )
• and also, as Tomas is pointing out, the QGCS is definitely able to interfere with the whole thing (missing messages etc) and you have to disable the debug console, or use different UART. The mapping of UART-X to ttyY is usually in the FC documentation.

You can try uploading flight logs to https://logs.px4.io/ and look for reason why the FC switched flight mode (is_failsafe graph lines I think).

Good luck,
Jarda

[MJavadZallaghi]

I also find this diagram for offboard mode in PX4 docs:

I don't understand why GCS should be ROS-enabled and have a MAVLink connection; however, in the CTU MRS docs, I never find such a requirement for a GCS, but I thought maybe my problem is related to this. If that's the case, how a USB connection between QGroundControl is able to solve the problem?

[klaxalk]

Hey, the GCS (if it is QGC) should have nothing to do with it in any positive way... it is not needed and it might even interfer when it is connected over WiFi through Mavros (since more data is pushed through the serial line). Moreover, there is a known issue that can cause the Mavros the crash when QGC is disconnected, so do not use this during a flight. Just to make sure we are on the same page:

• Are you running UART between the offboard computer and PixHawk?
• what baud rate are you using?
• what /etc/extras.txt configuration are you using? (ours here: https://github.com/ctu-mrs/mrs_uav_deployment/tree/master/miscellaneous/pixhawk_sdcard_config)

The message "we are out of offboard' is just a report on the change of state of PX4. This means that PX4 decided to exit offboard for on of the following reasons:

• the commnands timeouted (approx. 0.2 s timeout), this might suggest a wobbly uart cable or a problem on the companion computer side
• you moved the mode switch or a stick on your transmitter (firmwares newere than 1.15 I think)
• you turned of the offboard using the transmitter

Tomas

[jvitku]

Have you tried uploading the logs to https://logs.px4.io/ ?

Here's an example of what I would be looking for.
In this situation, it clearly shows the situation where the MRS fell of OFFBOARD and stopped sending control signal. It also explains that the flight mode change was caused by the stick movement on RC.

s.

[MJavadZallaghi]

Yeah, here you can see the log for one of my tests that I have this issue:
https://review.px4.io/plot_app?log=68cbb943-9105-4345-a099-64c4530d0ee5

[jvitku]

Hm, not sure. Just couple of notes until more knowledgeable guys appear:

• it failsafed, MRS stopped sending offboard commands (not sure what was first)
• it complains about losing connection to mission computer and GCS
• the Z estimation seems wrong? (and the XY estimation not available)

[MJavadZallaghi]

MRS was not first — I have checked with time stamps of my ROS logs.
The estimations you are mentioning are from EKF2 of PX4, that don't matter to MRS. In MRS side (ROS) I's using mrs_point_lio and have tested it tens of times. So my MRS states estimator works perfectly all the times.

[jvitku]

Ok then it’s the PX4 failsafe engaged (then MRS stops sending).