Reliable communication in crazyswarm

[swarm5]

@whoenig I hope to use less positioning data (motive outputs 30HZ positioning data, and crazyswarm only sends position data) to use cmdposition commands to control more cfs. In the case of using only a single PA, I have successfully implemented the flow control of the cmdposition of 10 cfs.
When I use a single PA and try more numbers, there are some communication-related phenomena:

1. When the high-level instruction is switched to the low-level instruction, some cfs do not respond. I use the takeoff command to take off, and all cfs can take off. But when the cf took off, I then used the cmdposition command to control the position of each cf, some of the aircraft seemed to have not received the command and still hovered in the air.
2. When taking off using the cmdposition command, some cfs flew out of the field as if there was a delay in receiving the command.
3. However, in the process of flying with cmdposition, all cfs performance is normal. All communication abnormalities occurred when cmdposition was first used.

Because in the whole process of the test, I only use one PA, this error phenomenon has nothing to do with the multi-threading problem of multiple PA communication.
I guess this phenomenon is related to what you said, there is no flow control between NRF51 and the STM32, and the firmware has bugs. Therefore, I want to try the repaired firmware. I would like to ask:

1. Has the crazyflie firmware used in the latest crazyswarm branch fixed the error in the communication firmware?
2. If crazyswarm has not been updated to the latest crazyflie firmware, I would like to know, how can I get the repaired crazyflie firmware now?

[swarm5]

@whoenig I have a little doubt, why I can control 10 cfs with a single PA without non-response (100HZ motive output, 20hz cmdposition output), but I cannot control 20cfs with a single PA (30hz motive output, 20hz output) cmdposition control)

[whoenig]

cmdposition is using unicast communication, so for 100Hz motive+20Hz cmdposition, you will send 100+20x10=300 packets/seconds. In the other case, you try to send 30+20x20=430 packets/second. In the use-case of cmdposition, I suggest using more radios to balance the load.

There are no current known bugs with respect to multi-threading and multiple Crazyradios. I don't think you are hit by the flow control issues (tracked bitcraze/crazyflie-firmware#703). There were a number of communication-related improvements in the firmware lately that you could try (but I think ultimately, you just need more radios). You can always install the latest official firmware, as the Crazyswarm does not require a special version anymore. Also make sure that logging is disabled, otherwise you need even more radio bandwidth.

[swarm5]

But shouldn't notifySetpointsStop have the same reliability level as other high-level commands? If notifySetpointsStop is occasionally dropped by your hardware, then we should expect commands like takeoff and land to be dropped occasionally too, yes? Is that happening?
Whoenig and I encountered the same situation.

1. When I use multiple PAs and transfer low-level commands to high-level commands, cfs will fly randomly.
2. There have been no abnormalities in the takeoff and land instructions.
3. Because of the situation of 1, I now use a single PA more, but when I use a single PA, when 20 cfs take off and transfer to cmdposition, some aircraft do not respond. But when 20cfs changed from cmdposition to land, there was no abnormality.

[swarm5]

Switching modes is not very well supported. I think high->low should work generally speaking, but low->high is very tricky and tends to not work reliably. I suggest that you only use the low-level mode, i.e., takeoff and land "manually", by sending appropriate position commands.

About your other questions:

• The radio is bandwidth limited, that's why I posted the num-packets-per-second calculation earlier.
• There are no known multi-threading issues, as stated earlier. There are known communication bugs, but I don't think you are triggering those (as also stated earlier).

Example to land "manually":

    if time_to_land > 0:
        dx = x_des[2] / time_to_land * dt
        for _ in range(int(time_to_land/dt)):
            x_des[2] -= dx
            cf.cmdPosition(x_des[0:3])
            rate.sleep()

    # cut motor power
    cf.cmdStop()

"Manual" takeoff works in a similar fashion (although I personally had no problems with high->low using 6 CFs recently).

I agree with your point: in the case of a small number, high-low command conversion is effective, but in the case of a large number of cfs, high-low command conversion does not seem to be very reliable, relying entirely on cmdposition seems more appropriate.
I have also conducted experiments using only cmdposition to take off and land, and I seem to have encountered some new troubles. I used a PA to control five cfs using cmdposition commands, and sometimes they turned over. From the visual effect, some cfs seemed to lag in receiving the order. I will add the video and the take-off code that I use cmdposition later.

[whoenig]

But shouldn't notifySetpointsStop have the same reliability level as other high-level commands? If notifySetpointsStop is occasionally dropped by your hardware, then we should expect commands like takeoff and land to be dropped occasionally too, yes? Is that happening?

Unfortunately, all the time for unicast commands. I always have to hack this right now to resend unicast commands, similar to what we do for broadcasts. I know what the problem is and it is fixed in crazyflie-link-cpp, but it will take some time to switch to that in the backend.

[whoenig]

@swarm5: there is no fair scheduling for unicast commands that share the same radio in the current code. In theory, it is possible that some CFs have a very high latency. This is going to be fixed with the upcoming crazyflie-link-cpp, which uses a round-robin scheduler. For the time being I suggest using multiple Crazyradios and only low-level commands. This should be working fine in the current code. For the flipping issue you see, please also specify the update rates (mocap, cmdposition), tracking mode, and used controller.

[swarm5]

@swarm5: there is no fair scheduling for unicast commands that share the same radio in the current code. In theory, it is possible that some CFs have a very high latency. This is going to be fixed with the upcoming crazyflie-link-cpp, which uses a round-robin scheduler. For the time being I suggest using multiple Crazyradios and only low-level commands. This should be working fine in the current code. For the flipping issue you see, please also specify the update rates (mocap, cmdposition), tracking mode, and used controller.

I have used a single PA and low-level commands to control the flight of 20 cfs.

[swarm5]

@whoenig @jpreiss I want to tell you that I found a problem with using cmdposition to take off multiple cfs in the test, and I found a feasible solution. I have only tested this solution to solve the cmdposition problem, but I still don't know the reason for this phenomenon.
【problem】:
When using a single PA, I used the cmdposition command to take off multiple cfs (20 cfs, 40HZmotive output, 10HZcmdposition output), and many cfs seemed to receive the message later and appeared to fly randomly.
In order to make it easier for everyone to reproduce this kind of problem, I use the program in crazyswarm: crazyswarm/ros_ws/src/crazyswarm/scripts/example_cmd_pos.py /
Reference video:

[My current solution]:
After the statement allcfs = swarm.allcfs, a program delay is required. After repeated tests, I found that this time needs to be greater than or equal to 3 seconds.

from pycrazyswarm import *
import numpy as np

swarm = Crazyswarm()
timeHelper = swarm.timeHelper
allcfs = swarm.allcfs
timeHelper.sleep(5)

Z = 0

while Z <1.0:
for cf in allcfs.crazyflies:
pos = np.array(cf.initialPosition) + np.array([0, 0, Z])
cf.cmdPosition(pos)
timeHelper.sleep(0.1)
Z += 0.05

while Z> 0.0:
for cf in allcfs.crazyflies:
pos = np.array(cf.initialPosition) + np.array([0, 0, Z])
cf.cmdPosition(pos)
timeHelper.sleep(0.1)
Z -= 0.05

for cf in allcfs.crazyflies:
cf.cmdStop()

[whoenig]

How are you running your script? The recommended way is to not put it in the launch file, but only run it after hover_swarm has successfully connected to all Crazyflies. Otherwise, this might be one of the many issues with the ROS1 pub/sub system.

[swarm5]

I have two questions:

1. Why do I need timeHelper.sleep when using cmdposition to take off?
2. As shown in the video, after sending the cf.cmdStop() command, some cf motors are still rotating and stop after a few seconds. What is the reason?

[swarm5]

How are you running your script? The recommended way is to not put it in the launch file, but only run it after hover_swarm has successfully connected to all Crazyflies. Otherwise, this might be one of the many issues with the ROS1 pub/sub system.

The way I run the script is strictly in accordance with the instructions in the crazyswarm documentation. After running hover_swarm, I opened a terminal again and ran python example_cmd_pos.py

[swarm5]

I would like to add one thing: I have done a test. Without timeHelper.sleep, even if I only use the cmdposition command to take off three cfs, the takeoff effect is unstable. The program is still: crazyswarm / ros_ws / src / crazyswarm / scripts / example_cmd_pos.py /

[whoenig]

• Why do I need timeHelper.sleep when using cmdposition to take off?

My guess is that this is a ROS1 pub/sub problem, as stated earlier. In ROS1 it takes some time to actually connect publishers and subscribers and there is no feedback about it. So your sleep just makes sure that all connections from your script to the Crazyflie_server are established.

• As shown in the video, after sending the cf.cmdStop() command, some cf motors are still rotating and stop after a few seconds. What is the reason?

Same reason as your troubles for the high->low transition: the unicast message is simply lost in some cases. They stop b/c there is a safety timeout in the firmware that disables the motor if no cmd is received for 2s. A hacky solution is to repeat the unicast message (you can wrap it in a loop). The proper solution will be to switch the backend to crazyflie-link-cpp, which will take a few months.