QUIC BBR not fully utilize the 5Gbps channel (emulated CORE Ubuntu)

[joynwang]

I am testing QUIC BBRv1, QUIC BBRv3 and QUIC Cubic under CORE (Common Open Research Emulator) in Ubuntu system. It is simple dumbell scenario where The bottleneck link is 5Gbps with 100ms delay, while the rest is 100Gbps. 1GB file transfer between client and service.
Wireshark at n3 shows the throughput for BBR hitting at most 300Mbps, but we have 5Gbps link. Please see attached file for details.

any idea on what is going on? I tried with QUIC BBRv1, QUIC BBRv3 and QUIC Cubic which behave the same. I did the test with 10Mbps bottleneck link and it works fine. is it to do with app-limited?

I appreciate any feedback on this. Thanks.

[huitema]

You are targeting a 5Gbps link. That's pretty close to the best performance obtained with picoquic, when running on a loopback interface on a fast CPU. Note that picoquic, by design, uses just one of the CPU cores -- we are working on "scale-up" solutions that run multiple picoquic instances in parallel on multiple cores, but these solutions do not directly address your problem. The observed performance vary between 2 Gbit/s and 5 Gbit/s depending on how powerful the CPU is.

You are observing throughputs that are much lower than 5 Gbit/s or even 2 Gbit/s. There are 3 plausible causes:

• Turning on logging options. Such options write to disk, and can drastically affect performance. There is an "in memory logging" option in picoquic that can be used instead, but it requires allocating lots of memory per process.

• Some bug in the congestion control algorithm. That's always possible, but it is a bit hard to believe that they would cause the performance to drop by a factor 10. The pacing issues under high CPU load described in a recent blog post occur with algorithms like BBRv3, BBRv1 or C4 that rely heavily on pacing, but would not affect Cubic.

• Something about how picoquic is deployed on your test environment.

When picoquic runs at high speed, the single core running the sender is maxed out. Analysis of "flame graphs" shows that the highest CPU consumption (70 to 80%) is caused by the socket calls to "sendmsg", with the reminder split between the protocol handling itself and the encryption functions. The high performance depends heavily on the UDP Generic Segmentation Offload (UDP GSO) functions that allow sending 10 or 20 packets in a single system call -- and thus reduce the CPU used for socket calls by a large factor. The reduction of performance from 2-5 Gbit/s to 300 Mbit/s would be consistent with the absence of support for UDP GSO.

I have observed that issue previously when using NS-3 in "real time" mode. The network drivers used for the NS-3 simulation do not support UDP GSO, and that is a limiting factor for protocols like QUIc that run on top of UDP. Could that be the same issue affecting the CORE simulations?