Unable to achieve 10 Gbit/s throughput on Hetzner server

[nh2]

I'm benchmarking Nebula with storage servers from dedicated server provider Hetzner where 10 Gbit/s links are cheap.

Unless you ask them to connect your servers by a dedicated switch, the MTU cannot be changed, so jumbo frames are not possible.

In this setup, I have not been able to achieve more than 2 Gbit/s with iperf3 over Nebula, no matter how I tune read_buffer/write_buffer/batch/routines.

---

In https://theorangeone.net/posts/nebula-intro/ it was said

Slack have seen Nebula networks fully saturate 5 / 10 gigabit links without sweat

and on https://i.reddit.com/r/networking/comments/iksyuu/overlay_network_mesh_options_nebula_wireguard/

Slack regularly does many gigabits per second over nebula on individual hosts.

but that's evidently not the case for me.

Did all those setups use jumbo frames?

Is there anything that can be done to achieve 10 Gbit/s throughput without jumbo frames?

[rawdigits]

I'm guessing this is cpu bottlenecked. What is the output of cat /proc/cpuinfo ?

[nh2]

The servers are SX133 and SX134 servers from Hetzner (linked in issue description).

They have Xeon W-2145 CPU and Ryzen 7 3700X respectively.

Click to expand `/proc/cpuinfo` details:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz
stepping        : 4
microcode       : 0x2006906
cpu MHz         : 1200.027
cache size      : 11264 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 8
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 22
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16crdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d
vmx flags       : vnmi preemption_timer posted_intr invvpid ept_x_only ept_ad ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid unrestricted_guest vapic_reg vid ple shadow_vmcs pml ept_mode_based_exec tsc_scaling
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips        : 7399.70
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 113
model name      : AMD Ryzen 7 3700X 8-Core Processor
stepping        : 0
microcode       : 0x8701021
cpu MHz         : 2251.193
cache size      : 512 KB
physical id     : 0
siblings        : 16
core id         : 0
cpu cores       : 8
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 16
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev ibpb stibp vmmcall sev_es fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif umip rdpid overflow_recov succor smca
bugs            : sysret_ss_attrs spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 7186.10
TLB size        : 3072 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

[nh2]

nebula CPU usage in htop is around 150% when the iperf3 is running; is that expected, or should it go higher?

The machines have 4 and 8 physical cores respectively.

[nh2]

Hey, just a short followup on whether anything can be done to achieve proper 10 Gbit/s throughput, or how to investigate when it doesn't happen.

[HenkVanMaanen]

Had the same problem with the Hetzner CX cloud servers. Without nebula iperf3 would report around 7 Gbit/s between two servers. With nebula it wouldn't go above 1 Gbit/s. I think it has something to do with nebula being tcp over udp traffic and that udp traffic on Hetzner is either rate limited or the routers can't handle the udp traffic. Tcp over tcp would be the solution IMO, but nebula does not support that at the moment.

[HenkVanMaanen]

Here's the link to the thread on the NebulaOSS slack channel: https://nebulaoss.slack.com/archives/CS01XE0KZ/p1619532900073100

[nh2]

that udp traffic on Hetzner is either rate limited or the routers can't handle the udp traffic

@HenkVanMaanen I cannot confirm that.

What speed does iperf3 show in UDP mode between your Hetzner servers?

For me it's as fast as TCP mode between 2 dedicated 10 Gbit/s servers (Hetzner SX133 and SX134):

• TCP via iperf3 -c otherserver: 9.41 Gbits/sec on on the other side
• UDP single-flow via iperf3 -c otherserver --udp -b 10G:

  [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
  [  5]   0.00-10.00  sec  4.76 GBytes  4.09 Gbits/sec  0.000 ms  0/3529420 (0%)  sender
  [  5]   0.00-10.00  sec  4.76 GBytes  4.09 Gbits/sec  0.001 ms  96/3529419 (0.0027%)  receiver
  

• UDP multi-flow (5 parallel flows, each limited to 2 Gbit/s) via iperf3 -c otherserver --udp -b 2G -P5 (the idea to need multiple flows comes from this AWS article on 100 Gbit/s networking):

  [ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
  [SUM]   0.00-10.00  sec  11.1 GBytes  9.56 Gbits/sec  0.000 ms  0/8254495 (0%)  sender
  [SUM]   0.00-10.00  sec  10.9 GBytes  9.35 Gbits/sec  0.004 ms  184417/8254338 (2.2%)  receiver
  

So 10 Gbit/s works on UDP between these 2 machines on the Hetzner network.

---

Nebula-based iperf3 tops out on ~3.5 Gbit/s between the same machines, no matter if via TCP or UDP (no matter the flows).

---

I also re-measured with nuttcp to confirm I'm not hitting iperf3-specific limitations, using e.g. for UDP:

nuttcp -S -P5200 -p5201  # this backgrounds itself
nuttcp -P5200 -p5201 -u -R9g -w2m otherserver

[nh2]

Here's the link to the thread on the NebulaOSS slack channel: https://nebulaoss.slack.com/archives/CS01XE0KZ/p1619532900073100

Replying to some more topics I read on that thread:

The fact that your cpu graphs in the first bit don't show a single core maxed out is unexpected. It means something is holding nebula back from running at full clip.

The same appears for me: Nebula uses only ~125% CPU usage which is evenly spread out across cores, not peaking out a single core. htop screenshot:

Interface MTUs:

internet0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

nebula: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1300

There is only 1 physical link active on my servers, so confusing different links is impossible.

• Ensure you aren't dropping packets at tun. This shows up in ifconfig with dropped <number> under the nebula1 interface. Raise tun.tx_queue until drops are not increasing.

I did see some packet drops at tun. I changed tun.tx_queue from the default (500) to 2000 and the tun drops disappeared.
But this did not improve the throughput.

• Ensure you aren't seeing packets dropped by the udp buffers (listen.read_buffer and listen.write_buffer should be increased until ss -numpile shows no dropped packets, it's the last field, d<number>). Generally the read buffer is the problem.

I **do see drops in ss -numpile on the receiving side.

Increasing listen.*_buffer didn't help though; I tried values between 10 MiB and 640 MiB, and I continue to see e.g. d403244 increasing in watch -n1 ss -numpile.

I tried with the default listen.batch setting, and setting it to 256.

Is it possible to verify that the listen.*_buffer settings are really in effect?

Similarly, in netstat -suna, these fields keep increasing during the iperf transmission over Nebula:

    35188361 packet receive errors
    35188361 receive buffer errors

Following this post I used dropwatch to get details of the drops as they happen. Output:

# dropwatch -l kas
Initializing kallsyms db
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
5 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
1 drops at unix_stream_connect+800 (0xffffffff9b586bc0) [software]
67498 drops at udp_queue_rcv_one_skb+396 (0xffffffff9b537f86) [software]
3 drops at udp_queue_rcv_one_skb+396 (0xffffffff9b537f86) [software]
5 drops at udp_queue_rcv_one_skb+396 (0xffffffff9b537f86) [software]
3 drops at udp_queue_rcv_one_skb+396 (0xffffffff9b537f86) [software]
2 drops at udp_queue_rcv_one_skb+396 (0xffffffff9b537f86) [software]
1 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
69804 drops at udp_queue_rcv_one_skb+396 (0xffffffff9b537f86) [software]
1 drops at udp_queue_rcv_one_skb+396 (0xffffffff9b537f86) [software]
2 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
3 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
68186 drops at udp_queue_rcv_one_skb+396 (0xffffffff9b537f86) [software]
... more of that ...

[nh2]

Increasing listen.*_buffer didn't help though; I tried values between 10 MiB and 640 MiB, and I continue to see e.g. d403244 increasing in watch -n1 ss -numpile.

I found that changing the sysctl net.core.rmem_default from its default 212992 to 100x that value (21299200) gets rid of all those drops (in ss and netstat -suna), and dropwatch now looks like:

# dropwatch -l kas                        
Initializing kallsyms db
dropwatch> 
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
5 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
1 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
5 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
1 drops at nf_hook_slow+8f (0xffffffff9b4f1e0f) [software]
1 drops at nf_hook_slow+8f (0xffffffff9b4f1e0f) [software]
1 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
4 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
1 drops at nf_hook_slow+8f (0xffffffff9b4f1e0f) [software]
1 drops at nf_hook_slow+8f (0xffffffff9b4f1e0f) [software]
1 drops at unix_stream_connect+800 (0xffffffff9b586bc0) [software]
5 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
1 drops at sk_stream_kill_queues+50 (0xffffffff9b458bf0) [software]
1 drops at nf_hook_slow+8f (0xffffffff9b4f1e0f) [software]

The fact that I had to set net.core.rmem_default suggests to me that Nebula's own buffer adjustments (listen.read_buffer) aren't working, as suspected above.

But even with all drops being fixed the throughput of Nebula does not improve.

[HenkVanMaanen]

These are the results between two CX servers, direct tunnel:

iperf3 -c otherserver -b 10G -b 5G -P2 = 5 Gbit/s

iperf3 -c otherserver --udp -b 10G -b 5G -P2 = 1 Gbit/s

[nh2]

-b 10G -b 5G -P2

@HenkVanMaanen You're giving -b twice -- I realise that this is because I typoed that in my summary above, and I also swapped my values of -b and -P (which I just fixed), sorry for that.

My run was with -b 2G -P5. Could you try with that, just for completeness (perhaps also with smaller values of -b, e.g. `-b 1G -P10)?

[nh2]

Some more info:

In the thread view in htop, I can see that there are generally 2 threads that use CPU:

• on the sender side: 60% and 40%
• on the receiver side: 98% and 60%

Interestingly, if I taskset -c 1 on both sides to pin nebula onto a single core, we get on the receiver: 60% and 10%.

So now in sum it takes less than 100%.

Then, for the time the process is spending, I checked in htop that the fractions are 25% user, 75% sys.

On the receiver side, timeout 10 strace -fyp "$(pidof nebula)" -c gives (strace started while iperf3 is transmitting over the single-threaded Nebula):

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 54.81   12.410432         439     28210        30 futex
 20.97    4.747149          38    122748           nanosleep
  9.42    2.132681          65     32333           recvmmsg
  7.19    1.628890           3    485574           write
  3.33    0.754306           3    242637           sendto
  2.25    0.510448           2    242631           read
  2.02    0.457547         490       932           epoll_pwait
  0.00    0.000162           3        43           sched_yield
  0.00    0.000088           1        56           getsockname
  0.00    0.000067           3        18           recvfrom
  0.00    0.000042          42         1           restart_syscall
  0.00    0.000020           3         6           socket
  0.00    0.000013           2         6           close
  0.00    0.000013           2         6           bind
  0.00    0.000011           1         6           getsockopt
  0.00    0.000009           9         1           tgkill
  0.00    0.000006           6         1           getpid
  0.00    0.000003           3         1           rt_sigreturn
------ ----------- ----------- --------- --------- ----------------
100.00   22.641887          19   1155210        30 total

Not sure how accurate that is, as the throughput over the single-threaded one breaks down from 1.4 Gbit/s to 0.44 Gbit/s when strace is active.

futex here is shown as the main bottleneck, but it may be that futex would vanish if it wasn't ptrace()d by strace.

I wonder what the futexes are though; there aren't even that many of them (2800 per second).

[HenkVanMaanen]

• --udp -b 2G -P5
• --udp -b 1G -P10
• --udp -b 5G -P2

All around 1 Gbit/s

Via TCP I get 4 Gbit/s

[sfxworks]

@nh2 just curious, for the encryption method in your config are you using aes?

[nh2]

@sfxworks Yes, AES.