Skip to content

Commit ee490a0

Browse files
jasonrandrewsjasonrandrews
authored
Merge pull request #1996 from kieranhejmadi01/iperf3
get-started-with-iperf3-LP
2 parents 5630359 + 37da31c commit ee490a0

File tree

9 files changed

+384
-0
lines changed

9 files changed

+384
-0
lines changed

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/_index.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
title: Get started with network microbenchmarking and tuning with iperf3
3+
4+
minutes_to_complete: 30
5+
6+
who_is_this_for: Performance Engineers, Linux system administrators or application developers looking to microbenchmark, simulate or tune the networking performance of distributed systems.
7+
8+
learning_objectives:
9+
- Understand how to use the iperf3 tool to microbenchmark different network conditions
10+
- Understand how to use the tc tool to simulate different network environments
11+
- Understand basic runtime parameters to tune performance for your application
12+
13+
prerequisites:
14+
- Foundational understanding on networking principles such as TCP/IP and UDP.
15+
- Access to Arm-based cloud instances or access to physical hardware
16+
17+
author: Kieran Hejmadi
18+
19+
### Tags
20+
skilllevels: Introductory
21+
subjects: Performance and Optimization
22+
armips:
23+
- Neoverse
24+
tools_software_languages:
25+
- iperf3
26+
operatingsystems:
27+
- Linux
28+
29+
30+
31+
further_reading:
32+
- resource:
33+
title: iperf3 user manual
34+
link: https://iperf.fr/iperf-doc.php
35+
type: documentation
36+
37+
38+
### FIXED, DO NOT MODIFY
39+
# ================================================================================
40+
weight: 1 # _index.md always has weight of 1 to order correctly
41+
layout: "learningpathall" # All files under learning paths have this same wrapper
42+
learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
43+
---

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/_next-steps.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
---
2+
# ================================================================================
3+
# FIXED, DO NOT MODIFY THIS FILE
4+
# ================================================================================
5+
weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation.
6+
title: "Next Steps" # Always the same, html page title.
7+
layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing.
8+
---

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/basic-microbenchmarking.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
---
2+
title: Microbenchmark Existing Network Connection
3+
weight: 3
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Microbenchmark TCP Connection
10+
11+
12+
First we will microbenchmark the bandwidth between the client and server. First start the `iperf` server on the server node with the following command.
13+
14+
```bash
15+
iperf3 -s
16+
```
17+
18+
```output
19+
-----------------------------------------------------------
20+
Server listening on 5201 (test #1)
21+
-----------------------------------------------------------
22+
23+
24+
```
25+
By default, the server listens on port 5201. Use the `-p` flag to specify another port if it is in use.
26+
27+
{{% notice Tip %}}
28+
If you already have an `iperf3` server running, you can manually kill the process with the following command.
29+
```bash
30+
sudo kill $(pgrep iperf3)
31+
```
32+
{{% /notice %}}
33+
34+
35+
Next, on the client node, run the following command to run a simple 10-second microbenchmark using the TCP protocol.
36+
37+
```bash
38+
iperf3 -c SERVER -V
39+
```
40+
41+
```output
42+
...
43+
[ 5] local 10.248.213.97 port 42176 connected to 10.248.213.104 port 5201
44+
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
45+
[ ID] Interval Transfer Bitrate Retr Cwnd
46+
[ 5] 0.00-1.00 sec 594 MBytes 4.98 Gbits/sec 0 1.48 MBytes
47+
[ 5] 1.00-2.00 sec 593 MBytes 4.97 Gbits/sec 0 2.07 MBytes
48+
[ 5] 2.00-3.00 sec 592 MBytes 4.97 Gbits/sec 0 2.07 MBytes
49+
[ 5] 3.00-4.00 sec 590 MBytes 4.96 Gbits/sec 0 2.07 MBytes
50+
[ 5] 4.00-5.00 sec 593 MBytes 4.97 Gbits/sec 0 2.18 MBytes
51+
[ 5] 5.00-6.00 sec 591 MBytes 4.96 Gbits/sec 0 2.18 MBytes
52+
[ 5] 6.00-7.00 sec 592 MBytes 4.97 Gbits/sec 0 2.18 MBytes
53+
[ 5] 7.00-8.00 sec 593 MBytes 4.97 Gbits/sec 0 2.18 MBytes
54+
[ 5] 8.00-9.00 sec 588 MBytes 4.93 Gbits/sec 0 2.18 MBytes
55+
[ 5] 9.00-10.00 sec 592 MBytes 4.96 Gbits/sec 0 2.18 MBytes
56+
- - - - - - - - - - - - - - - - - - - - - - - - -
57+
Test Complete. Summary Results:
58+
[ ID] Interval Transfer Bitrate Retr
59+
[ 5] 0.00-10.00 sec 5.78 GBytes 4.96 Gbits/sec 0 sender
60+
[ 5] 0.00-10.00 sec 5.78 GBytes 4.96 Gbits/sec receiver
61+
CPU Utilization: local/sender 5.3% (0.1%u/5.2%s), remote/receiver 26.7% (1.2%u/25.5%s)
62+
snd_tcp_congestion cubic
63+
rcv_tcp_congestion cubic
64+
65+
iperf Done.
66+
```
67+
68+
- The`Cwnd` stands for the control window size and corresponds to the allowed number of TCP transactions inflight before receiving an acknowledgment `ACK` from the server. This adjusts dynamically to not overwhelm the receiver and adjust for variable link connection strengths.
69+
70+
- The `CPU Utilization` row shows both the usage on the sender and receiver. If you are migrating your workload to a different platform, such as from `x86` to `AArch64`, there may be subtle variations.
71+
72+
- The `snd_tcp_congestion cubic` abd `rcv_tcp_congestion cubic` variables show the congestion control algorithm used.
73+
74+
- This `bitrate` shows the throughput achieved under this microbenchmark. As we can see from the above, we have saturated the 5 Gbps bandwidth available to our `t4g.xlarge` AWS instance.
75+
76+
![instance-network-size](./instance-network-size.png)
77+
78+
### Microbenchmark UDP connection
79+
80+
We can also microbenchmark the `UDP` protocol with the `-u` flag. As a reminder, UDP does not guarantee packet delivery with some packets being lost. As such we need to observe the statistics on the server side to see the % of packet lost and the variation in packet arrival time (jitter). The UDP protocol is widely used in applications that need timely packet delivery, such as online gaming on video calls.
81+
82+
Run the following command from the client to send 2 parallel UDP streams with the `-P 2` option.
83+
84+
```bash
85+
iperf3 -c SERVER -V -u -P 2
86+
```
87+
88+
Looking at the server output we can observe 0% of packets where lost for our short test.
89+
90+
```output
91+
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
92+
[ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.016 ms 0/147 (0%) receiver
93+
[ 6] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.014 ms 0/147 (0%) receiver
94+
[SUM] 0.00-10.00 sec 2.51 MBytes 2.10 Mbits/sec 0.015 ms 0/294 (0%) receiver
95+
```
96+
97+
Additionally on the client side, our 2 streams saturated 2 of our 4 cores in the local node.
98+
99+
```output
100+
CPU Utilization: local/sender 200.3% (200.3%u/0.0%s), remote/receiver 0.2% (0.0%u/0.2%s)
101+
```

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/example_traffic_rules.png‎

29 KB
Loading

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/instance-network-size.png‎

99.8 KB
Loading

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/server-ip.png‎

14.8 KB
Loading

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/setup.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
---
2+
title: Setup
3+
weight: 2
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
### Setup
10+
11+
For this demonstration I will be using instances available from AWS within a virtual private cloud (VPC)
12+
13+
Create 2 Arm-based linux instances, 1 to act as the server and the other to act as the client. In this tutorial I will be using two `t4g.xlarge` instance running Ubuntu 22.04 LTS.
14+
15+
16+
### Install dependencies
17+
18+
Run the following command to install the microbenchmark tool, `iperf3`.
19+
20+
```bash
21+
sudo apt update
22+
sudo apt install iperf3 -y
23+
```
24+
25+
26+
### Update Security Rules
27+
28+
Next, we need to update the default security rules to enable specific inbound and outbound protocols. From the AWS console, navigate to the security tab. Edit the inbound rules to enable `ICMP`, `UDP` and `TCP` traffic to enable communication between the client and server
29+
30+
31+
![example_traffic](./example_traffic_rules.png)
32+
33+
{{% notice Note %}}
34+
For security set the source and port ranges to those that are being used
35+
{{% /notice %}}
36+
37+
38+
### Update local DNS
39+
40+
For readability, we will add the server IP address and an alias to the local DNS cache in `/etc/hosts`. The local IP address of the server and client can be found in the AWS dashboard.
41+
42+
On the client, add the IP address of the server to the `/etc/hosts` file. Likewise on the server add the IP address of the client to the `/etc/hosts` file.
43+
44+
![server-ip](./server-ip.png).
45+
46+
### Confirm server is reachable
47+
48+
Finally, confirm the client can reach the server with the ping command below. As a reference we also ping the localhost.
49+
50+
```bash
51+
ping SERVER -c 3 && ping 127.0.0.1 -c 3
52+
```
53+
54+
The output below shows that both SERVER and localhost (127.0.0.1) are reachable. Naturally, on this system local host response tile is ~10x faster than the server. Your results will vary depending on geographic colocation and other networking factors.
55+
56+
```output
57+
PING SERVER (10.248.213.104) 56(84) bytes of data.
58+
64 bytes from SERVER (10.248.213.104): icmp_seq=1 ttl=64 time=0.217 ms
59+
64 bytes from SERVER (10.248.213.104): icmp_seq=2 ttl=64 time=0.218 ms
60+
64 bytes from SERVER (10.248.213.104): icmp_seq=3 ttl=64 time=0.219 ms
61+
62+
--- SERVER ping statistics ---
63+
3 packets transmitted, 3 received, 0% packet loss, time 2056ms
64+
rtt min/avg/max/mdev = 0.217/0.218/0.219/0.000 ms
65+
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
66+
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.022 ms
67+
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.032 ms
68+
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.029 ms
69+
70+
--- 127.0.0.1 ping statistics ---
71+
3 packets transmitted, 3 received, 0% packet loss, time 2046ms
72+
rtt min/avg/max/mdev = 0.022/0.027/0.032/0.004 ms
73+
74+
```
75+

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/simulating-network-conditions.md‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
---
2+
title: Simulating Different Network Conditions
3+
weight: 4
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
## Adding a delay to a TCP connection
10+
11+
The linux `tc` utility can be used to manipulate traffic control settings. First, find the name of interface with the following command.
12+
13+
```bash
14+
ip addr show
15+
```
16+
17+
The output below shows the `ens5` network interface device (NIC) is the device we want to manipulate.
18+
19+
```output
20+
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
21+
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
22+
inet 127.0.0.1/8 scope host lo
23+
valid_lft forever preferred_lft forever
24+
inet6 ::1/128 scope host noprefixroute
25+
valid_lft forever preferred_lft forever
26+
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
27+
link/ether 0a:92:1b:a9:63:29 brd ff:ff:ff:ff:ff:ff
28+
inet 10.248.213.97/26 metric 100 brd 10.248.213.127 scope global dynamic ens5
29+
valid_lft 1984sec preferred_lft 1984sec
30+
inet6 fe80::892:1bff:fea9:6329/64 scope link
31+
valid_lft forever preferred_lft forever
32+
33+
```
34+
35+
Run the following command to add an emulated delay of 10ms on `ens5`.
36+
37+
```bash
38+
sudo tc qdisc add dev ens5 root netem delay 10ms
39+
```
40+
41+
Rerunning the basic TCP test (`iperf3 -c SERVER -V`) with a delay we observe the `Cwnd` size has grew larger to compensate for the longer response time. Additionally, the bitrate has dropped from ~4.9 to ~2.3 `Gbit/sec`.
42+
43+
44+
```output
45+
[ 5] local 10.248.213.97 port 43170 connected to 10.248.213.104 port 5201
46+
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
47+
[ ID] Interval Transfer Bitrate Retr Cwnd
48+
[ 5] 0.00-1.00 sec 282 MBytes 2.36 Gbits/sec 0 8.02 MBytes
49+
[ 5] 1.00-2.00 sec 302 MBytes 2.53 Gbits/sec 0 8.02 MBytes
50+
[ 5] 2.00-3.00 sec 301 MBytes 2.52 Gbits/sec 0 8.02 MBytes
51+
[ 5] 3.00-4.00 sec 302 MBytes 2.54 Gbits/sec 0 8.02 MBytes
52+
[ 5] 4.00-5.00 sec 302 MBytes 2.53 Gbits/sec 0 8.02 MBytes
53+
[ 5] 5.00-6.00 sec 304 MBytes 2.55 Gbits/sec 0 8.02 MBytes
54+
[ 5] 6.00-7.00 sec 302 MBytes 2.53 Gbits/sec 0 8.02 MBytes
55+
[ 5] 7.00-8.00 sec 303 MBytes 2.54 Gbits/sec 0 8.02 MBytes
56+
[ 5] 8.00-9.00 sec 303 MBytes 2.54 Gbits/sec 0 8.02 MBytes
57+
[ 5] 9.00-10.00 sec 301 MBytes 2.53 Gbits/sec 0 8.02 MBytes
58+
- - - - - - - - - - - - - - - - - - - - - - - - -
59+
Test Complete. Summary Results:
60+
[ ID] Interval Transfer Bitrate Retr
61+
[ 5] 0.00-10.00 sec 2.93 GBytes 2.52 Gbits/sec 0 sender
62+
[ 5] 0.00-10.01 sec 2.93 GBytes 2.52 Gbits/sec receiver
63+
CPU Utilization: local/sender 3.4% (0.0%u/3.4%s), remote/receiver 11.0% (0.4%u/10.7%s)
64+
snd_tcp_congestion cubic
65+
rcv_tcp_congestion cubic
66+
67+
iperf Done.
68+
```
69+
70+
### Simulating Packet Loss
71+
72+
To test the resiliency of a distributed application we can add a simulated packet loss of 1%. As opposed to a 10ms delay, this will result in no acknowledgment being received for 1% of packets. Given TCP is a lossless protocol a retry must be sent.
73+
74+
```bash
75+
sudo tc qdisc del dev ens5 root
76+
sudo tc qdisc add dev ens5 root netem loss 1%
77+
```
78+
79+
Rerunning the basic TCP test we observe an increased number of retries (`Retr`) and a corresponding drop in bitrate.
80+
81+
```bash
82+
iperf3 -c SERVER -V
83+
```
84+
```output
85+
Test Complete. Summary Results:
86+
[ ID] Interval Transfer Bitrate Retr
87+
[ 5] 0.00-10.00 sec 4.41 GBytes 3.78 Gbits/sec 5030 sender
88+
[ 5] 0.00-10.00 sec 4.40 GBytes 3.78 Gbits/sec receiver
89+
```
90+
91+
Please see the `tc` [user documentation](https://man7.org/linux/man-pages/man8/tc.8.html) for the different ways to simulate different perturbation and your systems resiliency to such events.

‎content/learning-paths/servers-and-cloud-computing/microbenchmark-network-iperf3/tuning.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
---
2+
title: Tuning Kernel Parameters
3+
weight: 5
4+
5+
### FIXED, DO NOT MODIFY
6+
layout: learningpathall
7+
---
8+
9+
### Connecting from Local Machine
10+
11+
Now we can observe ways to mitigate performance degradation due to events such as packet loss. In this example, I will connect to the AWS server node from my local machine to demonstrate a longer response time. Please check the `iperf3` installation guide on the [official documentation](https://iperf.fr/iperf-download.php) if you're not using Ubuntu. As the output below shows we have a larger round trip time in excess of 40ms.
12+
13+
```output
14+
5 packets transmitted, 5 packets received, 0.0% packet loss
15+
round-trip min/avg/max/stddev = 44.896/46.967/49.279/1.444 ms
16+
```
17+
18+
Running a standard TCP client connection with the `iperf3 -c SERVER -V` command shows an average bitrate of 157 Mbps.
19+
20+
```output
21+
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
22+
...
23+
Test Complete. Summary Results:
24+
[ ID] Interval Transfer Bitrate
25+
[ 8] 0.00-10.01 sec 187 MBytes 157 Mbits/sec sender
26+
[ 8] 0.00-10.03 sec 187 MBytes 156 Mbits/sec receiver
27+
```
28+
29+
30+
31+
### Modify kernel parameters
32+
33+
On the server, we can configure linux kernel runtime parameters with the `sysctl` command.
34+
35+
There are a plenthora of dials to tune that relate to performance and security. The following command can be used to list all available dials. The corresponding [kernel documentation](https://docs.kernel.org/networking/ip-sysctl.html#ip-sysctl) can provide a more detailed description of each parameter.
36+
37+
```bash
38+
sysctl -a | grep tcp
39+
```
40+
41+
{{% notice Note %}}
42+
Depending on your operating system, some parameters may not be available. For example on AWS Ubuntu 22.04 LTS only the `cubic` and `reno` congestion control algorithms are available.
43+
```bash
44+
net.ipv4.tcp_available_congestion_control = reno cubic
45+
```
46+
{{% /notice %}}
47+
48+
49+
We can increase the read and write max buffer sizes of the kernel on the server to enable more data to be held. This is at the tradeoff of increased memory utilisation. run the following commands from the server.
50+
51+
```bash
52+
sudo sysctl net.core.rmem_max=134217728 # default = 212992
53+
sudo sysctl net.core.wmem_max=134217728 # default = 212992
54+
```
55+
56+
Restart the `iperf3` server. Run the `iperf3 -c SERVER -V` command from the client leads to significantly improved bitrate with no modification on the client side.
57+
58+
```output
59+
Test Complete. Summary Results:
60+
[ ID] Interval Transfer Bitrate
61+
[ 8] 0.00-10.00 sec 308 MBytes 258 Mbits/sec sender
62+
[ 8] 0.00-10.03 sec 307 MBytes 257 Mbits/sec receiver
63+
64+
```
65+
66+
This learning path serves as an introduction to microbenchmarking and performance tuning. Which parameters to adjust depends on your own use case and non-functional performance requirements of your system.

0 commit comments

Comments
 (0)