Skip to content

Commit d38d43f

Browse files
ywc689ywc689
authored
Merge pull request #846 from ywc689/v1.9.2
Release v1.9.2
2 parents 793e11b + 018c908 commit d38d43f

File tree

10 files changed

+15325
-6
lines changed

10 files changed

+15325
-6
lines changed

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -253,6 +253,8 @@ Our test shows the forwarding speed (pps) of DPVS is several times than LVS and
253253

254254
![performance](./pic/performance.png)
255255

256+
Click [here](./test/release/v1.9.2/performance.md) for the lastest performance data.
257+
256258
# License
257259

258260
Please refer to the [License](./LICENSE.md) file for details.

‎src/VERSION‎

Lines changed: 47 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,53 @@
1-
#!/bin/sh -
1+
#!/bin/sh
22
# program: dpvs
3-
# Jan 4, 2022
4-
#
5-
# Rebase v1.8.12 to v1.9.0
3+
# Jul 19, 2022
64
#
75

86
export VERSION=1.9
9-
export RELEASE=1.alpha
7+
export RELEASE=2
108

119
echo $VERSION-$RELEASE
10+
11+
## Features
12+
#* Dpvs: Add ipset framework and 12 set types.
13+
#* Dpvs: Add an ipset based tc classifier -- tc_cls_ipset.
14+
#* Dpvs: Add l2/l3/l4 header parse apis for mbuf.
15+
#* Dpvs: Add config option "dedicated_queues" for bonding mode 4 (802.3ad).
16+
#* Dpvs: Isolate kni ingress traffic using kni address flow.
17+
#* Dpvs: Update rss reta table according to configured workers after device bootup.
18+
#* Dpvs: Expire quiescent connections after realserver was removed.
19+
#* Dpvs: Make async log mempool size and log timestamp configurable.
20+
#* Dpvs: Enable dpvs log only when macro CONFIG_DPVS_LOG is defined.
21+
#* Dpvs: Make debug fields in dp_vs_conn configurable for memory optimization.
22+
#* Toa: Support linux kernel verison v5.7.0+.
23+
#* Keepalived: Add UDP_CHECK health checker.
24+
#* Test: Add flame graph scripts for performance tests.
25+
#* Test: Add performance benchmark tests of DPVS v1.9.2.
26+
#* Docs: Update some docs.
27+
#
28+
## Bugfix
29+
#* Dpvs: Fix a crash problem when timer is scheduled from within another timer's callback.
30+
#* Dpvs: Fix a crash problem caused by incorrect mbuf pointer in IPv4 fragmentation.
31+
#* Dpvs: Fix a crash problem caused by using unsafe list macro in conhash.
32+
#* Dpvs: Fix the fullnat tcp forwarding failure problem when defer_rs_syn enabled.
33+
#* Dpvs: Fix the ipvs rr/wrr/wlc problem of uneven load distribution across dests.
34+
#* Dpvs: Fix the weight ratio update problem in conhash schedule algorithm.
35+
#* Dpvs: Send tcp rst to both ends when snat conneciton expired.
36+
#* Dpvs: Use unified dest validation in mh scheduling algorithm.
37+
#* Dpvs: Fix the icmp sending failure problem when no route cached in mbuf.
38+
#* Dpvs: Fix the compiling failure problem when icmp debug is enabled.
39+
#* Dpvs: Fix the icmpv6 sending failure problem caused by incorrect mtu.
40+
#* Dpvs: Fix icmpv6 checksum error caused by incorrect payload length endian in ipv6 header.
41+
#* Dpvs: Fix the checksum problem caused by incorrect netif interface.
42+
#* Dpvs: Fix the bonding mode 4 problem caused by LACP failure.
43+
#* Dpvs: Fix the ipv6 neighbour ring full problem to kni isolated lcore.
44+
#* Dpvs: Fix the list/edit problem for MATCH type service (snat service).
45+
#* Dpvs: Fix incorrect oifname typo in MATCH type.
46+
#* Dpvs: Fix the dpvs worker blocking problem when async log is enabled.
47+
#* Dpvs: Fix some memory overflow problems when log messages are truncated.
48+
#* Dpvs: Fix the msg sequence duplicated problem in ipvs allow list.
49+
#* Dpvs: Fix the incorrect uoa client source port problem in fnat64.
50+
#* Uoa: Fix uoa data parse problem of ipv4 opp, and add a module parameter to parse uoa data in netfilter forward chain.
51+
#* Keepalived: Fix an exit problem when reload.
52+
#* Keepalived: Fix some compile problems found on ubuntu.
53+
#* Ipvsadm: Use correct flag in listing ipvs connections.

‎test/flameGraph/run.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ outfile=$2
99
[ ! -f $infile ] && echo -e "can't find input perf.data file $inflie, please use `perf record` to generate it"
1010
[ _$outfile = _ ] && echo -e "invalid out.file name" && exit 1
1111

12-
perf script -i perf.data &> perf.unfold
12+
perf script -i $infile > perf.unfold
1313
./stackcollapse-perf.pl perf.unfold &> perf.folded
1414
./flamegraph.pl perf.folded > $outfile.svg
1515
rm -f perf.unfold

‎test/release/v1.9.2/performance.data‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
* TCP CPS/CC Tests
2+
workers,cps;ipackets/pps,opackets/pps,ibytes/Bps,obytes/Bps;connections;pktRx,pktTx,bitsRx,bitsTx,dropTx
3+
1,200000;1211533,1211847,99143458,102396220;1472000;600020,599988,393618488,382378808,0
4+
2,360000;2166961,2166955,177320954,183100299;2701000;1072119,1076034,703360424,685830112,0
5+
4,660000;3960726,3960788,324114391,334680450;4941000;1980045,1980054,1298916032,1261958232,0
6+
8,1060000;6360626,6360628,520511025,537472046;7949000;3180092,3180068,2086137680,2026768232,0
7+
10,1240000;7440784,7440727,608903706,628741279;9299000;3718514,3719316,2439334056,2370499504,0
8+
16,1070000;6420639,6420548,525422150,542537169;8019000;3210000,3209989,2105751088,2045839664,0 (cross-numa-node)
9+
10+
* UDP PPS Tests
11+
workers,connections;ipackets/pps,opackets/pps,ibytes/Bps,obytes/Bps;pktRx,pktTx,bitsRx,bitsTx,dropTx
12+
1,2900;2900244,2900221,174014668,174013684;1449993,1450000,695996816,498800000,0
13+
2,5000;5000418,5000370,300024968,300022497;2499954,2500000,1199978096,860000000,0
14+
4,9200;9201066,9201048,552063906,552062986;4486101,4600001,2153329128,1582400344,0
15+
8,9450;9451027,9451004,567061568,567060365;4723923,4724932,2267483216,1625376608,0
16+
17+
* Throughput Tests
18+
workers,connections;ipackets/pps,opackets/pps,ibytes/Bps,obytes/Bps;pktRx,pktTx,bitsRx,bitsTx,dropTx
19+
1,1000;1424608,1424599,1215824068,1215816616;712263,712285,4866168760,4860632840,0
20+
2,1000;1424748,1424738,1215947746,1215939706;712247,712263,4866065328,4860482712,0
21+
4,1000;1424876,1424870,1216052235,1216047912;712258,712238,4866134600,4860312112,0
22+
8,1000;1424788,1424787,1215971428,1215970249;712261,712260,4866160976,4860462240,0

‎test/release/v1.9.2/performance.md‎

Lines changed: 257 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,257 @@
1+
DPVS v1.9.2 Performance Tests
2+
===
3+
4+
* [Test Platform](#platform)
5+
* [TCP CPS/CC Tests](#cps/cc)
6+
* [UDP PPS Tests](#pps)
7+
* [Throughput Tests](#throughput)
8+
9+
10+
<a id='platform'/>
11+
12+
# Test Platform
13+
14+
The performance of DPVS v1.9.2 is examined on two physical servers, one serves as DPVS server, and the other as both backend server(RS) and client(Client). RS and Client take advantages of [dperf](https://github.com/baidu/dperf), a high performance benchmark tool based on DPDK developed by baidu. The dperf server process and dperf client process use isolated NIC interfaces, CPU coers, and hugepage memory in order to run both processes on a single node.
15+
16+
### DPVS Server
17+
18+
+ CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz, 2 Sockets, 12 Cores per Socket, 2 Threads per Core
19+
+ Memory: 188G Bytes
20+
+ NIC: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2
21+
+ OS: Centos 7.6
22+
+ DPVS: v1.9.2
23+
24+
### Dperf Server/Client
25+
26+
+ CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz, 2 Sockets, 10 Cores per Socket, 2 Threads per Core
27+
+ Memory: 62G Bytes
28+
+ NIC: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2
29+
+ OS: Centos 7.6
30+
+ Dperf: v1.2.0
31+
32+
<a id='cps/cc'/>
33+
34+
# TCP CPS/CC Tests
35+
36+
CPS(Connections per Second) and CC (Concurrent Connections) tests are performed by using the extreme small sized packets (payload_size=1) and variable `cps` of dperf clients. We gradually increase the `cps` of dperf clients until packet loss is seen in DPVS, and then the corresponding CPS and CC are the performance data that we need.
37+
38+
### Dperf Client
39+
40+
```
41+
mode client
42+
cpu 8-15
43+
slow_start 60
44+
tx_burst 128
45+
launch_num 10
46+
payload_size 1
47+
duration 90s
48+
protocol tcp
49+
cps [refer to performance data]
50+
port 0000:04:00.0 192.168.0.30 192.168.7.254
51+
client 192.168.3.0 50
52+
server 192.168.5.1 8
53+
listen 80 1
54+
```
55+
56+
### Dperf Server
57+
58+
```
59+
mode server
60+
cpu 0-7
61+
tx_burst 128
62+
payload_size 1
63+
duration 100d
64+
port 0000:04:00.1 192.168.1.30 192.168.7.254
65+
client 192.168.0.28 1
66+
client 192.168.1.28 1
67+
client 192.168.1.30 1
68+
client 192.168.3.0 200
69+
server 192.168.6.100 8
70+
listen 80 1
71+
```
72+
73+
### DPVS
74+
75+
+ Service: 192.168.5.[1-8]:80, TCP, FullNAT, rr, syn-proxy off
76+
+ Local IP: 192.168.3.[100-149]
77+
78+
```
79+
TCP 192.168.5.1:80 rr
80+
-> 192.168.6.100:80 FullNat 100 0 4
81+
-> 192.168.6.101:80 FullNat 100 0 4
82+
-> 192.168.6.102:80 FullNat 100 0 2
83+
-> 192.168.6.103:80 FullNat 100 0 1
84+
-> 192.168.6.104:80 FullNat 100 0 0
85+
-> 192.168.6.105:80 FullNat 100 0 0
86+
-> 192.168.6.106:80 FullNat 100 0 1
87+
-> 192.168.6.107:80 FullNat 100 0 2
88+
TCP 192.168.5.2:80 rr
89+
-> 192.168.6.100:80 FullNat 100 0 1
90+
-> 192.168.6.101:80 FullNat 100 0 2
91+
...
92+
...
93+
```
94+
95+
### Performance Data
96+
97+
| workers | cps | ipackets/pps | opackets/pps | ibytes/Bps | obytes/Bps | connections | dperf:pktRx | dperf:pktTx | dperf:bitsRx | dperf:bitsTx | dperf:dropTx |
98+
| ------- | --------- | ------------ | ------------ | ----------- | ----------- | ----------- | ----------- | ----------- | ------------- | ------------- | ------------ |
99+
| 1 | 200,000 | 1,211,533 | 1,211,847 | 99,143,458 | 102,396,220 | 1,472,000 | 600,020 | 599,988 | 393,618,488 | 382,378,808 | 0 |
100+
| 2 | 360,000 | 2,166,961 | 2,166,955 | 177,320,954 | 183,100,299 | 2,701,000 | 1,072,119 | 1,076,034 | 703,360,424 | 685,830,112 | 0 |
101+
| 4 | 660,000 | 3,960,726 | 3,960,788 | 324,114,391 | 334,680,450 | 4,941,000 | 1,980,045 | 1,980,054 | 1,298,916,032 | 1,261,958,232 | 0 |
102+
| 8 | 1,060,000 | 6,360,626 | 6,360,628 | 520,511,025 | 537,472,046 | 7,949,000 | 3,180,092 | 3,180,068 | 2,086,137,680 | 2,026,768,232 | 0 |
103+
| 10 | 1,240,000 | 7,440,784 | 7,440,727 | 608,903,706 | 628,741,279 | 9,299,000 | 3,718,514 | 3,719,316 | 2,439,334,056 | 2,370,499,504 | 0 |
104+
| 16 | 1,070,000 | 6,420,639 | 6,420,548 | 525,422,150 | 542,537,169 | 8,019,000 | 3,210,000 | 3,209,989 | 2,105,751,088 | 2,045,839,664 | 0 |
105+
106+
107+
![CPS/CC](./pics/tcp_cps.png)
108+
109+
In case of 8-workers, DPVS v1.9.2 can establish **1,000,000 new connections per second**, and hold **8,000,000 concurrent connections** in the meanwhile. The performance gains approximately linearly when worker number is below 10. But an obvious performance loss is seen in 16-workers. One reason is that DPVS doesn't eliminate all racing conditions in datapath, and the problem gets worse with the increase of worker number. Besides, some DPVS workers are assigned to the CPU cores on NUMA socket different from that of NIC when running with 16-workers. Our DPVS server only has 12 CPU cores available per socket.
110+
111+
Let's make a deep insight into the `cpu-clock` events of DPVS with Linux performance analysis tool `perf`. We build DPVS with debug info and then run CPC/CC tests with 1-worker and 8-workers, with dperf `cps` configured to be 100,000 and 600,000 respectively. The performance flame graphs are shown below.
112+
113+
![perf-flame-worker-1](./pics/worker1.svg)
114+
115+
![perf-flame-worker-8](./pics/worker8.svg)
116+
117+
<a id='pps'/>
118+
119+
# UDP PPS Tests
120+
121+
In PPS tests, dperf clients keep a fixed `cps` of 3k and `keepalive` of 2ms, and adjust concurrent connections `cc` to generate different `pps` traffic. The same with CPS/CC tests, an extreme small payload of 1 byte is used. We use UDP protocol for the tests. Besides, `tx_burst` in dperf client is set to 1 to reduce traffic surge.
122+
123+
### Dperf Client
124+
125+
```
126+
mode client
127+
cpu 8-15
128+
slow_start 60
129+
tx_burst 128
130+
launch_num 1
131+
payload_size 1
132+
duration 90s
133+
protocol udp
134+
cps 3k
135+
cc [refer to performance data]
136+
keepalive 2ms
137+
port 0000:04:00.0 192.168.0.30 192.168.7.254
138+
client 192.168.3.0 50
139+
server 192.168.5.1 8
140+
listen 80 1
141+
```
142+
### Dperf Server
143+
144+
```
145+
mode server
146+
cpu 0-7
147+
tx_burst 128
148+
payload_size 1
149+
duration 100d
150+
protocol udp
151+
keepalive 10s
152+
port 0000:04:00.1 192.168.1.30 192.168.7.254
153+
client 192.168.0.28 1
154+
client 192.168.1.28 1
155+
client 192.168.1.30 1
156+
client 192.168.3.0 200
157+
server 192.168.6.100 8
158+
listen 80 1
159+
```
160+
161+
### DPVS
162+
163+
+ Service: 192.168.5.[1-8]:80, UDP, FullNAT, rr, uoa off
164+
+ Local IP: 192.168.3.[100-149]
165+
166+
```
167+
UDP 192.168.5.1:80 rr
168+
-> 192.168.6.100:80 FullNat 100 0 0
169+
-> 192.168.6.101:80 FullNat 100 0 0
170+
-> 192.168.6.102:80 FullNat 100 0 0
171+
-> 192.168.6.103:80 FullNat 100 0 0
172+
-> 192.168.6.104:80 FullNat 100 0 0
173+
-> 192.168.6.105:80 FullNat 100 0 0
174+
-> 192.168.6.106:80 FullNat 100 0 0
175+
-> 192.168.6.107:80 FullNat 100 0 0
176+
UDP 192.168.5.2:80 rr
177+
-> 192.168.6.100:80 FullNat 100 0 0
178+
-> 192.168.6.101:80 FullNat 100 0 0
179+
...
180+
...
181+
```
182+
183+
### Performance Data
184+
185+
| workers | connections | ipackets/pps | opackets/pps | ibytes/Bps | obytes/Bps | dperf:pktRx | dperf:pktTx | dperf:bitsRx | dperf:bitsTx | dperf:dropTx |
186+
| ------- | ----------- | ------------ | ------------ | ----------- | ----------- | ----------- | ----------- | ------------- | ------------- | ------------ |
187+
| 1 | 2,900 | 2,900,244 | 2,900,221 | 174,014,668 | 174,013,684 | 1,449,993 | 1,450,000 | 695,996,816 | 498,800,000 | 0 |
188+
| 2 | 5,000 | 5,000,418 | 5,000,370 | 300,024,968 | 300,022,497 | 2,499,954 | 2,500,000 | 1,199,978,096 | 860,000,000 | 0 |
189+
| 4 | 9,200 | 9,201,066 | 9,201,048 | 552,063,906 | 552,062,986 | 4,486,101 | 4,600,001 | 2,153,329,128 | 1,582,400,344 | 0 |
190+
| 8 | 9,450 | 9,451,027 | 9,451,004 | 567,061,568 | 567,060,365 | 4,723,923 | 4,724,932 | 2,267,483,216 | 1,625,376,608 | 0 |
191+
192+
![PPS](./pics/udp_pps.png)
193+
194+
As shown above, DPVS v1.9.2 can reach the peak of PPS (i.e, about 9,000,000 PPS) with 4-workers in the tests. We may need a 25G/100G NIC for a higher PPS test.
195+
196+
<a id='throughput'/>
197+
198+
# Throughput Tests
199+
200+
In throughput tests, dperf clients keep a fixed `cps` of 400 and `keepalive` of 1ms, and adjust concurrent connections `cc` to generate different `pps` traffic. The `payload_size` of both dperf server and dperf client are set to 800 bytes, and TCP protocol is used.
201+
202+
### Dperf Client
203+
204+
```
205+
mode client
206+
cpu 8-15
207+
slow_start 60
208+
tx_burst 128
209+
launch_num 10
210+
payload_size 800
211+
duration 90s
212+
protocol tcp
213+
cps 400
214+
cc [refer to performance data]
215+
keepalive 1ms
216+
port 0000:04:00.0 192.168.0.30 192.168.7.254
217+
client 192.168.3.0 50
218+
server 192.168.5.1 8
219+
listen 80 1
220+
```
221+
222+
### Dperf Server
223+
224+
```
225+
mode server
226+
cpu 0-7
227+
tx_burst 128
228+
payload_size 800
229+
duration 100d
230+
protocol tcp
231+
keepalive 10s
232+
port 0000:04:00.1 192.168.1.30 192.168.7.254
233+
client 192.168.0.28 1
234+
client 192.168.1.28 1
235+
client 192.168.1.30 1
236+
client 192.168.3.0 200
237+
server 192.168.6.100 8
238+
listen 80 1
239+
```
240+
241+
## DPVS
242+
243+
DPVS configurations are the same with the `TCP CPS/CC Tests`.
244+
245+
246+
### Performance Data
247+
248+
| workers | connections | ipackets/pps | opackets/pps | ibytes/Bps | obytes/Bps | dperf:pktRx | dperf:pktTx | dperf:bitsRx | dperf:bitsTx | dperf:dropTx |
249+
| ------- | ----------- | ------------ | ------------ | ------------- | ------------- | ----------- | ----------- | ------------- | ------------- | ------------ |
250+
| 1 | 1,000 | 1,424,608 | 1,424,599 | 1,215,824,068 | 1,215,816,616 | 712,263 | 712,285 | 4,866,168,760 | 4,860,632,840 | 0 |
251+
| 2 | 1,000 | 1,424,748 | 1,424,738 | 1,215,947,746 | 1,215,939,706 | 712,247 | 712,263 | 4,866,065,328 | 4,860,482,712 | 0 |
252+
| 4 | 1,000 | 1,424,876 | 1,424,870 | 1,216,052,235 | 1,216,047,912 | 712,258 | 712,238 | 4,866,134,600 | 4,860,312,112 | 0 |
253+
| 8 | 1,000 | 1,424,788 | 1,424,787 | 1,215,971,428 | 1,215,970,249 | 712,261 | 712,260 | 4,866,160,976 | 4,860,462,240 | 0 |
254+
255+
![Throughput](./pics/tcp_throughput.png)
256+
257+
As shown above, DPVS v1.9.2 fills with ease the full bandwith of 10G NIC using only one worker.

‎test/release/v1.9.2/pics/tcp_cps.png‎

42.7 KB
Loading

‎test/release/v1.9.2/pics/tcp_throughput.png‎

13.1 KB
Loading

‎test/release/v1.9.2/pics/udp_pps.png‎

24.8 KB
Loading

0 commit comments

Comments
 (0)