Skip to content

Commit 6266bcc

Browse files
sionsmithsionsmith
committed
network performance testing
1 parent 1a4dd47 commit 6266bcc

File tree

14 files changed

+1335
-104
lines changed

14 files changed

+1335
-104
lines changed

‎stable/network-benchmark-testing/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# Testing Kubernetes cluster network performance
2+
3+
A standardized benchmark to measure Kubernetes networking performance on multiple host platforms and network stacks.
4+
5+
### Usage
6+
The test suite can be executed via a single Go binary that triggers all the automated testing present on the netperf Docker container.
7+
8+
***NOTE*** A minimum of two Kubernetes worker nodes are necessary for this test. If using minikube you must enable the Docker registry as the image is needed on multiple hosts.
9+
10+
```shell
11+
#create 2 node cluster
12+
minikube start --nodes 2 --cpus=6 --memory=20019 --kubernetes-version=v1.21.0
13+
14+
#enable the docker registry and push image to make available on all nodes
15+
minikube addons enable registry
16+
docker run --rm -it --network=host alpine ash -c "apk add socat && socat TCP-LISTEN:5000,reuseaddr,fork TCP:$(minikube ip):5000" &
17+
docker build ./docker -t netperf:latest
18+
docker tag netperf:latest localhost:5000/netperf:latest
19+
docker push localhost:5000/netperf:latest
20+
21+
#run the tests
22+
cd docker && go run ./launch.go --iterations 1
23+
```
24+
25+
### Kubernetes Network benchmark results
26+
The tests will take some time, after ~35mins you should see the following output.
27+
28+
```shell
29+
│ ALL TESTCASES AND MSS RANGES COMPLETE - GENERATING CSV OUTPUT │
30+
│ MSS , Maximum, 96, 160, 224, 288, 352, 416, 480, 544, 608, 672, 736, 800, 864, 928, 992, 1056, 1120, 1184, 1248, 1312, 1376, 1440, │
31+
│ 1 iperf TCP. Same VM using Pod IP ,43787.000000,40615,41787,41644,34696,38652,40741,39204,33203,39646,24538,43787,32438,27078,16714,33450,41896,35655,41853,40867,41655,40782,41521, │
32+
│ 2 iperf TCP. Same VM using Virtual IP ,45900.000000,37386,36459,37758,40965,41688,38859,39247,41394,37100,35232,40901,35395,36015,41664,45293,44693,40741,42982,41322,44138,40716,45900, │
33+
│ 3 iperf TCP. Remote VM using Pod IP ,27177.000000,26005,24857,26076,24670,24648,24399,24301,23858,23164,21852,24095,24304,25836,24940,25639,25975,22841,24274,24734,26783,24797,27177, │
34+
│ 4 iperf TCP. Remote VM using Virtual IP ,26820.000000,24622,24673,24447,23553,22981,23879,23464,24202,25872,24083,26820,23440,25935,24738,25447,25284,23834,25140,24318,25430,24247,25509, │
35+
│ 5 iperf TCP. Hairpin Pod to own Virtual IP ,43544.000000,38842,39599,38957,39019,39219,38770,39179,39829,41485,39045,43544,39219,38747,42666,40890,41004,37079,37665,37634,41949,37629,42241, │
36+
│ 6 iperf SCTP. Same VM using Pod IP ,2547.000000,2472,2512,2505,2495,2502,2499,2531,1329,1653,1745,1568,1265,175,1259,506,484,2249,2135,2254,2332,2414,2547, │
37+
│ 7 iperf SCTP. Same VM using Virtual IP ,2491.000000,2491,2451,2435,2441,2444,2429,2411,1229,1680,1490,1523,194,300,525,698,1557,1957,2119,2060,2250,2330,2408, │
38+
│ 8 iperf SCTP. Remote VM using Pod IP ,1467.000000,1319,1416,1384,1361,1467,1462,1393,544,1073,1079,921,1020,844,1150,1103,382,1205,1352,1408,1422,1418,1437, │
39+
│ 9 iperf SCTP. Remote VM using Virtual IP ,1540.000000,1438,1409,1216,1364,1534,1451,1540,948,863,1167,995,726,767,736,1240,1231,1191,1400,1441,1476,1441,1517, │
40+
│ 10 iperf SCTP. Hairpin Pod to own Virtual IP ,2356.000000,2236,2344,2300,2356,2309,2149,2146,1257,1476,1424,1378,807,364,954,1447,1409,1866,2025,2143,2210,1517,1131, │
41+
│ 11 iperf UDP. Same VM using Pod IP ,4130.000000,4130, │
42+
│ 12 iperf UDP. Same VM using Virtual IP ,3677.000000,3677, │
43+
│ 13 iperf UDP. Remote VM using Pod IP ,1314.000000,1314, │
44+
│ 14 iperf UDP. Remote VM using Virtual IP ,1281.000000,1281, │
45+
│ 15 netperf. Same VM using Pod IP ,9016.800000,9016.80, │
46+
│ 16 netperf. Same VM using Virtual IP ,0.000000,0, │
47+
│ 17 netperf. Remote VM using Pod IP ,7078.690000,7078.69, │
48+
│ 18 netperf. Remote VM using Virtual IP ,0.000000,0,
49+
```
50+
51+
### Understanding the benchmark data
52+
More information and charting these results [Benchmarking Kubernetes Networking](https://github.com/kubernetes/perf-tests/tree/master/network/benchmarks/netperf#output-raw-csv-data)

‎stable/network-benchmark-testing/docker/Dockerfile‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
ARG GOLANG_VERSION=1.18
2+
FROM golang:${GOLANG_VERSION} as builder
3+
WORKDIR /workspace
4+
5+
COPY nptests/nptest.go nptest.go
6+
COPY go.sum go.sum
7+
COPY go.mod go.mod
8+
9+
RUN go build -o nptests
10+
11+
FROM debian:jessie
12+
ENV LD_LIBRARY_PATH=/usr/local/lib
13+
14+
# install binary and remove cache
15+
RUN apt-get update \
16+
&& apt-get install -y curl wget net-tools gcc make libsctp-dev \
17+
&& rm -rf /var/lib/apt/lists/*
18+
RUN mkdir -p /tmp
19+
20+
21+
# Download and build iperf3 from sources
22+
RUN curl -LO https://downloads.es.net/pub/iperf/iperf-3.1.tar.gz && tar zxf iperf-3.1.tar.gz
23+
RUN cd iperf-3.1 && ./configure --prefix=/usr/local --bindir /usr/local/bin && make && make install
24+
25+
# Download and build netperf from sources
26+
RUN curl -LO https://github.com/HewlettPackard/netperf/archive/netperf-2.7.0.tar.gz && tar -xzf netperf-2.7.0.tar.gz && mv netperf-netperf-2.7.0/ netperf-2.7.0
27+
RUN cd netperf-2.7.0 && ./configure --prefix=/usr/local --bindir /usr/local/bin && make && make install
28+
29+
COPY --from=builder /workspace/nptests /usr/bin/
30+
31+
ENTRYPOINT ["nptests"]

‎stable/network-benchmark-testing/docker/go.mod‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
module k8s.io/perf-tests/network
2+
3+
go 1.13
4+
5+
require (
6+
github.com/davecgh/go-spew v1.1.1 // indirect
7+
github.com/ghodss/yaml v1.0.1-0.20180820084758-c7ce16629ff4 // indirect
8+
github.com/gogo/protobuf v1.1.2-0.20181010092945-fd322a3c4963 // indirect
9+
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect
10+
github.com/google/btree v1.0.0 // indirect
11+
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf // indirect
12+
github.com/googleapis/gnostic v0.2.3-0.20180520015035-48a0ecefe2e4 // indirect
13+
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 // indirect
14+
github.com/imdario/mergo v0.3.6 // indirect
15+
github.com/json-iterator/go v1.1.6-0.20180914014843-2433035e5132 // indirect
16+
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
17+
github.com/modern-go/reflect2 v1.0.1 // indirect
18+
github.com/peterbourgon/diskv v2.0.2-0.20180312054125-0646ccaebea1+incompatible // indirect
19+
github.com/spf13/pflag v1.0.3 // indirect
20+
github.com/stretchr/testify v1.5.1 // indirect
21+
golang.org/x/oauth2 v0.0.0-20160902055913-3c3a985cb79f // indirect
22+
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2 // indirect
23+
google.golang.org/appengine v1.6.6 // indirect
24+
gopkg.in/inf.v0 v0.9.1 // indirect
25+
k8s.io/api v0.0.0-20181011064954-26c7a45db378
26+
k8s.io/apimachinery v0.0.0-20181011064652-56cf97ad69c7
27+
k8s.io/client-go v0.0.0-20181011105049-9b03088ac34f
28+
)

‎stable/network-benchmark-testing/docker/go.sum‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
2+
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
3+
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
4+
github.com/ghodss/yaml v1.0.1-0.20180820084758-c7ce16629ff4 h1:PaTU+9BARuIOAz1ixvps39DJjfq/SxOj3axzIlh7nFo=
5+
github.com/ghodss/yaml v1.0.1-0.20180820084758-c7ce16629ff4/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
6+
github.com/gogo/protobuf v1.1.2-0.20181010092945-fd322a3c4963 h1:qxvPMOEHQtKCpBDHjx3NiCnM3NfMEyeA9c72+IljOP4=
7+
github.com/gogo/protobuf v1.1.2-0.20181010092945-fd322a3c4963/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
8+
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
9+
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
10+
github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg=
11+
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
12+
github.com/google/btree v1.0.0 h1:0udJVsspx3VBr5FwtLhQQtuAsVc79tTq0ocGIPAU6qo=
13+
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
14+
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf h1:+RRA9JqSOZFfKrOeqr2z77+8R2RKyh8PG66dcu1V0ck=
15+
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI=
16+
github.com/googleapis/gnostic v0.2.3-0.20180520015035-48a0ecefe2e4 h1:Z09Qt6AGDtg0cC/YgnX/iymzIqmZf5aasP5JZFxmkNQ=
17+
github.com/googleapis/gnostic v0.2.3-0.20180520015035-48a0ecefe2e4/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY=
18+
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM=
19+
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
20+
github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
21+
github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
22+
github.com/json-iterator/go v1.1.6-0.20180914014843-2433035e5132 h1:Y1mRUIuPBTUZfUBQW0V9iItFGJnksjotNXSOzTtxfIo=
23+
github.com/json-iterator/go v1.1.6-0.20180914014843-2433035e5132/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
24+
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
25+
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
26+
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
27+
github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
28+
github.com/peterbourgon/diskv v2.0.2-0.20180312054125-0646ccaebea1+incompatible h1:FhnA4iH8T/yYW+AolPONZjGE897wxj3MAzfEbrZkSYw=
29+
github.com/peterbourgon/diskv v2.0.2-0.20180312054125-0646ccaebea1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
30+
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
31+
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
32+
github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
33+
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
34+
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
35+
github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4=
36+
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
37+
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2 h1:VklqNMn3ovrHsnt90PveolxSbWFaJdECFbxSq0Mqo2M=
38+
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
39+
golang.org/x/net v0.0.0-20190603091049-60506f45cf65 h1:+rhAzEzT3f4JtomfC371qB+0Ola2caSKcY69NUBZrRQ=
40+
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
41+
golang.org/x/oauth2 v0.0.0-20160902055913-3c3a985cb79f h1:VWt05OS3Al9w09GSPgltoHP90whAHlpik/Bys7HVEDE=
42+
golang.org/x/oauth2 v0.0.0-20160902055913-3c3a985cb79f/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
43+
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU=
44+
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
45+
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
46+
golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
47+
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
48+
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2 h1:+DCIGbF/swA92ohVg0//6X2IVY3KZs6p9mix0ziNYJM=
49+
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
50+
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
51+
google.golang.org/appengine v1.6.6 h1:lMO5rYAqUxkmaj76jAkRUvt5JZgFymx/+Q5Mzfivuhc=
52+
google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
53+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
54+
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
55+
gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
56+
gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
57+
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
58+
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
59+
k8s.io/api v0.0.0-20181011064954-26c7a45db378 h1:lBCdUUoFs8PEMxhQoSNnVXxpKcrNrzuOw4/ofBd+0do=
60+
k8s.io/api v0.0.0-20181011064954-26c7a45db378/go.mod h1:iuAfoD4hCxJ8Onx9kaTIt30j7jUFS00AXQi6QMi99vA=
61+
k8s.io/apimachinery v0.0.0-20181011064652-56cf97ad69c7 h1:l1WAFAYv7ba/kpouP/gXH+l3ZqsRKbGH8F1VMDYmIbM=
62+
k8s.io/apimachinery v0.0.0-20181011064652-56cf97ad69c7/go.mod h1:ccL7Eh7zubPUSh9A3USN90/OzHNSVN6zxzde07TDCL0=
63+
k8s.io/client-go v0.0.0-20181011105049-9b03088ac34f h1:+izcn/RUZFQiB2q/wTE/xgLViQ5iFIZhlQParzg2ha4=
64+
k8s.io/client-go v0.0.0-20181011105049-9b03088ac34f/go.mod h1:7vJpHMYJwNQCWgzmNV+VYUl1zCObLyodBc8nIyt8L5s=

0 commit comments

Comments
 (0)