Skip to content

Commit ec49b70

Browse files
author
Matt Pryor
authored
Merge pull request #5 from stackhpc/update/fio-tests
Add fio benchmark
2 parents 7348a52 + 96f44bf commit ec49b70

File tree

13 files changed

+739
-74
lines changed

13 files changed

+739
-74
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,4 @@ __pycache__
22
kubeconfig*
33
.python-version
44
*.egg-info
5+
venv

‎README.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,3 +228,74 @@ spec:
228228
extraArgs:
229229
- --tclass=96
230230
```
231+
232+
### Filesystem performance characteristics
233+
234+
Runs filesystem performance benchmarking using [fio](https://fio.readthedocs.io). All
235+
available `spec` options are given below. Fio configration options match broadly with
236+
those defined in the fio documentation.
237+
238+
Setting `.spec.volumeClaimTemplate` allows the provision of stable storage using
239+
`PersistentVolumes` provisioned by a [`PersistentVolume`](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
240+
Provisioner.
241+
242+
When `spec.volumeClaimTemplate.accessModes` contains `ReadWriteMany`, this benchmark
243+
will create a single `PersistentVolume` per `BenchmarkSet` iteration, and attach all
244+
worker pods participting in the set (equal to `spec.numWorkers`) to the same volume.
245+
Otherwise, a `PersistentVolume` per-pod is created and attached to each worker pod
246+
participating in the benchmark.
247+
248+
```yaml
249+
apiVersion: perftest.stackhpc.com/v1alpha1
250+
kind: Fio
251+
metadata:
252+
name: fio-filesystem
253+
spec:
254+
# fio benchmark configuration options
255+
direct: 1
256+
iodepth: 8
257+
ioengine: libaio
258+
nrfiles: 1
259+
numJobs: 1
260+
bs: 1M
261+
rw: read
262+
percentageRandom: 100
263+
runtime: 10s
264+
rwmixread: 50
265+
size: 1G
266+
267+
# kube-perftest benchmark configuration
268+
# options
269+
numWorkers: 1
270+
thread: false
271+
hostNetwork: false
272+
273+
# PersistentVolume configuration options
274+
volumeClaimTemplate:
275+
accessModes:
276+
- ReadWriteOnce
277+
storageClassName: csi-cinder
278+
resources:
279+
requests:
280+
storage: 5Gi
281+
```
282+
283+
## Operator development
284+
285+
```
286+
# Install dependencies in a virtual environment
287+
python3 -m venv venv
288+
source venv/bin/activate
289+
pip install -U pip
290+
pip install -r python/requirements.txt
291+
pip install -e python
292+
293+
# Set the default image tag
294+
export KUBE_PERFTEST__DEFAULT_IMAGE_TAG=<dev branch name>
295+
296+
# Set the default image pull policy
297+
export KUBE_PERFTEST__DEFAULT_IMAGE_PULL_POLICY=Always
298+
299+
# Run the operator
300+
kopf run -m perftest.operator -A
301+
```

‎examples/filesystem/fio-csi-cinder-nvme-set.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
apiVersion: perftest.stackhpc.com/v1alpha1
2+
kind: BenchmarkSet
3+
metadata:
4+
name: fio-csi-cinder-nvme
5+
spec:
6+
template:
7+
apiVersion: perftest.stackhpc.com/v1alpha1
8+
kind: Fio
9+
spec:
10+
direct: 1
11+
imagePullPolicy: Always
12+
iodepth: 8
13+
ioengine: libaio
14+
nrfiles: 1
15+
numWorkers: 1
16+
percentageRandom: 100
17+
runtime: 10s
18+
rwmixread: 50
19+
size: 1G
20+
thread: false
21+
hostNetwork: false
22+
volumeClaimTemplate:
23+
accessModes:
24+
- ReadWriteOnce
25+
storageClassName: csi-cinder-nvme
26+
resources:
27+
requests:
28+
storage: 5Gi
29+
permutations:
30+
product:
31+
numWorkers: [2,4]
32+
numJobs: [2,4]
33+
bs: ['1M', '4M']
34+
rw: ['read', 'write']

‎examples/filesystem/fio-csi-cinder-set.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
apiVersion: perftest.stackhpc.com/v1alpha1
2+
kind: BenchmarkSet
3+
metadata:
4+
name: fio-csi-cinder
5+
spec:
6+
template:
7+
apiVersion: perftest.stackhpc.com/v1alpha1
8+
kind: Fio
9+
spec:
10+
direct: 1
11+
imagePullPolicy: Always
12+
iodepth: 8
13+
ioengine: libaio
14+
nrfiles: 1
15+
numWorkers: 1
16+
percentageRandom: 100
17+
runtime: 10s
18+
rw: read
19+
rwmixread: 50
20+
size: 1G
21+
thread: false
22+
hostNetwork: false
23+
volumeClaimTemplate:
24+
accessModes:
25+
- ReadWriteOnce
26+
storageClassName: csi-cinder
27+
resources:
28+
requests:
29+
storage: 5Gi
30+
permutations:
31+
product:
32+
numWorkers: [2,4]
33+
numJobs: [2,4]
34+
bs: ['1M', '4M']

‎examples/filesystem/fio-local-path-set.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
apiVersion: perftest.stackhpc.com/v1alpha1
2+
kind: BenchmarkSet
3+
metadata:
4+
name: fio-local-path
5+
spec:
6+
template:
7+
apiVersion: perftest.stackhpc.com/v1alpha1
8+
kind: Fio
9+
spec:
10+
direct: 1
11+
imagePullPolicy: Always
12+
iodepth: 8
13+
ioengine: libaio
14+
nrfiles: 1
15+
numWorkers: 1
16+
percentageRandom: 100
17+
runtime: 10s
18+
rw: read
19+
rwmixread: 50
20+
size: 1G
21+
thread: false
22+
hostNetwork: false
23+
volumeClaimTemplate:
24+
storageClassName: local-path
25+
accessModes:
26+
- ReadWriteOnce
27+
resources:
28+
requests:
29+
storage: 5Gi
30+
permutations:
31+
product:
32+
numJobs: [2,4]
33+
rw: ['read', "write", "randrw", "rw,readwrite"]
34+
bs: ['32k', '1M', '4M']

‎examples/filesystem/fio-rook-nfs-nvme.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
apiVersion: perftest.stackhpc.com/v1alpha1
2+
kind: BenchmarkSet
3+
metadata:
4+
name: fio-rook-nfs
5+
spec:
6+
template:
7+
apiVersion: perftest.stackhpc.com/v1alpha1
8+
kind: Fio
9+
spec:
10+
direct: 1
11+
imagePullPolicy: Always
12+
iodepth: 8
13+
ioengine: libaio
14+
nrfiles: 1
15+
numWorkers: 1
16+
percentageRandom: 100
17+
runtime: 10s
18+
rw: read
19+
rwmixread: 50
20+
size: 100M
21+
thread: false
22+
hostNetwork: false
23+
volumeClaimTemplate:
24+
storageClassName: rook-nfs-share1
25+
accessModes:
26+
- ReadWriteMany
27+
resources:
28+
requests:
29+
storage: 50Gi
30+
permutations:
31+
product:
32+
numJobs: [2,4]
33+
rw: ['read', "write", "randrw", "rw,readwrite"]
34+
bs: ['32k', '1M', '4M']

‎images/fio/Dockerfile‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
1-
FROM debian:latest
1+
ARG FEDORA_VERSION=37
2+
FROM fedora:${FEDORA_VERSION}
23

3-
ARG FIO_VERSION=3.25-2
4-
RUN apt-get update && \
5-
apt-get install -y tini "fio=$FIO_VERSION" && \
6-
rm -rf /var/lib/apt/lists/*
7-
8-
# Install the setup and run scripts
9-
COPY fio-run.sh /usr/local/bin/fio-run
4+
ARG FIO_VERSION=3.30-2
5+
ARG FEDORA_VERSION
6+
RUN dnf install -y tini fio-${FIO_VERSION}.fc${FEDORA_VERSION} && \
7+
dnf clean all && \
8+
rm -rf /var/cache/dnf
109

1110
ENTRYPOINT ["tini", "-g", "--"]
12-
CMD ["fio-run"]
11+
# Install the setup and run scripts
12+
COPY scripts/* /usr/local/bin/

‎images/fio/fio-run.sh‎

Lines changed: 0 additions & 65 deletions
This file was deleted.

‎images/fio/scripts/fio-master‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
worker_hostfile=$1
6+
fio_jobfile=$2
7+
8+
# Don't buffer stdout or stderr to make sure that fio writes its complete log
9+
# before the process exits and the container is killed
10+
11+
# Moving output-format to the end of the command cleans up json output?
12+
# https://github.com/axboe/fio/issues/731#issuecomment-871259173
13+
14+
stdbuf -o0 -e0 fio --client=${worker_hostfile} ${fio_jobfile} --output-format=json+
15+
16+
# Add a sleep to make sure buffers are flushed
17+
sleep 2

‎images/fio/scripts/fio-worker‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/usr/bin/env bash
2+
3+
set -euo pipefail
4+
5+
fio --server=,${FIO_PORT:-8765}

0 commit comments

Comments
 (0)