Skip to content

Commit 4f37432

Browse files
fcostaoliveirafcostaoliveira
authored
Added 142 new benchmarks focused on latency tracking (rate-limited) (#273)
* Added 142 new benchmarks focused on latency tracking (rate-limited) * Bumping version from 0.1.240 to 0.1.241 * Added 142 new benchmarks focused on latency tracking (rate-limited) * Using docker image redislabs/memtier_benchmark:2.1.0 on latency related benchmarks * Removed benchmarks above 25gb * Removed 2TB test * Deleted wrongly designed test * Added 3M keys 512 and 1000B GET/SET use-cases
1 parent ad690c3 commit 4f37432

File tree

149 files changed

+7894
-143
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

149 files changed

+7894
-143
lines changed

‎pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[tool.poetry]
22
name = "redis-benchmarks-specification"
3-
version = "0.1.240"
3+
version = "0.1.245"
44
description = "The Redis benchmarks specification describes the cross-language/tools requirements and expectations to foster performance and observability standards around redis related technologies. Members from both industry and academia, including organizations and individuals are encouraged to contribute."
55
authors = ["filipecosta90 <[email protected]>","Redis Performance Group <[email protected]>"]
66
readme = "Readme.md"

‎redis_benchmarks_specification/test-suites/latency-rate-limited-10000_qps-memtier_benchmark-100Kkeys-hash-hgetall-50-fields-100B-values.yml

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
version: 0.4
2+
name:
3+
latency-rate-limited-10000_qps-memtier_benchmark-100Kkeys-hash-hgetall-50-fields-100B-values
4+
description: Runs memtier_benchmark, for a keyspace length of 1M keys pre-loading
5+
HASHes in which the value has a data size of 10 Bytes. After pre-loading the data
6+
it issues HGETALL command. Rate limited to 10000 ops/sec.
7+
dbconfig:
8+
configuration-parameters:
9+
save: '""'
10+
check:
11+
keyspacelen: 100000
12+
preload_tool:
13+
run_image: redislabs/memtier_benchmark:2.1.0
14+
tool: memtier_benchmark
15+
arguments: '"--data-size" "100" --command "HSET __key__ field:1 __data__ field:2
16+
__data__ field:3 __data__ field:4 __data__ field:5 __data__ field:6 __data__
17+
field:7 __data__ field:8 __data__ field:9 __data__ field:10 __data__ field:11
18+
__data__ field:12 __data__ field:13 __data__ field:14 __data__ field:15 __data__
19+
field:16 __data__ field:17 __data__ field:18 __data__ field:19 __data__ field:20
20+
__data__ field:21 __data__ field:22 __data__ field:23 __data__ field:24 __data__
21+
field:25 __data__ field:26 __data__ field:27 __data__ field:28 __data__ field:29
22+
__data__ field:30 __data__ field:31 __data__ field:32 __data__ field:33 __data__
23+
field:34 __data__ field:35 __data__ field:36 __data__ field:37 __data__ field:38
24+
__data__ field:39 __data__ field:40 __data__ field:41 __data__ field:42 __data__
25+
field:43 __data__ field:44 __data__ field:45 __data__ field:46 __data__ field:47
26+
__data__ field:48 __data__ field:49 __data__ field:50 __data__" --command-key-pattern="P"
27+
--key-minimum=1 --key-maximum 1000000 -n 500 -c 50 -t 4 --hide-histogram'
28+
resources:
29+
requests:
30+
memory: 2g
31+
tested-groups:
32+
- hash
33+
tested-commands:
34+
- hgetall
35+
redis-topologies:
36+
- oss-standalone
37+
build-variants:
38+
- gcc:8.5.0-amd64-debian-buster-default
39+
- dockerhub
40+
clientconfig:
41+
run_image: redislabs/memtier_benchmark:2.1.0
42+
tool: memtier_benchmark
43+
arguments: "--command \"HGETALL __key__\" --command-key-pattern=\"R\" --key-minimum=1
44+
--key-maximum 100000 --test-time 120 -c 50 -t 4 --hide-histogram --rate-limiting
45+
50"
46+
resources:
47+
requests:
48+
cpus: '4'
49+
memory: 2g
50+
51+
priority: 96

‎redis_benchmarks_specification/test-suites/latency-rate-limited-10000_qps-memtier_benchmark-100Kkeys-load-hash-50-fields-with-1000B-values.yml

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
version: 0.4
2+
name:
3+
latency-rate-limited-10000_qps-memtier_benchmark-100Kkeys-load-hash-50-fields-with-1000B-values
4+
description: Runs memtier_benchmark, for a keyspace length of 100K keys loading HASHES
5+
with 50 fields each. Each field value has a data size of 1000 Bytes. Rate limited
6+
to 10000 ops/sec.
7+
dbconfig:
8+
configuration-parameters:
9+
save: '""'
10+
check:
11+
keyspacelen: 0
12+
resources:
13+
requests:
14+
memory: 6g
15+
tested-groups:
16+
- hash
17+
tested-commands:
18+
- hset
19+
redis-topologies:
20+
- oss-standalone
21+
build-variants:
22+
- gcc:8.5.0-amd64-debian-buster-default
23+
- dockerhub
24+
clientconfig:
25+
run_image: redislabs/memtier_benchmark:2.1.0
26+
tool: memtier_benchmark
27+
arguments: "--test-time 120 --distinct-client-seed \"--data-size\" \"1000\" --command
28+
\"HSET __key__ field:1 __data__ field:2 __data__ field:3 __data__ field:4 __data__
29+
field:5 __data__ field:6 __data__ field:7 __data__ field:8 __data__ field:9 __data__
30+
field:10 __data__ field:11 __data__ field:12 __data__ field:13 __data__ field:14
31+
__data__ field:15 __data__ field:16 __data__ field:17 __data__ field:18 __data__
32+
field:19 __data__ field:20 __data__ field:21 __data__ field:22 __data__ field:23
33+
__data__ field:24 __data__ field:25 __data__ field:26 __data__ field:27 __data__
34+
field:28 __data__ field:29 __data__ field:30 __data__ field:31 __data__ field:32
35+
__data__ field:33 __data__ field:34 __data__ field:35 __data__ field:36 __data__
36+
field:37 __data__ field:38 __data__ field:39 __data__ field:40 __data__ field:41
37+
__data__ field:42 __data__ field:43 __data__ field:44 __data__ field:45 __data__
38+
field:46 __data__ field:47 __data__ field:48 __data__ field:49 __data__ field:50
39+
__data__\" --command-key-pattern=\"R\" --key-minimum=1 --key-maximum 100000 -c
40+
50 -t 4 --hide-histogram --rate-limiting 50"
41+
resources:
42+
requests:
43+
cpus: '4'
44+
memory: 2g
45+
46+
priority: 5

‎redis_benchmarks_specification/test-suites/latency-rate-limited-10000_qps-memtier_benchmark-100Kkeys-load-hash-50-fields-with-100B-values.yml

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
version: 0.4
2+
name:
3+
latency-rate-limited-10000_qps-memtier_benchmark-100Kkeys-load-hash-50-fields-with-100B-values
4+
description: Runs memtier_benchmark, for a keyspace length of 100K keys loading HASHES
5+
with 50 fields each. Each field value has a data size of 100 Bytes. Rate limited
6+
to 10000 ops/sec.
7+
dbconfig:
8+
configuration-parameters:
9+
save: '""'
10+
check:
11+
keyspacelen: 0
12+
resources:
13+
requests:
14+
memory: 6g
15+
tested-groups:
16+
- hash
17+
tested-commands:
18+
- hset
19+
redis-topologies:
20+
- oss-standalone
21+
build-variants:
22+
- gcc:8.5.0-amd64-debian-buster-default
23+
- dockerhub
24+
clientconfig:
25+
run_image: redislabs/memtier_benchmark:2.1.0
26+
tool: memtier_benchmark
27+
arguments: "--test-time 120 --distinct-client-seed \"--data-size\" \"100\" --command
28+
\"HSET __key__ field:1 __data__ field:2 __data__ field:3 __data__ field:4 __data__
29+
field:5 __data__ field:6 __data__ field:7 __data__ field:8 __data__ field:9 __data__
30+
field:10 __data__ field:11 __data__ field:12 __data__ field:13 __data__ field:14
31+
__data__ field:15 __data__ field:16 __data__ field:17 __data__ field:18 __data__
32+
field:19 __data__ field:20 __data__ field:21 __data__ field:22 __data__ field:23
33+
__data__ field:24 __data__ field:25 __data__ field:26 __data__ field:27 __data__
34+
field:28 __data__ field:29 __data__ field:30 __data__ field:31 __data__ field:32
35+
__data__ field:33 __data__ field:34 __data__ field:35 __data__ field:36 __data__
36+
field:37 __data__ field:38 __data__ field:39 __data__ field:40 __data__ field:41
37+
__data__ field:42 __data__ field:43 __data__ field:44 __data__ field:45 __data__
38+
field:46 __data__ field:47 __data__ field:48 __data__ field:49 __data__ field:50
39+
__data__\" --command-key-pattern=\"R\" --key-minimum=1 --key-maximum 100000 -c
40+
50 -t 4 --hide-histogram --rate-limiting 50"
41+
resources:
42+
requests:
43+
cpus: '4'
44+
memory: 2g
45+
46+
priority: 5

‎redis_benchmarks_specification/test-suites/latency-rate-limited-10000_qps-memtier_benchmark-100Kkeys-load-hash-50-fields-with-10B-values.yml

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
version: 0.4
2+
name:
3+
latency-rate-limited-10000_qps-memtier_benchmark-100Kkeys-load-hash-50-fields-with-10B-values
4+
description: Runs memtier_benchmark, for a keyspace length of 100K keys loading HASHES
5+
with 50 fields each. Each field value has a data size of 10 Bytes. Rate limited
6+
to 10000 ops/sec.
7+
dbconfig:
8+
configuration-parameters:
9+
save: '""'
10+
check:
11+
keyspacelen: 0
12+
resources:
13+
requests:
14+
memory: 6g
15+
tested-groups:
16+
- hash
17+
tested-commands:
18+
- hset
19+
redis-topologies:
20+
- oss-standalone
21+
build-variants:
22+
- gcc:8.5.0-amd64-debian-buster-default
23+
- dockerhub
24+
clientconfig:
25+
run_image: redislabs/memtier_benchmark:2.1.0
26+
tool: memtier_benchmark
27+
arguments: "--test-time 120 --distinct-client-seed \"--data-size\" \"10\" --command
28+
\"HSET __key__ field:1 __data__ field:2 __data__ field:3 __data__ field:4 __data__
29+
field:5 __data__ field:6 __data__ field:7 __data__ field:8 __data__ field:9 __data__
30+
field:10 __data__ field:11 __data__ field:12 __data__ field:13 __data__ field:14
31+
__data__ field:15 __data__ field:16 __data__ field:17 __data__ field:18 __data__
32+
field:19 __data__ field:20 __data__ field:21 __data__ field:22 __data__ field:23
33+
__data__ field:24 __data__ field:25 __data__ field:26 __data__ field:27 __data__
34+
field:28 __data__ field:29 __data__ field:30 __data__ field:31 __data__ field:32
35+
__data__ field:33 __data__ field:34 __data__ field:35 __data__ field:36 __data__
36+
field:37 __data__ field:38 __data__ field:39 __data__ field:40 __data__ field:41
37+
__data__ field:42 __data__ field:43 __data__ field:44 __data__ field:45 __data__
38+
field:46 __data__ field:47 __data__ field:48 __data__ field:49 __data__ field:50
39+
__data__\" --command-key-pattern=\"R\" --key-minimum=1 --key-maximum 100000 -c
40+
50 -t 4 --hide-histogram --rate-limiting 50"
41+
resources:
42+
requests:
43+
cpus: '4'
44+
memory: 2g
45+
46+
priority: 5

‎redis_benchmarks_specification/test-suites/latency-rate-limited-10000_qps-memtier_benchmark-10Mkeys-load-hash-5-fields-with-100B-values-pipeline-10.yml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
version: 0.4
2+
name:
3+
latency-rate-limited-10000_qps-memtier_benchmark-10Mkeys-load-hash-5-fields-with-100B-values-pipeline-10
4+
description: Runs memtier_benchmark, for a keyspace length of 10M keys loading HASHES
5+
with 5 fields each. Each field value has a data size of 100 Bytes. Rate limited
6+
to 10000 ops/sec.
7+
dbconfig:
8+
configuration-parameters:
9+
save: '""'
10+
check:
11+
keyspacelen: 0
12+
resources:
13+
requests:
14+
memory: 9g
15+
tested-commands:
16+
- hset
17+
redis-topologies:
18+
- oss-standalone
19+
build-variants:
20+
- gcc:8.5.0-amd64-debian-buster-default
21+
- dockerhub
22+
clientconfig:
23+
run_image: redislabs/memtier_benchmark:2.1.0
24+
tool: memtier_benchmark
25+
arguments: "\"--pipeline\" \"10\" \"--data-size\" \"100\" --command \"HSET __key__
26+
field1 __data__ field2 __data__ field3 __data__ field4 __data__ field5 __data__\"\
27+
\ \"--command-key-pattern\" \"P\" --key-minimum=1 --key-maximum 10000000 -c 50
28+
-t 4 --hide-histogram --test-time=180 --rate-limiting 50"
29+
resources:
30+
requests:
31+
cpus: '4'
32+
memory: 2g
33+
34+
tested-groups:
35+
- hash
36+
priority: 5

‎redis_benchmarks_specification/test-suites/latency-rate-limited-10000_qps-memtier_benchmark-10Mkeys-load-hash-5-fields-with-100B-values.yml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
version: 0.4
2+
name:
3+
latency-rate-limited-10000_qps-memtier_benchmark-10Mkeys-load-hash-5-fields-with-100B-values
4+
description: Runs memtier_benchmark, for a keyspace length of 10M keys loading HASHES
5+
with 5 fields each. Each field value has a data size of 100 Bytes. Rate limited
6+
to 10000 ops/sec.
7+
dbconfig:
8+
configuration-parameters:
9+
save: '""'
10+
check:
11+
keyspacelen: 0
12+
resources:
13+
requests:
14+
memory: 9g
15+
tested-commands:
16+
- hset
17+
redis-topologies:
18+
- oss-standalone
19+
build-variants:
20+
- gcc:8.5.0-amd64-debian-buster-default
21+
- dockerhub
22+
clientconfig:
23+
run_image: redislabs/memtier_benchmark:2.1.0
24+
tool: memtier_benchmark
25+
arguments: "--test-time 180 \"--data-size\" \"100\" --command \"HSET __key__ field1
26+
__data__ field2 __data__ field3 __data__ field4 __data__ field5 __data__\" --command-key-pattern=\"\
27+
P\" --key-minimum=1 --key-maximum 10000000 -c 50 -t 4 --hide-histogram --rate-limiting
28+
50"
29+
resources:
30+
requests:
31+
cpus: '4'
32+
memory: 2g
33+
34+
tested-groups:
35+
- hash
36+
priority: 5

‎redis_benchmarks_specification/test-suites/latency-rate-limited-10000_qps-memtier_benchmark-10Mkeys-load-hash-5-fields-with-10B-values-pipeline-10.yml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
version: 0.4
2+
name:
3+
latency-rate-limited-10000_qps-memtier_benchmark-10Mkeys-load-hash-5-fields-with-10B-values-pipeline-10
4+
description: Runs memtier_benchmark, for a keyspace length of 10M keys loading HASHES
5+
with 5 fields each. Each field value has a data size of 10 Bytes. Rate limited to
6+
10000 ops/sec.
7+
dbconfig:
8+
configuration-parameters:
9+
save: '""'
10+
check:
11+
keyspacelen: 0
12+
resources:
13+
requests:
14+
memory: 2g
15+
tested-commands:
16+
- hset
17+
redis-topologies:
18+
- oss-standalone
19+
build-variants:
20+
- gcc:8.5.0-amd64-debian-buster-default
21+
- dockerhub
22+
clientconfig:
23+
run_image: redislabs/memtier_benchmark:2.1.0
24+
tool: memtier_benchmark
25+
arguments: "--test-time 180 \"--pipeline\" \"10\" \"--data-size\" \"10\" --command
26+
\"HSET __key__ field1 __data__ field2 __data__ field3 __data__ field4 __data__
27+
field5 __data__\" --command-key-pattern=\"P\" --key-minimum=1 --key-maximum 10000000
28+
-c 50 -t 4 --hide-histogram --rate-limiting 50"
29+
resources:
30+
requests:
31+
cpus: '4'
32+
memory: 2g
33+
34+
tested-groups:
35+
- hash
36+
priority: 5

‎redis_benchmarks_specification/test-suites/latency-rate-limited-10000_qps-memtier_benchmark-10Mkeys-load-hash-5-fields-with-10B-values.yml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
version: 0.4
2+
name:
3+
latency-rate-limited-10000_qps-memtier_benchmark-10Mkeys-load-hash-5-fields-with-10B-values
4+
description: Runs memtier_benchmark, for a keyspace length of 10M keys loading HASHES
5+
with 5 fields each. Each field value has a data size of 10 Bytes. Rate limited to
6+
10000 ops/sec.
7+
dbconfig:
8+
configuration-parameters:
9+
save: '""'
10+
check:
11+
keyspacelen: 0
12+
resources:
13+
requests:
14+
memory: 2g
15+
tested-commands:
16+
- hset
17+
redis-topologies:
18+
- oss-standalone
19+
build-variants:
20+
- gcc:8.5.0-amd64-debian-buster-default
21+
- dockerhub
22+
clientconfig:
23+
run_image: redislabs/memtier_benchmark:2.1.0
24+
tool: memtier_benchmark
25+
arguments: "--test-time 180 \"--data-size\" \"10\" --command \"HSET __key__ field1
26+
__data__ field2 __data__ field3 __data__ field4 __data__ field5 __data__\" --command-key-pattern=\"\
27+
P\" --key-minimum=1 --key-maximum 10000000 -c 50 -t 4 --hide-histogram --rate-limiting
28+
50"
29+
resources:
30+
requests:
31+
cpus: '4'
32+
memory: 2g
33+
34+
tested-groups:
35+
- hash
36+
priority: 5

‎redis_benchmarks_specification/test-suites/latency-rate-limited-10000_qps-memtier_benchmark-1Mkeys-100B-expire-use-case.yml

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
version: 0.4
2+
name: latency-rate-limited-10000_qps-memtier_benchmark-1Mkeys-100B-expire-use-case
3+
description: Runs memtier_benchmark, for a keyspace length of 1M keys with a data
4+
size of 100 Bytes for each key, which 50% of the keys have expiration set during
5+
the benchmark. Rate limited to 10000 ops/sec.
6+
dbconfig:
7+
configuration-parameters:
8+
save: '""'
9+
check:
10+
keyspacelen: 1000000
11+
preload_tool:
12+
run_image: redislabs/memtier_benchmark:2.1.0
13+
tool: memtier_benchmark
14+
arguments: '"--data-size" "100" "--command" "SET __key__ __data__" "--command-key-pattern"
15+
"P" "-c" "50" "-t" "2" "--hide-histogram" "--key-minimum" "1"'
16+
resources:
17+
requests:
18+
memory: 1g
19+
tested-groups:
20+
- string
21+
- generic
22+
tested-commands:
23+
- set
24+
- setx
25+
- get
26+
- del
27+
- setex
28+
redis-topologies:
29+
- oss-standalone
30+
build-variants:
31+
- gcc:8.5.0-amd64-debian-buster-default
32+
- dockerhub
33+
clientconfig:
34+
run_image: redislabs/memtier_benchmark:2.1.0
35+
tool: memtier_benchmark
36+
arguments: "\"--data-size\" \"100\" --command \"SETEX __key__ 10 __data__\" --command-key-pattern=\"\
37+
R\" --command \"SET __key__ __data__\" --command-key-pattern=\"R\" --command \"\
38+
GET __key__\" --command-key-pattern=\"R\" --command \"DEL __key__\" --command-key-pattern=\"\
39+
R\" -c 50 -t 2 --hide-histogram --test-time 300 --rate-limiting 100"
40+
resources:
41+
requests:
42+
cpus: '3'
43+
memory: 2g
44+
45+
priority: 33

0 commit comments

Comments
 (0)