Skip to content

Commit f1d0877

Browse files
authored
Add performance tuning section for RDMA (#190)
valkey-server is performance sensitive on RDMA, add man page section for deeply performance tuning. [Link](valkey-io/valkey#1209) for valkey changes. --------- Signed-off-by: zhenwei pi <[email protected]>
1 parent 4256863 commit f1d0877

File tree

1 file changed

+45
-0
lines changed

1 file changed

+45
-0
lines changed

topics/RDMA.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,51 @@ Or:
5050

5151
ibv_devices (ibverbs-utils package of Debian/Ubuntu)
5252

53+
## Performance tuning
54+
The RDMA completion queue will use the completion vector to signal completion events
55+
via hardware interrupts. A large number of hardware interrupts can affect CPU performance.
56+
It is possible to tune the performance using `rdma-comp-vector`.
57+
58+
See [RDMA CQ completion vector](https://man7.org/linux/man-pages/man3/ibv_create_cq.3.html)
59+
60+
### Example 1
61+
62+
- Pin hardware interrupt vectors [0, 3] to CPU [0, 3].
63+
- Set CPU affinity for valkey to CPU [4, X].
64+
- Any valkey server uses a random RDMA completion vector [-1].
65+
66+
All valkey servers will not affect each other and will be isolated from kernel interrupts.
67+
68+
```
69+
SYS SYS SYS SYS VALKEY VALKEY VALKEY
70+
| | | | | | |
71+
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 ... CPUX
72+
| | | |
73+
INTR0 INTR1 INTR2 INTR3
74+
```
75+
76+
### Example 2
77+
78+
- Pin hardware interrupt vectors [0, X] to CPU [0, X].
79+
- Set CPU affinity for valkey [M] to CPU [M].
80+
- Valkey server [M] uses RDMA completion vector [M].
81+
82+
A single CPU [M] handles hardware interrupts, the RDMA completion vector [M],
83+
and the valkey server [M] within its context only.
84+
This avoids overhead and function calls across multiple CPUs, fully isolating
85+
each valkey server from one another.
86+
87+
```
88+
VALKEY VALKEY VALKEY VALKEY VALKEY VALKEY VALKEY
89+
| | | | | | |
90+
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 ... CPUX
91+
| | | | | | |
92+
INTR0 INTR1 INTR2 INTR3 INTR4 INTR5 INTRX
93+
```
94+
95+
Use 0 and positive numbers to specify the RDMA completion vector, or specify -1 to allow
96+
the server to use a random vector for a new connection. The default vector is -1.
97+
5398

5499
## Protocol
55100

0 commit comments

Comments
 (0)