|
50 | 50 |
|
51 | 51 | ibv_devices (ibverbs-utils package of Debian/Ubuntu) |
52 | 52 |
|
| 53 | +## Performance tuning |
| 54 | +The RDMA completion queue will use the completion vector to signal completion events |
| 55 | +via hardware interrupts. A large number of hardware interrupts can affect CPU performance. |
| 56 | +It is possible to tune the performance using `rdma-comp-vector`. |
| 57 | + |
| 58 | +See [RDMA CQ completion vector](https://man7.org/linux/man-pages/man3/ibv_create_cq.3.html) |
| 59 | + |
| 60 | +### Example 1 |
| 61 | + |
| 62 | +- Pin hardware interrupt vectors [0, 3] to CPU [0, 3]. |
| 63 | +- Set CPU affinity for valkey to CPU [4, X]. |
| 64 | +- Any valkey server uses a random RDMA completion vector [-1]. |
| 65 | + |
| 66 | +All valkey servers will not affect each other and will be isolated from kernel interrupts. |
| 67 | + |
| 68 | +``` |
| 69 | + SYS SYS SYS SYS VALKEY VALKEY VALKEY |
| 70 | + | | | | | | | |
| 71 | + CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 ... CPUX |
| 72 | + | | | | |
| 73 | + INTR0 INTR1 INTR2 INTR3 |
| 74 | +``` |
| 75 | + |
| 76 | +### Example 2 |
| 77 | + |
| 78 | +- Pin hardware interrupt vectors [0, X] to CPU [0, X]. |
| 79 | +- Set CPU affinity for valkey [M] to CPU [M]. |
| 80 | +- Valkey server [M] uses RDMA completion vector [M]. |
| 81 | + |
| 82 | +A single CPU [M] handles hardware interrupts, the RDMA completion vector [M], |
| 83 | +and the valkey server [M] within its context only. |
| 84 | +This avoids overhead and function calls across multiple CPUs, fully isolating |
| 85 | +each valkey server from one another. |
| 86 | + |
| 87 | +``` |
| 88 | +VALKEY VALKEY VALKEY VALKEY VALKEY VALKEY VALKEY |
| 89 | + | | | | | | | |
| 90 | + CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 ... CPUX |
| 91 | + | | | | | | | |
| 92 | + INTR0 INTR1 INTR2 INTR3 INTR4 INTR5 INTRX |
| 93 | +``` |
| 94 | + |
| 95 | +Use 0 and positive numbers to specify the RDMA completion vector, or specify -1 to allow |
| 96 | +the server to use a random vector for a new connection. The default vector is -1. |
| 97 | + |
53 | 98 |
|
54 | 99 | ## Protocol |
55 | 100 |
|
|
0 commit comments