Skip to content

Commit

Permalink
Add performance tuning section for RDMA
Browse files Browse the repository at this point in the history
valkey-server is performance sensitive on RDMA, add man page for
deeply performance tuning.

Signed-off-by: zhenwei pi <[email protected]>
  • Loading branch information
pizhenwei committed Nov 21, 2024
1 parent 4256863 commit c0e3c65
Showing 1 changed file with 41 additions and 0 deletions.
41 changes: 41 additions & 0 deletions topics/RDMA.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,47 @@ Or:

ibv_devices (ibverbs-utils package of Debian/Ubuntu)

### Performance tuning
The RDMA completion queue will use the completion vector to signal completion events
via hardware interrupts. A large number of hardware interrupts can affect CPU performance.
It is possible to tune the performance using `rdma-comp-vector`.

#### Example 1

- Pin hardware interrupt vectors [0, 3] to CPU [0, 3].
- Set CPU affinity for valkey to CPU [4, X].
- Any valkey server uses a random RDMA completion vector.

All valkey servers will not affect each other and will be isolated from kernel interrupts.

```
SYS SYS SYS SYS VALKEY VALKEY VALKEY
| | | | | | |
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 ... CPUX
| | | |
INTR0 INTR1 INTR2 INTR3
```

#### Example 2

- Pin hardware interrupt vectors [0, X] to CPU [0, X].
- Set CPU affinity for valkey to CPU [0, X].
- Valkey server [M] uses RDMA completion vector [M].

A single CPU handles hardware interrupts, the RDMA completion queue, and the valkey server.
This avoids overhead and function calls across multiple CPUs.

```
VALKEY VALKEY VALKEY VALKEY VALKEY VALKEY VALKEY
| | | | | | |
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 ... CPUX
| | | | | | |
INTR0 INTR1 INTR2 INTR3 INTR4 INTR5 INTRX
```

Use 0 and positive numbers to specify the RDMA completion vector, or specify -1 to allow
the server to use a random vector for a new connection. The default vector is -1.


## Protocol

Expand Down

0 comments on commit c0e3c65

Please sign in to comment.