Skip to content

Commit 61fc351

Browse files
shen-shanshanshen-shanshan
authored
[Doc] Add performance tuning doc to main (vllm-project#1392)
### What this PR does / why we need it? Add performance tuning doc to main. Closes: vllm-project#1387 - vLLM version: v0.9.1 - vLLM main: vllm-project/vllm@923147b --------- Signed-off-by: shen-shanshan <[email protected]> Signed-off-by: Shanshan Shen <[email protected]>
1 parent 540336e commit 61fc351

File tree

2 files changed

+184
-0
lines changed

2 files changed

+184
-0
lines changed

‎docs/source/developer_guide/performance/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,5 @@
55
:maxdepth: 1
66
performance_benchmark
77
profile_execute_duration
8+
optimization_and_tuning
89
:::

‎docs/source/developer_guide/performance/optimization_and_tuning.md‎

Lines changed: 183 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,183 @@
1+
# Optimization and Tuning
2+
3+
This guide aims to help users to improve vllm-ascend performance on system level. It includes OS configuration, library optimization, deploy guide and so on. Any feedback is welcome.
4+
5+
## Preparation
6+
7+
Run the container:
8+
9+
```{code-block} bash
10+
:substitutions:
11+
# Update DEVICE according to your device (/dev/davinci[0-7])
12+
export DEVICE=/dev/davinci0
13+
# Update the cann base image
14+
export IMAGE=m.daocloud.io/quay.io/ascend/cann:|cann_image_tag|
15+
docker run --rm \
16+
--name performance-test \
17+
--device $DEVICE \
18+
--device /dev/davinci_manager \
19+
--device /dev/devmm_svm \
20+
--device /dev/hisi_hdc \
21+
-v /usr/local/dcmi:/usr/local/dcmi \
22+
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
23+
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
24+
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
25+
-v /etc/ascend_install.info:/etc/ascend_install.info \
26+
-v /root/.cache:/root/.cache \
27+
-it $IMAGE bash
28+
```
29+
30+
Configure your environment:
31+
32+
```{code-block} bash
33+
:substitutions:
34+
# Configure the mirror
35+
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy main restricted universe multiverse" > /etc/apt/sources.list && \
36+
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy main restricted universe multiverse" >> /etc/apt/sources.list && \
37+
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
38+
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
39+
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
40+
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
41+
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list && \
42+
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list
43+
44+
# Install os packages
45+
apt update && apt install wget gcc g++ libnuma-dev git vim -y
46+
```
47+
48+
Install vllm and vllm-ascend:
49+
50+
```{code-block} bash
51+
:substitutions:
52+
# Install necessary dependencies
53+
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
54+
pip install modelscope pandas datasets gevent sacrebleu rouge_score pybind11 pytest
55+
56+
# Configure this var to speed up model download
57+
VLLM_USE_MODELSCOPE=true
58+
```
59+
60+
Please follow the [Installation Guide](https://vllm-ascend.readthedocs.io/en/latest/installation.html) to make sure vllm, vllm-ascend and mindie-turbo is installed correctly.
61+
62+
:::{note}
63+
Make sure your vllm and vllm-ascend are installed after your python configuration completed, because these packages will build binary files using the python in current environment. If you install vllm, vllm-ascend and mindie-turbo before chapter 1.1, the binary files will not use the optimized python.
64+
:::
65+
66+
## Optimizations
67+
68+
### 1. Compilation Optimization
69+
70+
#### 1.1. Install optimized `python`
71+
72+
Python supports **LTO** and **PGO** optimization starting from version `3.6` and above, which can be enabled at compile time. And we have offered compilation optimized `python` packages directly to users for the sake of convenience. You can also reproduce the `python` build follow this [tutorial](https://www.hiascend.com/document/detail/zh/Pytorch/600/ptmoddevg/trainingmigrguide/performance_tuning_0063.html) according to your specific scenarios.
73+
74+
```{code-block} bash
75+
:substitutions:
76+
mkdir -p /workspace/tmp
77+
cd /workspace/tmp
78+
79+
# Download prebuilt lib and packages
80+
wget https://repo.oepkgs.net/ascend/pytorch/vllm/lib/libcrypto.so.1.1
81+
wget https://repo.oepkgs.net/ascend/pytorch/vllm/lib/libomp.so
82+
wget https://repo.oepkgs.net/ascend/pytorch/vllm/lib/libssl.so.1.1
83+
wget https://repo.oepkgs.net/ascend/pytorch/vllm/python/py311_bisheng.tar.gz
84+
85+
# Configure python and pip
86+
cp ./*.so* /usr/local/lib
87+
tar -zxvf ./py311_bisheng.* -C /usr/local/
88+
mv /usr/local/py311_bisheng/ /usr/local/python
89+
sed -i "1c#\!/usr/local/python/bin/python3.11" /usr/local/python/bin/pip3
90+
sed -i "1c#\!/usr/local/python/bin/python3.11" /usr/local/python/bin/pip3.11
91+
ln -sf /usr/local/python/bin/python3 /usr/bin/python
92+
ln -sf /usr/local/python/bin/python3 /usr/bin/python3
93+
ln -sf /usr/local/python/bin/python3.11 /usr/bin/python3.11
94+
ln -sf /usr/local/python/bin/pip3 /usr/bin/pip3
95+
ln -sf /usr/local/python/bin/pip3 /usr/bin/pip
96+
97+
export PATH=/usr/bin:/usr/local/python/bin:$PATH
98+
```
99+
100+
### 2. OS Optimization
101+
102+
#### 2.1. jemalloc
103+
104+
**jemalloc** is a memory allocator that improves performance for multi-threads scenario and can reduce memory fragment. jemalloc use thread local memory manager to allocate variables, which can avoid lock competition between multi-threads and can hugely optimize performance.
105+
106+
```{code-block} bash
107+
:substitutions:
108+
# Install jemalloc
109+
sudo apt update
110+
sudo apt install libjemalloc2
111+
112+
# Configure jemalloc
113+
export LD_PRELOAD=/usr/lib/"$(uname -i)"-linux-gnu/libjemalloc.so.2 $LD_PRELOAD
114+
```
115+
116+
#### 2.2. Tcmalloc
117+
118+
**Tcmalloc (Thread Counting Malloc)** is a universal memory allocator that improves overall performance while ensuring low latency by introducing a multi-level cache structure, reducing mutex competition and optimizing large object processing flow. Find more details [here](https://www.hiascend.com/document/detail/zh/Pytorch/700/ptmoddevg/trainingmigrguide/performance_tuning_0068.html).
119+
120+
```{code-block} bash
121+
:substitutions:
122+
# Install tcmalloc
123+
sudo apt update
124+
sudo apt install libgoogle-perftools4 libgoogle-perftools-dev
125+
126+
# Get the location of libtcmalloc.so*
127+
find /usr -name libtcmalloc.so*
128+
129+
# Make the priority of tcmalloc higher
130+
# The <path> is the location of libtcmalloc.so we get from the upper command
131+
# Example: "$LD_PRELOAD:/usr/lib/aarch64-linux-gnu/libtcmalloc.so"
132+
export LD_PRELOAD="$LD_PRELOAD:<path>"
133+
134+
# Verify your configuration
135+
# The path of libtcmalloc.so will be contained in the result if your configuration is valid
136+
ldd `which python`
137+
```
138+
139+
### 3. `torch_npu` Optimization
140+
141+
Some performance tuning features in `torch_npu` are controlled by environment variables. Some features and their related environment variables are shown below.
142+
143+
Memory optimization:
144+
145+
```{code-block} bash
146+
:substitutions:
147+
# Upper limit of memory block splitting allowed (MB), Setting this parameter can prevent large memory blocks from being split.
148+
export PYTORCH_NPU_ALLOC_CONF="max_split_size_mb:250"
149+
150+
# When operators on the communication stream have dependencies, they all need to be ended before being released for reuse. The logic of multi-stream reuse is to release the memory on the communication stream in advance so that the computing stream can be reused.
151+
export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"
152+
```
153+
154+
Schedule optimization:
155+
156+
```{code-block} bash
157+
:substitutions:
158+
# Optimize operator delivery queue, this will affect the memory peak value, and may degrade if the memory is tight.
159+
export TASK_QUEUE_ENABLE=2
160+
161+
# This will greatly improve the CPU bottleneck model and ensure the same performance for the NPU bottleneck model.
162+
export CPU_AFFINITY_CONF=1
163+
```
164+
165+
### 4. CANN Optimization
166+
167+
#### 4.1. HCCL Optimization
168+
169+
There are some performance tuning features in HCCL, which are controlled by environment variables.
170+
171+
You can configure HCCL to use "AIV" mode to optimize performance by setting the environment variable shown below. In "AIV" mode, the communication is scheduled by AI vector core directly with ROCE, instead of being scheduled by AI cpu.
172+
173+
```{code-block} bash
174+
:substitutions:
175+
export HCCL_OP_EXPANSION_MODE="AIV"
176+
```
177+
178+
Plus, there are more features for performance optimization in specific scenarios, which are shown below.
179+
180+
- `HCCL_INTRA_ROCE_ENABLE`: Use RDMA link instead of SDMA link between two 8Ps as the mesh interconnect link, find more details [here](https://www.hiascend.com/document/detail/zh/Pytorch/600/ptmoddevg/trainingmigrguide/performance_tuning_0044.html).
181+
- `HCCL_RDMA_TC`: Use this var to configure traffic class of RDMA network card, find more details [here](https://www.hiascend.com/document/detail/zh/Pytorch/600/ptmoddevg/trainingmigrguide/performance_tuning_0045.html).
182+
- `HCCL_RDMA_SL`: Use this var to configure service level of RDMA network card, find more details [here](https://www.hiascend.com/document/detail/zh/Pytorch/600/ptmoddevg/trainingmigrguide/performance_tuning_0046.html).
183+
- `HCCL_BUFFSIZE`: Use this var to control the cache size for sharing data between two NPUs, find more details [here](https://www.hiascend.com/document/detail/zh/Pytorch/600/ptmoddevg/trainingmigrguide/performance_tuning_0047.html).

0 commit comments

Comments
 (0)