Skip to content

Commit 9a44436

Browse files
Add better readme
Signed-off-by: Diego-Castan <[email protected]>
1 parent 012d956 commit 9a44436

File tree

1 file changed

+9
-0
lines changed

1 file changed

+9
-0
lines changed

install_pod_files/v3/README.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,18 @@ VLLM_USE_PRECOMPILED=1 pip install -e vllm
1919
1- Run in terminal 1:
2020
>> VLLM_LOGGING_LEVEL=DEBUG vllm serve meta-llama/Llama-3.2-1B
2121
22+
or a 8B parameters using:
23+
>> VLLM_LOGGING_LEVEL=DEBUG vllm serve meta-llama/Llama-3.1-8B-Instruct
24+
2225
2- Run in terminal 2:
26+
>> python3 vllm/benchmarks/benchmark_serving.py --backend vllm --model meta-llama/Llama-3.1-8B-Instruct --endpoint /v1/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 10
27+
28+
or a 8B parameters using:
2329
>> python3 vllm/benchmarks/benchmark_serving.py --backend vllm --model meta-llama/Llama-3.2-1B --endpoint /v1/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 10
2430
31+
Also, there is a new version of using
32+
>> vllm bench serve --backend vllm --model meta-llama/Llama-3.2-1B --endpoint /v1/completions --dataset-name sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 10
33+
2534
If you need to download the dataset (shareGPT) do:
2635
wget <https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json>
2736

0 commit comments

Comments
 (0)