Skip to content

Commit 27b804c

Browse files
authored
[Doc] Update ROCm getting started instruction (vllm-project#678)
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
1 parent 0cb7ec6 commit 27b804c

File tree

2 files changed

+46
-5
lines changed

2 files changed

+46
-5
lines changed

docs/getting_started/installation/gpu.md

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -49,15 +49,18 @@ vLLM-Omni is a Python library that supports the following GPU variants. The libr
4949

5050
## Set up using Docker
5151

52+
### Pre-built images
5253

53-
### Build your own docker image
54+
=== "NVIDIA CUDA"
55+
56+
--8<-- "docs/getting_started/installation/gpu/cuda.inc.md:pre-built-images"
5457

5558
=== "AMD ROCm"
5659

57-
--8<-- "docs/getting_started/installation/gpu/rocm.inc.md:build-docker"
60+
--8<-- "docs/getting_started/installation/gpu/rocm.inc.md:pre-built-images"
5861

59-
### Pre-built images
62+
### Build your own docker image
6063

61-
=== "NVIDIA CUDA"
64+
=== "AMD ROCm"
6265

63-
--8<-- "docs/getting_started/installation/gpu/cuda.inc.md:pre-built-images"
66+
--8<-- "docs/getting_started/installation/gpu/rocm.inc.md:build-docker"

docs/getting_started/installation/gpu/rocm.inc.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,4 +52,42 @@ bash
5252

5353
# --8<-- [start:pre-built-images]
5454

55+
vLLM-Omni offers an official docker image for deployment. These images are built on top of vLLM docker images and available on Docker Hub as [vllm/vllm-omni-rocm](https://hub.docker.com/r/vllm/vllm-omni-rocm/tags). The version of vLLM-Omni indicates which release of vLLM it is based on.
56+
57+
#### Launch vLLM-Omni Server
58+
Here's an example deployment command that has been verified on 2 x MI300's:
59+
```bash
60+
docker run -it \
61+
--network=host \
62+
--group-add=video \
63+
--ipc=host \
64+
--cap-add=SYS_PTRACE \
65+
--security-opt seccomp=unconfined \
66+
--device /dev/kfd \
67+
--device /dev/dri \
68+
-v <path/to/model>:/app/model \
69+
-v ~/.cache/huggingface:/root/.cache/huggingface \
70+
--env "HF_TOKEN=$HF_TOKEN" \
71+
vllm/vllm-omni-rocm:v0.12.0rc1 \
72+
vllm serve --model Qwen/Qwen3-Omni-30B-A3B-Instruct --omni --port 8091
73+
```
74+
75+
#### Launch an interactive terminal with prebuilt docker image.
76+
If you want to run in dev environment you can launch the docker image as follows:
77+
```bash
78+
docker run -it \
79+
--network=host \
80+
--group-add=video \
81+
--ipc=host \
82+
--cap-add=SYS_PTRACE \
83+
--security-opt seccomp=unconfined \
84+
--device /dev/kfd \
85+
--device /dev/dri \
86+
-v <path/to/model>:/app/model \
87+
-v ~/.cache/huggingface:/root/.cache/huggingface \
88+
--env "HF_TOKEN=$HF_TOKEN" \
89+
vllm/vllm-omni-rocm:v0.12.0rc1 \
90+
bash
91+
```
92+
5593
# --8<-- [end:pre-built-images]

0 commit comments

Comments
 (0)