Merge branch 'main' into upstream_merge_2025_06_02

Author: gshtras

docs/dev-docker/README.md

@@ -21,9 +21,7 @@ Pull the most recent validated docker image with `docker pull rocm/vllm-dev:main
 
 ## What is New
 
-- Out of memory bug fix
-- PyTorch fixes
-- Tunable ops fixes
+- AITER V1 engine performance improvement
 
 ## Known Issues and Workarounds
 - None
@@ -39,14 +37,14 @@ The table below shows performance data where a local inference client is fed req
 
 | Model | Precision | TP Size | Input | Output | Num Prompts | Max Num Seqs | Throughput (tokens/s) |
 |-------|-----------|---------|-------|--------|-------------|--------------|-----------------------|
-| Llama 3.1 70B (amd/Llama-3.1-70B-Instruct-FP8-KV) | FP8 | 8 | 128 | 2048 | 3200 | 3200 | 16892.6  |
-|       |           |         | 128   | 4096   | 1500        | 1500         | 13916.7               |
-|       |           |         | 500   | 2000   | 2000        | 2000         | 13616.1               |
-|       |           |         | 2048  | 2048   | 1500        | 1500         | 8491.8                |
-| Llama 3.1 405B (amd/Llama-3.1-405B-Instruct-FP8-KV) | FP8 | 8 | 128 | 2048 | 1500 | 1500 | 4380.3 |
-|       |           |         | 128   | 4096   | 1500        | 1500         | 3404.2                |
-|       |           |         | 500   | 2000   | 2000        | 2000         | 3251.3                |
-|       |           |         | 2048  | 2048   | 500         | 500          | 2249.3                |
+| Llama 3.1 70B (amd/Llama-3.1-70B-Instruct-FP8-KV) | FP8 | 8 | 128 | 2048 | 3200 | 3200 | 16622.2  |
+|       |           |         | 128   | 4096   | 1500        | 1500         | 13779.8               |
+|       |           |         | 500   | 2000   | 2000        | 2000         | 13424.9               |
+|       |           |         | 2048  | 2048   | 1500        | 1500         | 8356.5                |
+| Llama 3.1 405B (amd/Llama-3.1-405B-Instruct-FP8-KV) | FP8 | 8 | 128 | 2048 | 1500 | 1500 | 4243.9 |
+|       |           |         | 128   | 4096   | 1500        | 1500         | 3394.4                |
+|       |           |         | 500   | 2000   | 2000        | 2000         | 3201.8                |
+|       |           |         | 2048  | 2048   | 500         | 500          | 2208.0                |
 
 *TP stands for Tensor Parallelism.*
 
@@ -56,38 +54,38 @@ The table below shows latency measurement, which typically involves assessing th
 
 | Model | Precision | TP Size | Batch Size | Input | Output | MI300X Latency (sec) |
 |-------|-----------|----------|------------|--------|---------|-------------------|
-| Llama 3.1 70B (amd/Llama-3.1-70B-Instruct-FP8-KV) | FP8 | 8 | 1 | 128 | 2048 | 15.591 |
-| | | | 2 | 128 | 2048 | 16.865 |
-| | | | 4 | 128 | 2048 | 17.295 |
-| | | | 8 | 128 | 2048 | 18.939 |
-| | | | 16 | 128 | 2048 | 20.891 |
-| | | | 32 | 128 | 2048 | 23.402 |
-| | | | 64 | 128 | 2048 | 30.633 |
-| | | | 128 | 128 | 2048 | 43.898 |
-| | | | 1 | 2048 | 2048 | 15.678 |
-| | | | 2 | 2048 | 2048 | 16.892 |
-| | | | 4 | 2048 | 2048 | 17.781 |
-| | | | 8 | 2048 | 2048 | 19.536 |
-| | | | 16 | 2048 | 2048 | 22.521 |
-| | | | 32 | 2048 | 2048 | 26.729 |
-| | | | 64 | 2048 | 2048 | 36.794 |
-| | | | 128 | 2048 | 2048 | 56.371 |
-| Llama 3.1 405B (amd/Llama-3.1-405B-Instruct-FP8-KV) | FP8 | 8 | 1 | 128 | 2048 | 45.446 |
-| | | | 2 | 128 | 2048 | 46.223 |
-| | | | 4 | 128 | 2048 | 47.833 |
-| | | | 8 | 128 | 2048 | 52.085 |
-| | | | 16 | 128 | 2048 | 54.378 |
-| | | | 32 | 128 | 2048 | 63.108 |
-| | | | 64 | 128 | 2048 | 81.764 |
-| | | | 128 | 128 | 2048 | 109.479 |
-| | | | 1 | 2048 | 2048 | 46.001 |
-| | | | 2 | 2048 | 2048 | 46.720 |
-| | | | 4 | 2048 | 2048 | 49.250 |
-| | | | 8 | 2048 | 2048 | 54.495 |
-| | | | 16 | 2048 | 2048 | 59.539 |
-| | | | 32 | 2048 | 2048 | 73.906 |
-| | | | 64 | 2048 | 2048 | 103.847 |
-| | | | 128 | 2048 | 2048 | 151.613 |
+| Llama 3.1 70B (amd/Llama-3.1-70B-Instruct-FP8-KV) | FP8 | 8 | 1 | 128 | 2048 | 15.851 |
+| | | | 2 | 128 | 2048 | 16.995 |
+| | | | 4 | 128 | 2048 | 17.578 |
+| | | | 8 | 128 | 2048 | 19.277 |
+| | | | 16 | 128 | 2048 | 21.111 |
+| | | | 32 | 128 | 2048 | 23.902 |
+| | | | 64 | 128 | 2048 | 30.976 |
+| | | | 128 | 128 | 2048 | 44.107 |
+| | | | 1 | 2048 | 2048 | 15.981 |
+| | | | 2 | 2048 | 2048 | 17.322 |
+| | | | 4 | 2048 | 2048 | 18.025 |
+| | | | 8 | 2048 | 2048 | 20.218 |
+| | | | 16 | 2048 | 2048 | 22.690 |
+| | | | 32 | 2048 | 2048 | 27.407 |
+| | | | 64 | 2048 | 2048 | 37.099 |
+| | | | 128 | 2048 | 2048 | 56.659 |
+| Llama 3.1 405B (amd/Llama-3.1-405B-Instruct-FP8-KV) | FP8 | 8 | 1 | 128 | 2048 | 45.929 |
+| | | | 2 | 128 | 2048 | 46.871 |
+| | | | 4 | 128 | 2048 | 48.763 |
+| | | | 8 | 128 | 2048 | 51.621 |
+| | | | 16 | 128 | 2048 | 54.822 |
+| | | | 32 | 128 | 2048 | 63.642 |
+| | | | 64 | 128 | 2048 | 82.256 |
+| | | | 128 | 128 | 2048 | 110.142 |
+| | | | 1 | 2048 | 2048 | 46.489 |
+| | | | 2 | 2048 | 2048 | 47.465 |
+| | | | 4 | 2048 | 2048 | 49.906 |
+| | | | 8 | 2048 | 2048 | 54.252 |
+| | | | 16 | 2048 | 2048 | 60.275 |
+| | | | 32 | 2048 | 2048 | 74.346 |
+| | | | 64 | 2048 | 2048 | 104.508 |
+| | | | 128 | 2048 | 2048 | 154.134 |
 
 *TP stands for Tensor Parallelism.*
 
@@ -489,7 +487,7 @@ To reproduce the release docker:
 ```bash
     git clone https://github.com/ROCm/vllm.git
     cd vllm
-    git checkout d60b5a337a552b6f74f511462d4ba67ea0ac4402
+    git checkout 91a56009841e11b84a2aeb9cc5aa305ab2808ede
     docker build -f docker/Dockerfile.rocm -t <your_tag> --build-arg USE_CYTHON=1 .
 ```
 
@@ -506,6 +504,9 @@ Use AITER release candidate branch instead:
 
 ## Changelog
 
+20250521_aiter:
+- AITER V1 engine performance improvement
+
 20250513_aiter:
 - Out of memory bug fix
 - PyTorch fixes