Skip to content

Conversation

@kliuae-amd
Copy link

@kliuae-amd kliuae-amd commented Jan 6, 2026

Purpose

This PR adds fusion of silu-and-mul and dynamic per-token fp8 quant to speed up ptpc-fp8 model inferencing.
To enable this fusion, add to the compilation config --compilation-config '{"pass_config": {"fuse_act_quant": "true"}}'.
This fusion pass requires new aiter features not yet added to upstream aiter, so the fusion is disabled by default. It is added here for model validation and faster upstream cherry picking once the features are added to aiter.

Test Plan

Unit test for the new fusion pass.
Model end-to-end test on RedhatAI/Qwen2.5-VL-72B-Instruct-FP8-dynamic with silu-mul-ptpcfp8 fusion.

Test Result

End-to-end tests with RedhatAI/Qwen2.5-VL-72B-Instruct-FP8-dynamic TP4 on MI300X

lm_eval with ChartQA

Metrics:
{
    "explicit_prompt_relaxed_correctness": 0.89,
    "anywhere_in_answer_relaxed_correctness": 0.892
}

Throughput test

vllm bench serve --port 8088 --backend openai-chat --model RedHatAI/Qwen2.5-VL-72B-Instruct-FP8-dynamic --dataset-name hf --dataset-path lmarena-ai/VisionArena-Chat --hf-split train --endpoint /v1/chat/completions --max-concurrency 64

without fusion

============ Serving Benchmark Result ============
Successful requests:                     1000
Failed requests:                         0
Maximum request concurrency:             64
Benchmark duration (s):                  169.15
Total input tokens:                      94327
Total generated tokens:                  114782
Request throughput (req/s):              5.91
Output token throughput (tok/s):         678.60
Peak output token throughput (tok/s):    2110.00
Peak concurrent requests:                121.00
Total token throughput (tok/s):          1236.26
---------------Time to First Token----------------
Mean TTFT (ms):                          6165.98
Median TTFT (ms):                        6321.34
P99 TTFT (ms):                           11041.16
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          43.59
Median TPOT (ms):                        41.07
P99 TPOT (ms):                           201.45
---------------Inter-token Latency----------------
Mean ITL (ms):                           112.09
Median ITL (ms):                         21.13
P99 ITL (ms):                            1945.71
==================================================

with fusion

============ Serving Benchmark Result ============
Successful requests:                     1000
Failed requests:                         0
Maximum request concurrency:             64
Benchmark duration (s):                  163.26
Total input tokens:                      94327
Total generated tokens:                  114472
Request throughput (req/s):              6.13
Output token throughput (tok/s):         701.17
Peak output token throughput (tok/s):    2298.00
Peak concurrent requests:                121.00
Total token throughput (tok/s):          1278.95
---------------Time to First Token----------------
Mean TTFT (ms):                          5883.03
Median TTFT (ms):                        6015.88
P99 TTFT (ms):                           10457.55
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          43.45
Median TPOT (ms):                        40.10
P99 TPOT (ms):                           140.93
---------------Inter-token Latency----------------
Mean ITL (ms):                           119.03
Median ITL (ms):                         20.26
P99 ITL (ms):                            2053.98
==================================================

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: kliuae <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants