Skip to content

Fix Qwen2.5-VL FP8 #272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Fix Qwen2.5-VL FP8 #272

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion examples/llm_ptq/example_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ def get_tokenizer(ckpt_path, trust_remote_code=False, **kwargs):
ckpt_path, trust_remote_code=trust_remote_code, **kwargs
)

if "qwen" in type(tokenizer).__name__.lower():
if "qwen" in type(tokenizer).__name__.lower() and "vl" not in ckpt_path.lower():
# qwen use token id 151643 as pad and eos tokens
tokenizer.pad_token = tokenizer.convert_ids_to_tokens(151643)
tokenizer.eos_token = tokenizer.convert_ids_to_tokens(151643)
Expand Down
4 changes: 4 additions & 0 deletions examples/vlm_ptq/scripts/huggingface_example.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,10 @@ case "${MODEL_TYPE}" in
VISUAL_FEATURE=576
VLM_ARGS=" --max_multimodal_len=$((BUILD_MAX_BATCH_SIZE * VISUAL_FEATURE)) "
;;
"qwen")
VISUAL_FEATURE=1280
VLM_ARGS=" --max_multimodal_len=$((BUILD_MAX_BATCH_SIZE * VISUAL_FEATURE)) "
;;
Comment on lines +148 to +151
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

🧩 Analysis chain

Duplicate 'qwen' case—later branch is unreachable and conflicts with new path.

The first qwen branch (lines 148–151) shadows the second (lines 156–158). ShellCheck SC2221 flags this; remove the later block.

Apply this diff:

@@
     "mllama")
         PTQ_ARGS+=" --kv_cache_qformat none "
         VLM_ARGS=" --max_encoder_input_len=6404 --skip_run"
         ;;
-    "qwen")
-        PTQ_ARGS+=" --kv_cache_qformat none "
-        ;;
 esac

Run:

Also applies to: 156-158

🏁 Script executed:

#!/bin/bash
shellcheck -s bash examples/vlm_ptq/scripts/huggingface_example.sh
rg -nA1 -e '^\s*"qwen"\)' examples/vlm_ptq/scripts/huggingface_example.sh

Length of output: 13458

Remove or consolidate duplicate "qwen" case arms.

The "qwen") pattern appears multiple times in examples/vlm_ptq/scripts/huggingface_example.sh (line 148 and again at lines 156–158 and 224–225); the first occurrence (line 148) overrides the later ones, making them unreachable. Delete the later duplicate arms or merge their needed assignments (e.g., PTQ_ARGS addition and the TRT mapping) into the first "qwen" branch.

🧰 Tools
🪛 Shellcheck (0.10.0)

[warning] 148-148: This pattern always overrides a later one on line 156.

(SC2221)

🤖 Prompt for AI Agents 
In examples/vlm_ptq/scripts/huggingface_example.sh around lines 148 to 151 (and
duplicate arms at lines 156–158 and 224–225), there are multiple "qwen") case
arms where the first occurrence overrides later ones making them unreachable;
remove the duplicate "qwen") branches and consolidate their needed assignments
into the first "qwen") block—merge any PTQ_ARGS additions and TRT mapping
assignments from the later arms into the initial arm so all required variables
are set in one place, then delete the subsequent duplicate case arms.

"mllama")
PTQ_ARGS+=" --kv_cache_qformat none "
VLM_ARGS=" --max_encoder_input_len=6404 --skip_run"
Expand Down
Loading