Skip to content

Reinstate int8_sq support for vlm_example. #333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 18, 2025
Merged

Reinstate int8_sq support for vlm_example. #333

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion CHANGELOG.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ Model Optimizer Changelog (Linux)
- Deprecated ``quantize_mode`` argument in ``examples/onnx_ptq/evaluate.py`` to support strongly typing. Use ``engine_precision`` instead.
- Deprecated TRT-LLM's TRT backend in ``examples/llm_ptq`` and ``examples/vlm_ptq``. Tasks ``build`` and ``benchmark`` support are removed and replaced with ``quant``. For performance evaluation, please use ``trtllm-bench`` directly.
- ``--export_fmt`` flag in ``examples/llm_ptq`` is removed. By default we export to the unified Hugging Face checkpoint format.
- ``int8_sq`` quantization format is deprecated from the ``examples/vlm_ptq`` with respect to the TensorRT-LLM's torch backend switch. Please refer to the previous releases if this quantization format is needed.
- Deprecated ``examples/vlm_eval`` as it depends on the deprecated TRT-LLM's TRT backend.

**New Features**
Expand Down
6 changes: 4 additions & 2 deletions examples/vlm_ptq/scripts/huggingface_example.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,10 @@ if [ -z "$MODEL_PATH" ]; then
fi

case $QFORMAT in
fp8|int4_awq|w4a8_awq|nvfp4)
fp8|int8_sq|int4_awq|w4a8_awq|nvfp4)
;;
*)
echo "Unknown quant argument: Expected one of: [fp8, int4_awq, w4a8_awq, nvfp4]" >&2
echo "Unknown quant argument: Expected one of: [fp8, int8_sq, int4_awq, w4a8_awq, nvfp4]" >&2
exit 1
Comment on lines 37 to 42
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

🧩 Analysis chain

Re‑enablement of int8_sq looks good.

Case arm and error message now include int8_sq; matches the PR goal.

Run to confirm downstream accepts this qformat:

🏁 Script executed:

#!/bin/bash
# Verify hf_ptq.py accepts int8_sq for --qformat and exposes the new inference flags
fd -t f hf_ptq.py
rg -nP -C2 -- 'add_argument\((["'\''])--qformat\1.*choices=.*int8_sq' $(fd -t f hf_ptq.py)
rg -nP -C2 -- 'add_argument\((["'\''])--inference_tensor_parallel\1' $(fd -t f hf_ptq.py)
rg -nP -C2 -- 'add_argument\((["'\''])--inference_pipeline_parallel\1' $(fd -t f hf_ptq.py)

# Check parser provides TP/PP defaults
fd -t f parser.sh
rg -n -C2 '\b(TP|PP)=' $(fd -t f parser.sh)

Length of output: 1012

Add int8_sq to --qformat choices in examples/llm_ptq/hf_ptq.py — hf_ptq.py's argparse for --qformat does not include int8_sq (no match found); --inference_tensor_parallel / --inference_pipeline_parallel are present and examples/llm_ptq/scripts/parser.sh sets TP=1 and PP=1.

🤖 Prompt for AI Agents 
In examples/vlm_ptq/scripts/huggingface_example.sh around lines 37-42 the script
accepts "int8_sq" in the case switch but examples/llm_ptq/hf_ptq.py's argparse
for --qformat does not include "int8_sq", causing a mismatch; update the
argparse definition in examples/llm_ptq/hf_ptq.py to include "int8_sq" among the
allowed choices for --qformat (keeping other existing options intact), and
ensure help text reflects this addition so the CLI and example script are
consistent with TP/PP handling from parser.sh.

esac

Expand Down Expand Up @@ -95,6 +95,8 @@ if [[ $TASKS =~ "quant" ]] || [[ ! -d "$SAVE_PATH" ]] || [[ ! $(ls -A $SAVE_PATH
--qformat=$QFORMAT \
--calib_size=$CALIB_SIZE \
--batch_size=$CALIB_BATCH_SIZE \
--inference_tensor_parallel=$TP \
--inference_pipeline_parallel=$PP \
$PTQ_ARGS
else
echo "Quantized model config $MODEL_CONFIG exists, skipping the quantization stage"
Expand Down
Loading