[Question] Argument use_liger_kernel missing in train_sft.py

[Bunnybeibei]

Hello,

I noticed in the latest codebase that the use_liger argument appears to have been removed from train_sft.py.

I wonder if the latest version still supports Liger Kernel?

Thanks!

[2U1]

Yes it still supports liger_kernel. I've just leaved it to huggingface trainer to apply it.

[Bunnybeibei]

Thank you for the reply.

However, I encountered a significant initialization delay when setting --use_liger_kernel True for Qwen3-VL fine-tuning. The process has been stuck in initialization for over 2 hours, whereas the startup is immediate when I set it to False.

I am using 4x A100 GPUs with DeepSpeed ZeRO-3. Could you please help identify if there are any misconfigurations in my script below that might be causing this bottleneck?

Here is my training script:

#!/bin/bash

# MODEL_NAME="Qwen/Qwen2-VL-7B-Instruct"
# MODEL_NAME="Qwen/Qwen2-VL-2B-Instruct"
# MODEL_NAME="Qwen/Qwen2.5-VL-3B-Instruct"
# MODEL_NAME="Qwen/Qwen2.5-VL-7B-Instruct"

export CUDA_VISIBLE_DEVICES=0,1,2,3
MODEL_NAME="Qwen/Qwen3-VL-8B-Instruct"

GLOBAL_BATCH_SIZE=128
BATCH_PER_DEVICE=2
NUM_DEVICES=4
GRAD_ACCUM_STEPS=$((GLOBAL_BATCH_SIZE / (BATCH_PER_DEVICE * NUM_DEVICES)))

export PYTHONPATH=src:$PYTHONPATH

# Note: For Qwen3-VL, image_min_pixels and max_pixels should be set as (N * 32 * 32)

deepspeed --master_port 29502 src/train/train_sft.py \
    --use_liger_kernel True \
    --deepspeed scripts/zero3.json \
    --model_id $MODEL_NAME \
    --data_path data/description.json \
    --image_folder data/train \
    --remove_unused_columns False \
    --freeze_vision_tower False \
    --freeze_llm False \
    --freeze_merger False \
    --bf16 True \
    --fp16 False \
    --disable_flash_attn2 False \
    --output_dir output2/test_fft \
    --num_train_epochs 100 \
    --per_device_train_batch_size $BATCH_PER_DEVICE \
    --gradient_accumulation_steps $GRAD_ACCUM_STEPS \
    --image_min_pixels $((512 * 32 * 32)) \
    --image_max_pixels $((1800 * 32 * 32)) \
    --learning_rate 1e-5 \
    --merger_lr 1e-5 \
    --vision_lr 2e-6 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --gradient_checkpointing True \
    --report_to tensorboard \
    --lazy_preprocess

[2U1]

#!/bin/bash

# MODEL_NAME="Qwen/Qwen2-VL-7B-Instruct"
# MODEL_NAME="Qwen/Qwen2-VL-2B-Instruct"
# MODEL_NAME="Qwen/Qwen2.5-VL-3B-Instruct"
# MODEL_NAME="Qwen/Qwen2.5-VL-7B-Instruct"

MODEL_NAME="Qwen/Qwen3-VL-8B-Instruct"

GLOBAL_BATCH_SIZE=8
BATCH_PER_DEVICE=2
NUM_DEVICES=4
GRAD_ACCUM_STEPS=$((GLOBAL_BATCH_SIZE / (BATCH_PER_DEVICE * NUM_DEVICES)))

export PYTHONPATH=src:$PYTHONPATH

# If you want to set the min pixels and max pixels for Qwen3-VL, You should set as (N * 32 * 32)

deepspeed src/train/train_sft.py \
    --use_liger_kernel True \
    --deepspeed scripts/zero3.json \
    --model_id $MODEL_NAME \
    --data_path /home/workspace/Qwen-VL-Series-Finetune/vlm20/conversations.json \
    --image_folder /home/workspace/Qwen-VL-Series-Finetune/vlm20/images \
    --remove_unused_columns False \
    --freeze_vision_tower False \
    --freeze_llm False \
    --freeze_merger False \
    --bf16 True \
    --fp16 False \
    --disable_flash_attn2 False \
    --output_dir output/test_fft \
    --num_train_epochs 1 \
    --per_device_train_batch_size $BATCH_PER_DEVICE \
    --gradient_accumulation_steps $GRAD_ACCUM_STEPS \
    --image_min_pixels $((512 * 32 * 32)) \
    --image_max_pixels $((1280 * 32 * 32)) \
    --learning_rate 1e-5 \
    --merger_lr 1e-5 \
    --vision_lr 2e-6 \
    --weight_decay 0.1 \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --gradient_checkpointing True \
    --report_to tensorboard \
    --lazy_preprocess True \
    --save_strategy "steps" \
    --save_steps 200 \
    --save_total_limit 10 \
    --dataloader_num_workers 4

I've tested with this setting and it works with both 8b and 4b on 4xA100(The issue is it takes really awfully long time). But I'm not sure why it takes more time than not using liger-kernel.
I think it's a problem of liger-kernel.

I could recommend the either one for now.

• Turn off liger-kernel
• Use zero2
• Use lora (This is fast)

I'll ask about this in the liger-kernel repo.

[pulkitkumar95]

Encountering the same issue, currently solved it by turning off liger kernel

[2U1]

@pulkitkumar95 I've made a issue in liger-kernel github. And responded they couldn't reproduce this issue. I'm still not sure why it happens.
I'll make some experiments when I have some time.

[rayyychen]

I encountered the same issue on A40 with zero2. The process stuck in initialization for a long time

[2U1]

@rayyychen For now, could you use the liger-kernel false?

[rayyychen]

Yes, I can run finetune_lora with liger_kernel set to False, but I can't set it to True

[2U1]

@rayyychen I couldn't really find the way, you could use liger-kernel when lora or turn it off.

[minhduc01168]

@2U1 If I use --use_liger_kernel False, I always encounter the following situation:
GLOBAL_BATCH_SIZE=4
BATCH_PER_DEVICE=2
NUM_DEVICES=1

Adding LoRA to the model...
[2026-01-22 09:55:47,372] [INFO] [real_accelerator.py:260:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-01-22 09:55:49,298] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 151645, 'bos_token_id': None, 'pad_token_id': 151643}.
Warning: The following arguments do not match the ones in the trainer_state.json within the checkpoint directory:
per_device_train_batch_size: 2 (from args) != 4 (from trainer_state.json)
{'loss': 0.3319, 'grad_norm': 5.558448314666748, 'learning_rate': 8.411214953271028e-06, 'epoch': 0.35}
{'loss': 0.3073, 'grad_norm': 1.1356265544891357, 'learning_rate': 1.775700934579439e-05, 'epoch': 0.36}
{'loss': 0.3522, 'grad_norm': 1.6197096109390259, 'learning_rate': 2.7102803738317755e-05, 'epoch': 0.37}
{'loss': 0.3168, 'grad_norm': 1.7248080968856812, 'learning_rate': 3.644859813084112e-05, 'epoch': 0.37}
{'loss': 0.2926, 'grad_norm': 1.9702904224395752, 'learning_rate': 4.579439252336449e-05, 'epoch': 0.38}
13%|███████████████████▉ | 451/3534 [01:53<1:44:21, 2.03s/it][rank0]: Traceback (most recent call last):
[rank0]: File "/home/dienvv/Qwen-VL-Series-Finetune/src/train/train_sft.py", line 291, in
[rank0]: train()
[rank0]: File "/home/dienvv/Qwen-VL-Series-Finetune/src/train/train_sft.py", line 263, in train
[rank0]: trainer.train(resume_from_checkpoint=True)
[rank0]: File "/home/dienvv/anaconda3/envs/qwen/lib/python3.12/site-packages/transformers/trainer.py", line 2325, in train
[rank0]: return inner_training_loop(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/dienvv/anaconda3/envs/qwen/lib/python3.12/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/dienvv/anaconda3/envs/qwen/lib/python3.12/site-packages/transformers/trainer.py", line 4071, in training_step
[rank0]: self.accelerator.backward(loss, **kwargs)
[rank0]: File "/home/dienvv/anaconda3/envs/qwen/lib/python3.12/site-packages/accelerate/accelerator.py", line 2734, in backward
[rank0]: loss.backward(**kwargs)
[rank0]: File "/home/dienvv/anaconda3/envs/qwen/lib/python3.12/site-packages/torch/_tensor.py", line 647, in backward
[rank0]: torch.autograd.backward(
[rank0]: File "/home/dienvv/anaconda3/envs/qwen/lib/python3.12/site-packages/torch/autograd/init.py", line 354, in backward
[rank0]: _engine_run_backward(
[rank0]: File "/home/dienvv/anaconda3/envs/qwen/lib/python3.12/site-packages/torch/autograd/graph.py", line 829, in _engine_run_backward
[rank0]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 9.93 GiB. GPU 0 has a total capacity of 39.39 GiB of which 5.78 GiB is free. Including non-PyTorch memory, this process has 33.60 GiB memory in use. Of the allocated memory 27.65 GiB is allocated by PyTorch, and 5.00 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
13%|████████████████████▏ | 451/3534 [01:56<13:16, 3.87it/s]
[2026-01-22 09:57:48,519] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 62669
[2026-01-22 09:57:48,519] [ERROR] [launch.py:325:sigkill_handler] ['/home/dienvv/anaconda3/envs/qwen/bin/python3.12', '-u', 'src/train/train_sft.py', '--local_rank=0', '--use_liger_kernel', 'False', '--lora_enable', 'True', '--vision_lora', 'True', '--use_dora', 'False', '--lora_namespan_exclude', "['lm_head', 'embed_tokens']", '--lora_rank', '32', '--lora_alpha', '64', '--lora_dropout', '0.05', '--num_lora_modules', '-1', '--model_id', 'Qwen/Qwen3-VL-2B-Instruct', '--data_path', 'dataset/qwen_finetune_vietnamese.json', '--image_folder', 'dataset/images/', '--remove_unused_columns', 'False', '--freeze_vision_tower', 'True', '--freeze_llm', 'True', '--freeze_merger', 'True', '--bf16', 'True', '--fp16', 'False', '--disable_flash_attn2', 'False', '--output_dir', 'output/qwen_finetuned_lora_1gpu', '--num_train_epochs', '3', '--per_device_train_batch_size', '2', '--gradient_accumulation_steps', '2', '--image_min_pixels', '200704', '--image_max_pixels', '1003520', '--learning_rate', '1e-4', '--merger_lr', '1e-5', '--vision_lr', '2e-6', '--weight_decay', '0.1', '--warmup_ratio', '0.03', '--lr_scheduler_type', 'cosine', '--logging_steps', '10', '--tf32', 'True', '--gradient_checkpointing', 'True', '--report_to', 'tensorboard', '--lazy_preprocess', 'True', '--save_strategy', 'steps', '--save_steps', '200', '--save_total_limit', '10', '--dataloader_num_workers', '4'] exits with return code = 1

[2U1]

@minhduc01168 I think you are resuming from the checkpoint and the previous setting was set to batch size 4.
You could rename the output_dir to other name.