Training Config for UniTok MLLM

[zheedong]

Thank you for your great work.

As I know, train.py is a part of the training code. I want to execute them, but I don't have any scripts or configs. Can you let me know?

And also I'd like to know when do you release full training code. Thank you.

[machuofan]

Hi there,

Thanks for your interest in our work. Below is the training config of UniTok MLLM.

# pretrain
torchrun --nproc_per_node= --nnodes= --node_rank= --master_addr= \
    --master_port= liquid/train/train_mem.py \
    --deepspeed ./scripts/zero3.json \
    --model_name_or_path Llama-2-7b-hf/ \
    --data_path Unitok_extracted_code \
    --version plain \
    --reload_embedder True \
    --tune_mmprojector False \
    --pretrained_mmprojector Unitok/llama2_mm_projector.bin \
    --percentage 1.0 \
    --T2I_ratio 1.0 \
    --shuffleseed 0 \
    --vq_resolution 256 \
    --vqconversion False \
    --mm_use_im_start_end False \
    --mm_use_im_patch_token False \
    --output_dir llama2_mix_pretrain_zero3 \
    --run_name 'llama2_mix_pretrain_zero3' \
    --bf16 True \
    --num_train_epochs 1.0 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 4 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 2000 \
    --save_total_limit 5 \
    --learning_rate 5e-5 \
    --weight_decay 1e-3 \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --lazy_preprocess True \
    --report_to wandb


# finetune
deepspeed --master_port 2935 --num_gpus 8 liquid/train/train_mem.py \
    --deepspeed ./scripts/zero3.json \
    --model_name_or_path llama2_mix_pretrain_zero3/  \
    --version v1 \
    --data_path Unitokfull1ep_SFT_data_vicuna^^FLUX_UnitokFULL1EP \
    --reload_embedder False \
    --tune_mmprojector False \
    --percentage 1.0^^0.05 \
    --T2I_ratio 1.0 \
    --shuffleseed 42 \
    --vq_resolution 256 \
    --vqconversion False  \
    --image_folder ./data/MiniGemini-Finetune \
    --mm_projector_type mlp2x_gelu \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end False \
    --mm_use_im_patch_token False \
    --image_aspect_ratio pad \
    --bf16 True \
    --output_dir ./checkpoints/liquid-7b-finetune \
    --run_name 'llama2_mix_finetune' \
    --num_train_epochs 1 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 2000 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --lazy_preprocess True \
    --report_to wandb

[zheedong]

Thank you for your reply. And congratulations for accepted in NeurIPS 2025.

I try to execute your UniTok MLLM training code, but I don't have how does the data looks like. Can you let me know? Or can you share a little sample of dataset?

And I wonder do you have any plan to open your MLLM training code. Thank you!

[machuofan]

Thanks for your congratulations. For UniTok MLLM data, we use the same data structure as Liquid, so you may refer to the repo of Liquid to get an idea of the data formats. We do not plan to make major updates to this repo as we are currently working on a v2 version of UniTok and the MLLM, which will come with new features and full training pipelines.