Issues Training GRIT with Qwen2.5-VL-3B-Instruct

[David-19940718]

Hi, authors, thanks for your awesome work!

I'm attempting to train Qwen/Qwen2.5-VL-3B-Instruct using the provided training script, but I've encountered several issues that I'd like to clarify:

Training Script

#!/bin/bash

setting='dozen_vsr_qwen_add_grounded_reasoning_single_turn_think_rethink'
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export WANDB_PROJECT=$setting

# Load config variables
source scripts/train_base_config.sh

# Run the training script with DeepSpeed
python -m accelerate.commands.launch \
    --config_file ./accelerate_configs/deepspeed_zero2.yaml \
    --main_process_port 20092 \
    grpo-gr/GRPO_GR.py \
    --train_data_path ./GRIT_data/tallyqa_train_10.jsonl,./GRIT_data/vsr_cot_train_10.jsonl \
    --train_image_folder_path ./GRIT_data/tallyqa,./GRIT_data/vsr \
    --eval_data_path ./GRIT_data/vsr_val.jsonl,./GRIT_data/mme_val.jsonl,./GRIT_data/tallyqa_val.jsonl,./GRIT_data/gqa_val.jsonl,./GRIT_data/mathvista_mini_val.jsonl,./GRIT_data/ovd_position_val.jsonl,./GRIT_data/ovd_relationship_val.jsonl,./GRIT_data/ovd_negation_val.jsonl \
    --eval_image_folder_path ./GRIT_data/vsr,./GRIT_data/mme,./GRIT_data/tallyqa,./GRIT_data/gqa,./GRIT_data/mathvista_mini,./GRIT_data/ovd_position,./GRIT_data/ovd_relationship,./GRIT_data/ovd_negation \
    --setting $setting \
    --max_turns 1 \
    --output_dir output/$setting \
    --hub_model_id $setting \
    $COMMON_ARGS \
    --eval_steps 50 \
    --save_steps 50 \
    --num_train_epochs 500 \
    --lr_scheduler_type cosine \
    --per_device_eval_batch_size 8

1. Dataset Issues

MME Dataset

Most datasets can be downloaded normally, but for the MME dataset, when I try to download from the repository path specified in the paper (link), I find that the image names in the downloaded files don't match the names listed in mme_val.jsonl.

Missing Label Files

The following label files are missing:

• ./GRIT_data/ovd_relationship_val.jsonl
• ./GRIT_data/ovd_negation_val.jsonl

Could you please provide these files or clarify how to obtain them?

2. Flash Attention Issues

During initial training, I encounter the following warning/error:

You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
...

The specific error indicates that float16 is not supported. I resolved this by manually specifying torch_dtype=torch.bfloat16 during model initialization. Did you encounter this issue during your training? What's the recommended approach to handle this?

GRIT/grpo-gr/GRPO_GRTrainer.py

Lines 233 to 234 in fd08d57

	if "qwen" in model_id.lower():
	model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model, **model_init_kwargs)

3. Training Hyperparameters

I'd like to confirm a few things about the training parameters:

1. Epochs: Is --num_train_epochs 500 an experimental parameter? This seems quite high - is this intentional?

2. Batch Size & Memory: When training on 48GB VRAM, I can only set per_device_train_batch_size to 1, otherwise I get OOM errors. Is this normal? If the batch size can only be 1, should the learning rate be scaled accordingly? What would be the recommended values?

3. Other Parameters: Are the other hyperparameters in the script reasonable for this model size and task?

4. Demo Environment

Regarding the gradio_qwen.py mentioned on the GitHub page, where can I find this file? It doesn't seem to be included in the current repository.

---

Environment:

• Model: Qwen/Qwen2.5-VL-3B-Instruct
• GPU: 8x GPUs with 48GB VRAM each
• Framework: DeepSpeed ZeRO-2

5. Logs

Also, It's very weird that reward scores always get zero.

Any guidance on these issues would be greatly appreciated. Thank you again for your work on this project!

[hungnh1125]

I’m experiencing the same issue. I’ve tried the following approaches, but the loss remains at 0 in all cases:

1. Used the same settings from the GitHub repository with a batch size of 4, training on 8 A100 80GB GPUs.
2. Tried a batch size of 1 on 8 A100 40GB GPUs.
3. Trained with the dataset mentioned in the paper and my own custom dataset.
4. Trained with the original Qwen 2.5 checkpoint as well as the pretrained checkpoint provided by the authors on GitHub.

Based on these tests, I believe the issue is not related to the batch size, dataset, or pretrained model.

[UeFan]

Hi, sorry for the late reply; I was sick since last weekend and am still suffering from a bad fever. I will provide a short, quick answer for now and update the repo later in the week. Sorry for the inconvenience.

@David-19940718

1. I will provide the script for processing MME data and for now you may remove mme_eval.jsonl, ovd_negation_val.jsonl and ovd_relationship_val.jsonl.

2. That should solve the problem. I think this is due to that I missed the parameter in the run script, I will add that later.
   3.1 As the training data only has 20 samples, each epoch only contains one step. I feel free to either early stop the training or just change --num_train_epochs to something smaller.
   3.2 I haven't tried it but I suggest
   --per_device_train_batch_size 1 --gradient_accumulation_steps 10

3. I will provide that later.

4. could you make some debug attempt in the reward functions in the rewards.py at
   except Exception:
   I could not reproduce the zero reward issue.

@hungnh1125 Could you specific if you face the 0 loss issue or 0 rewards issue?

[David-19940718]

Hi @UeFan,

Thanks so much for the quick reply, especially while you're not feeling well. I really appreciate you taking the time. Please prioritize your health and get plenty of rest!

Regarding point 4 (the zero reward issue), and your thought in point 2 about a missed parameter in the run script, I did some debugging, and I think I might have found a possible cause.

It seems to be related to training_args.setting. I noticed that in the shell script(s) you have provided, the string add_grounded_thinking doesn't appear to be included in the setting argument.

Based on the code logic:

if "add_grounded_thinking" in training_args.setting:
    prompt = ""
    prompt_suffix = ""
    if "_think_rethink" in training_args.setting:
        prompt_suffix += " First, think between <think> and </think> while output necessary coo..."
    # ... other conditions
elif "SFT" in training_args.setting:
    # ...
else:
    prompt_suffix = ""
    prompt = ""

If add_grounded_thinking is not present in training_args.setting (and neither is SFT, for example), the prompt and prompt_suffix could remain as empty strings (or not be populated with the specific instructions for thinking steps).

This would mean the LLM wouldn't be explicitly prompted to include the <think>, <rethink>, and <answer> tags in its output. Consequently, the reward function would very likely evaluate the output as not conforming to the expected structure, definitely leading to a zero reward score.

This could explain why you couldn't reproduce the zero reward issue if your local tests perhaps included add_grounded_thinking in the setting, while the script I'm referencing might have missed it.

I'll also try out your other suggestions (removing the jsonl files for now, and looking into the epoch/batch size settings) and continue exp and report here.

Thanks again for your help and insights!

Hope you feel much better very soon. Take care!

Best regards,

[David-19940718]

Q1: Dataset Correctness (tallyqa_train_10)

After inspecting the tallyqa_train_10 dataset, it appears that many of the bounding box coordinates don't accurately align with the target objects described by the labels. I've tried overlaying the labels and coordinates on the images to verify this. Wouldn't such misalignments significantly impact the model's ability to learn grounded reasoning or accurate tool use, as it would be learning from incorrect ground truth? I can provide some visual examples if that would be helpful. train_dataset_vis.zip

In addition, for the mathvista_mini_val dataset, the images downloaded from the provided link are all in JPG format. However, the image field in the label file provided by the project uses PNG extensions. Were these changed later, or is there a discrepancy?

---

Q2: Another issue was raised: the training phase breaks after approximately 50 steps. The terminal output is shown below:

click

[rank3]:   File "/root/grpo-gr/GRPO_GR.py", line 182, in <module>
[rank3]:     trainer.train()
[rank3]:   File "/root/miniconda3/envs/GRIT/lib/python3.12/site-packages/transformers/trainer.py", line 2243, in train
[rank3]:     return inner_training_loop(
[rank3]:            ^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/GRIT/lib/python3.12/site-packages/transformers/trainer.py", line 2614, in _inner_training_loop
[rank3]:     self._maybe_log_save_evaluate(
[rank3]:   File "/root/miniconda3/envs/GRIT/lib/python3.12/site-packages/transformers/trainer.py", line 3087, in _maybe_log_save_evaluate
[rank3]:     metrics = self._evaluate(trial, ignore_keys_for_eval)
[rank3]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/GRIT/lib/python3.12/site-packages/transformers/trainer.py", line 3041, in _evaluate
[rank3]:     metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
[rank3]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/GRIT/lib/python3.12/site-packages/transformers/trainer.py", line 4089, in evaluate
[rank3]:     dataset_metrics = self.evaluate(
[rank3]:                       ^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/GRIT/lib/python3.12/site-packages/transformers/trainer.py", line 4107, in evaluate
[rank3]:     output = eval_loop(
[rank3]:              ^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/GRIT/lib/python3.12/site-packages/transformers/trainer.py", line 4301, in evaluation_loop
[rank3]:     losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
[rank3]:                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/grpo-gr/GRPO_GRTrainer.py", line 1079, in prediction_step
[rank3]:     inputs = self._prepare_inputs(inputs, is_train=False)
[rank3]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/grpo-gr/GRPO_GRTrainer.py", line 635, in _prepare_inputs
[rank3]:     image_input, video_input = process_vision_info(inp['message'])
[rank3]:                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/grpo-gr/vision_process.py", line 364, in process_vision_info
[rank3]:     image_inputs.append(fetch_image(vision_info))
[rank3]:                         ^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/grpo-gr/vision_process.py", line 119, in fetch_image
[rank3]:     image = to_rgb(image_obj)
[rank3]:             ^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/grpo-gr/vision_process.py", line 94, in to_rgb
[rank3]:     return pil_image.convert("RGB")
[rank3]:            ^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]:   File "/root/miniconda3/envs/GRIT/lib/python3.12/site-packages/PIL/Image.py", line 993, in convert
[rank3]:     self.load()
[rank3]:   File "/root/miniconda3/envs/GRIT/lib/python3.12/site-packages/PIL/ImageFile.py", line 297, in load
[rank3]:     raise OSError(msg)
[rank3]: OSError: image file is truncated (20 bytes not processed)

To avoid this potential issue, I suggest adding the following code:

# grpo-gr/vision_process.py
from PIL import Image
Image.LOAD_TRUNCATED_IMAGES = True

...
def to_rgb(pil_image: Image.Image) -> Image.Image:
    try:
        if pil_image.mode == 'RGBA':
            white_background = Image.new("RGB", pil_image.size, (255, 255, 255))
            white_background.paste(pil_image, mask=pil_image.split()[3])
            return white_background
        else:
            return pil_image.convert("RGB")
    except OSError as e:
        if "truncated" in str(e).lower():
            # For truncated images, try to create a new RGB image from the data we have
            rgb_image = Image.new("RGB", pil_image.size, (255, 255, 255))
            try:
                rgb_image.paste(pil_image, (0, 0))
            except:
                pass
            return rgb_image
        raise

This will allow PIL to load images even if they are slightly corrupted or incomplete.

---

Q3: Below is the training curve. Could you please confirm whether it looks normal or if there might be an issue? If you want to more context, feel free to tell me. If you need any more context, feel free to ask.

---

Q4: I noticed a discrepancy in the GRPO_GRTrainer.py implementation. During evaluation, the reward_name list contains 6 elements, but the corresponding reward_list reports 7 values.

The issue appears to be in the _prepare_inputs method, where an extra value of -3 is explicitly appended to each reward_list:

# In _prepare_inputs method
return_content = {
    # ...
    "reward_name": [reward_func_name.__name__ for reward_func_name in self.reward_funcs],
    "reward_list": current_batch_rewards.tolist(),
    # ...
}

for i in range(len(return_content['reward_list'])):
    return_content['reward_list'][i].append(-3)  # This adds an extra element

This causes the lengths to be mismatched (7 values in reward_list vs 6 names in reward_name).

Is this extra -3 value intended for some specific purpose? If so, should there be a corresponding name in the reward_name list?

Thank you!

[hungnh1125]

@UeFan Thank you for your reply. I hope you recover soon.

I also updated the name in the settings as suggested by @David-19940718, and the rewards are now non-zero. However, the loss remains zero. After investigating, I found that the issue is with the "advantages"—they are all zero, which explains why the loss is also zero (https://github.com/eric-ai-lab/GRIT/blob/master/grpo-gr/GRPO_GRTrainer.py#L1020).

Do you have any insights into what might be causing this?

[hungnh1125]

@UeFan I’m encountering another issue while fine-tuning Qwen2.5 on my custom dataset (24 samples):

When using the initial weights from Qwen/Qwen2.5-VL-3B-Instruct, the training crashes with a CUDA out-of-memory error after about 30 steps.
I also tried fine-tuning with the pretrained weights from yfan1997/GRIT-20-Qwen2.5-VL-3B, but the same issue occurred—even with a batch size of 2 on an A100 80GB GPU.
Do you have any suggestions to address this problem?

[UeFan]

Hi @UeFan,

Thanks so much for the quick reply, especially while you're not feeling well. I really appreciate you taking the time. Please prioritize your health and get plenty of rest!

Regarding point 4 (the zero reward issue), and your thought in point 2 about a missed parameter in the run script, I did some debugging, and I think I might have found a possible cause.

It seems to be related to training_args.setting. I noticed that in the shell script(s) you have provided, the string add_grounded_thinking doesn't appear to be included in the setting argument.

Based on the code logic:

if "add_grounded_thinking" in training_args.setting:
prompt = ""
prompt_suffix = ""
if "_think_rethink" in training_args.setting:
prompt_suffix += " First, think between and while output necessary coo..."
# ... other conditions
elif "SFT" in training_args.setting:
# ...
else:
prompt_suffix = ""
prompt = ""
If add_grounded_thinking is not present in training_args.setting (and neither is SFT, for example), the prompt and prompt_suffix could remain as empty strings (or not be populated with the specific instructions for thinking steps).

This would mean the LLM wouldn't be explicitly prompted to include the <think>, <rethink>, and <answer> tags in its output. Consequently, the reward function would very likely evaluate the output as not conforming to the expected structure, definitely leading to a zero reward score.

This could explain why you couldn't reproduce the zero reward issue if your local tests perhaps included add_grounded_thinking in the setting, while the script I'm referencing might have missed it.

I'll also try out your other suggestions (removing the jsonl files for now, and looking into the epoch/batch size settings) and continue exp and report here.

Thanks again for your help and insights!

Hope you feel much better very soon. Take care!

Best regards,

Thank you. Yes, you are quite right here. I made the mistake of not syncing the "add_grounded_reasoning" and "add_grounded_thinking". I am going to make them all "add_grounded_reasoning".

[UeFan]

After inspecting the tallyqa_train_10 dataset, it appears that many of the bounding box coordinates don't accurately align with the target objects described by the labels. I've tried overlaying the labels and coordinates on the images to verify this. Wouldn't such misalignments significantly impact the model's ability to learn grounded reasoning or accurate tool use, as it would be learning from incorrect ground truth? I can provide some visual examples if that would be helpful. train_dataset_vis.zip

This is fine. The reason is that the training does not rely on the bounding boxes. Only evaluation needs ground truth bounding boxes and we have made sure the bounding boxes in validation data is much better (we manually examined the bounding boxes from VSR and GQA).

[UeFan]

In addition, for the mathvista_mini_val dataset, the images downloaded from the provided link are all in JPG format. However, the image field in the label file provided by the project uses PNG extensions. Were these changed later, or is there a discrepancy?

I only reformat them from JPEGs to PNGs. Sorry for the inconvenience. I will share the processing scripts.

[UeFan]

Q2: Another issue was raised: the training phase breaks after approximately 50 steps. The terminal output is shown below:

click
To avoid this potential issue, I suggest adding the following code:

grpo-gr/vision_process.py

from PIL import Image
Image.LOAD_TRUNCATED_IMAGES = True

...
def to_rgb(pil_image: Image.Image) -> Image.Image:
try:
if pil_image.mode == 'RGBA':
white_background = Image.new("RGB", pil_image.size, (255, 255, 255))
white_background.paste(pil_image, mask=pil_image.split()[3])
return white_background
else:
return pil_image.convert("RGB")
except OSError as e:
if "truncated" in str(e).lower():
# For truncated images, try to create a new RGB image from the data we have
rgb_image = Image.new("RGB", pil_image.size, (255, 255, 255))
try:
rgb_image.paste(pil_image, (0, 0))
except:
pass
return rgb_image
raise
This will allow PIL to load images even if they are slightly corrupted or incomplete.

Thanks for the suggestion. I didn't meet this error before, but I will add these.

[UeFan]

Q3: Below is the training curve. Could you please confirm whether it looks normal or if there might be an issue? If you want to more context, feel free to tell me. If you need any more context, feel free to ask.

I attached one example training log below. it is using the same setting as introduced in the paper. I think yours looking wired. It seems like the model is not learning. While I think more about the potential causes, could you let me know the args you passed in in the .sh file corresponding to your training log?

[UeFan]

Q4: I noticed a discrepancy in the GRPO_GRTrainer.py implementation. During evaluation, the reward_name list contains 6 elements, but the corresponding reward_list reports 7 values.

The issue appears to be in the _prepare_inputs method, where an extra value of -3 is explicitly appended to each reward_list:

In _prepare_inputs method

return_content = {
# ...
"reward_name": [reward_func_name.name for reward_func_name in self.reward_funcs],
"reward_list": current_batch_rewards.tolist(),
# ...
}

for i in range(len(return_content['reward_list'])):
return_content['reward_list'][i].append(-3) # This adds an extra element
This causes the lengths to be mismatched (7 values in reward_list vs 6 names in reward_name).

Is this extra -3 value intended for some specific purpose? If so, should there be a corresponding name in the reward_name list?

Thank you!

Thanks for reporting this. These are lines that supposed be removed. I missed it during some branch merging.

[UeFan]

@UeFan Thank you for your reply. I hope you recover soon.

I also updated the name in the settings as suggested by @David-19940718, and the rewards are now non-zero. However, the loss remains zero. After investigating, I found that the issue is with the "advantages"—they are all zero, which explains why the loss is also zero (https://github.com/eric-ai-lab/GRIT/blob/master/grpo-gr/GRPO_GRTrainer.py#L1020).

Do you have any insights into what might be causing this?


Hi, I think your training log looks fine for me. The loss is small at the beginning as the model at the beginning always gives less ideal outputs and receives almost 0 rewards. There is nothing to "reinforce" if the models outputs all bad outputs.

Below is the loss from one of my training.

[UeFan]

@UeFan I’m encountering another issue while fine-tuning Qwen2.5 on my custom dataset (24 samples):

When using the initial weights from Qwen/Qwen2.5-VL-3B-Instruct, the training crashes with a CUDA out-of-memory error after about 30 steps. I also tried fine-tuning with the pretrained weights from yfan1997/GRIT-20-Qwen2.5-VL-3B, but the same issue occurred—even with a batch size of 2 on an A100 80GB GPU. Do you have any suggestions to address this problem?

Does your model produce longer and longer output? I think one easy fix is to use deepspeed zero 3 instead of zero 2, you can do this by --config_file ./accelerate_configs/deepspeed_zero3.yaml, but this reduce the training speed for at least 2X.

[David-19940718]

Q3: Below is the training curve. Could you please confirm whether it looks normal or if there might be an issue? If you want to more context, feel free to tell me. If you need any more context, feel free to ask.

I attached one example training log below. it is using the same setting as introduced in the paper. I think yours looking wired. It seems like the model is not learning. While I think more about the potential causes, could you let me know the args you passed in in the .sh file corresponding to your training log?

Sure!

#!/bin/bash

setting='dozen_vsr_qwen_add_grounded_reasoning_single_turn_think_rethink'
export CUDA_VISIBLE_DEVICES=0,1,2,3
export WANDB_PROJECT=$setting

# Load config variables
source scripts/train_base_config.sh

# Run the training script with DeepSpeed
python -m accelerate.commands.launch \
    --config_file ./accelerate_configs/deepspeed_zero2_4.yaml \
    --main_process_port 20092 \
    grpo-gr/GRPO_GR.py \
    --train_data_path ./GRIT_data/tallyqa_train_10.jsonl,./GRIT_data/vsr_cot_train_10.jsonl \
    --train_image_folder_path ./GRIT_data/tallyqa,./GRIT_data/vsr \
    --eval_data_path ./GRIT_data/vsr_val.jsonl,./GRIT_data/tallyqa_val.jsonl,./GRIT_data/gqa_val.jsonl,./GRIT_data/mathvista_mini_val.jsonl,./GRIT_data/ovd_position_val.jsonl \
    --eval_image_folder_path ./GRIT_data/vsr,./GRIT_data/tallyqa,./GRIT_data/gqa,./GRIT_data/mathvista_mini,./GRIT_data/ovd_position \
    --setting $setting \
    --max_turns 1 \
    --output_dir output/$setting \
    --hub_model_id $setting \
    $COMMON_ARGS \
    --eval_steps 100 \
    --save_steps 100 \
    --num_train_epochs 200 \
    --torch_dtype bfloat16 \
    --lr_scheduler_type cosine \
    --per_device_eval_batch_size 4 \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 4

Regarding the report in the paper, I noticed that the single-GPU batch size reached 16 (using ZeRO-2) on an A100 (80GB), while under the same conditions, the batch size could only go up to 1 on a 4090 (45GB). Isn't this supposed to scale more or less linearly?