Unexpectedly Low Training Results with LLaVA-v1.5-7B

[yoon-yoon]

Dear authors,

Thank you for your great work on this project.

I have been training the model under the same environment described in the paper, but I am not getting consistent results. While I have slightly reduced the training scale (e.g., fewer samples), the performance is significantly lower than expected.

Specifically, I am training from scratch using llava-v1.5-7b, but the current results are as follows:

CIoU: 0.074

GIoU: 0.095

This is quite different from what I observed when training the LISA model from scratch in the past, where I obtained:

CIoU: 0.519

GIoU: 0.414

Is there a chance that I might have misconfigured something? Or is such low performance expected in the early stages of training?

Any insights or suggestions would be greatly appreciated.
Thank you again for your valuable work!

Best regards,

def parse_args(args):
    parser = argparse.ArgumentParser(description="READ Model Training")
    parser.add_argument("--local_rank", default=0, type=int, help="node rank")
    parser.add_argument(
        "--version", default="liuhaotian/llava-v1.5-7b"
    )
    parser.add_argument(
        "--precision",
        default="bf16",
        type=str,
        choices=["fp32", "bf16", "fp16"],
        help="precision for inference",
    )
    parser.add_argument("--image_size", default=1024, type=int, help="image size")
    parser.add_argument("--model_max_length", default=512, type=int)
    parser.add_argument("--lora_r", default=8, type=int)
    parser.add_argument(
        "--vision_tower", default="openai/clip-vit-large-patch14-336", type=str
    )
    parser.add_argument("--load_in_8bit", action="store_true", default=False)
    parser.add_argument("--load_in_4bit", action="store_true", default=False)

    parser.add_argument("--dataset", default="refer_seg||correct_refer_seg||vqa||neg_refer_seg", type=str)
    parser.add_argument("--sample_rates", default="9,3,3", type=str)
    parser.add_argument(
        "--sem_seg_data",
        default="ade20k||cocostuff||pascal_part||paco_lvis||mapillary",
        type=str,
    )
    parser.add_argument(
        "--refer_seg_data", default="refclef||refcoco||refcoco+||refcocog", type=str
    )
    parser.add_argument(
        "--neg_refer_seg_data", default="R-refcocog||R-refcoco||R-refcoco+", type=str
    )
    parser.add_argument(
        "--correct_refer_seg_data",
        default="fprefcocog||fprefcoco||fprefcoco+",
        type=str,
    )
    parser.add_argument("--vqa_data", default="llava_instruct_150k", type=str)
    parser.add_argument("--reason_seg_data", default="ReasonSeg|train", type=str)
    parser.add_argument("--val_dataset", default="ReasonSeg", type=str)
    parser.add_argument("--val_split", default="val", type=str)
    parser.add_argument("--dataset_dir", default="./dataset", type=str)
    parser.add_argument("--log_base_dir", default="./runs", type=str)
    parser.add_argument("--exp_name", default="read_referseg", type=str)
    parser.add_argument("--epochs", default=20, type=int)
    parser.add_argument("--steps_per_epoch", default=50, type=int)    ###
    parser.add_argument(
        "--batch_size", default=12, type=int, help="batch size per device per step"
    )
    parser.add_argument(
        "--grad_accumulation_steps",
        default=1,
        type=int,
    )
    parser.add_argument("--val_batch_size", default=1, type=int)
    parser.add_argument("--workers", default=4, type=int)
    parser.add_argument("--lr", default=0.0003, type=float)    ###
    parser.add_argument("--ce_loss_weight", default=1.0, type=float)
    parser.add_argument("--dice_loss_weight", default=0.5, type=float)
    parser.add_argument("--bce_loss_weight", default=2.0, type=float)
    parser.add_argument("--lora_alpha", default=16, type=int)
    parser.add_argument("--lora_dropout", default=0.05, type=float)
    parser.add_argument("--lora_target_modules", default="q_proj,v_proj", type=str)
    parser.add_argument("--beta1", default=0.9, type=float)
    parser.add_argument("--beta2", default=0.95, type=float)
    parser.add_argument("--num_classes_per_sample", default=1, type=int)
    parser.add_argument("--no_eval", action="store_true", default=False)
    parser.add_argument("--eval_only", action="store_true", default=False)
    parser.add_argument("--vision_pretrained", default="PATH_TO_SAM_ViT-H", type=str)
    parser.add_argument("--out_dim", default=256, type=int)
    parser.add_argument("--resume", default="", type=str)
    parser.add_argument("--print_freq", default=3, type=int)
    parser.add_argument("--start_epoch", default=0, type=int)
    parser.add_argument("--gradient_checkpointing", action="store_true", default=True)
    parser.add_argument("--train_mask_decoder", action="store_true", default=True)
    parser.add_argument("--use_mm_start_end", action="store_true", default=True)
    parser.add_argument("--auto_resume", action="store_true", default=False)
    parser.add_argument("--use_wandb", action="store_true", default=False)
    parser.add_argument("--use_released_param", action="store_true", default=False)   ###
    parser.add_argument(
        "--conv_type",
        default="llava_v1",
        type=str,
        choices=["llava_v1", "llava_llama_2"],
    )

VERSION="../dataset/llava-v1.5-7b"
VISION_TOWER="../dataset/clip-vit-large-patch14-336"
DATASET_DIR="../dataset"
MODEL_MAX_LENGTH=2048
EXP_NAME="READ-LLaVA-v1.5-7B"
BATCH_SIZE=2     
GRAD_ACCUMULATION_STEPS=10
NUM_CLASSES_PER_SAMPLE=2

[rui-qian]

CIoU: 0.074

GIoU: 0.095

Was this result obtained during training or during evaluation after training was completed?

[yoon-yoon]

These results (CIoU: 0.074, GIoU: 0.095) were obtained during training.

May I ask how the CIoU and GIoU metrics typically evolve during training in your experiments?
Specifically, I’m curious about the progression:

• when training from scratch, and
• when fine-tuning from SESAME.

Understanding the expected trend would help me better assess my own training process.

[rui-qian]

These results (CIoU: 0.074, GIoU: 0.095) were obtained during training.

May I ask how the CIoU and GIoU metrics typically evolve during training in your experiments? Specifically, I’m curious about the progression:

• when training from scratch, and
• when fine-tuning from SESAME.

Understanding the expected trend would help me better assess my own training process.

When training from scratch (e.g., LLaVA 1.5-13B),

When fine-tuning from SESAME (e.g., SESAME-LLaVA-v1.5-7B),

Fine-tuning from SESAME can converge quickly—typically achieving good results within about 4 epochs.

I noticed that the results you mentioned (CIoU: 0.074, GIoU: 0.095) were obtained during training. My suggestion is to observe the metrics in the first 1–2 epochs; usually, a CIoU of around 0.25–0.35 in the first epoch indicates normal training from scratch. I’ve also encountered results like CIoU: 0.074 and GIoU: 0.095 in two situations: (1) when initially training the LISA model, and (2) when experimenting with different combinations of (CLIP@336px, CLIP@224px) × (SAM-ViT-Base, SAM-ViT-Large, SAM-ViT-Huge). In case (2), I resolved the issue by reducing the learning rate.

Feel free to reach out if you have any questions.

[yoon-yoon]

Thank you so much for your kind and detailed response.

I have a few additional questions, if you don’t mind:

1. Could you share any insights or recommendations on setting the learning rate? If you've discovered any patterns or best practices during your experiments, I’d greatly appreciate it.

2. If I fine-tune based on the READ-7B checkpoint, would reducing the default learning rate (e.g., 0.0001) even further be beneficial in your experience?

Thank you again for your time and support!

[rui-qian]

Thank you so much for your kind and detailed response.

I have a few additional questions, if you don’t mind:

1. Could you share any insights or recommendations on setting the learning rate? If you've discovered any patterns or best practices during your experiments, I’d greatly appreciate it.
2. If I fine-tune based on the READ-7B checkpoint, would reducing the default learning rate (e.g., 0.0001) even further be beneficial in your experience?

Thank you again for your time and support!

1. As mentioned in the README.md:

" We found that the hyperparameters: lora_r, lr, dice_loss_weight, num_points (in SESAME.py), and SAMPLE_RATES_REASONSEG (in train_sesame.sh) have a significant impact on model performance. Trying different combinations of these hyperparameters may lead to performance improvements. See also: READ-7B, READ-13B. "

2. Simply reducing the default learning rate (e.g., to 0.0001) may not lead to significant improvements, especially if the model is already training normally(i.e., the first epoch looks fine, with cIoU around 0.25–0.35). Based on my experience, a smaller learning rate tends to stabilize performance and reduce fluctuations. However, due to limited computational resources (approximately 1–2 80GB GPUs), these observations may not be entirely conclusive — you are encouraged to experiment and validate through your own practice.

Feel free to reach out if you have any questions.

[yoon-yoon]

Hi, thank you for your impressive work and for sharing the code.

I have a question regarding the RefCOCO results reported in your paper.
Could you please clarify how the model was trained to obtain those results?

For example:

• Did you fine-tune the model specifically on RefCOCO?
• What was the training setup?
• Were there any special tricks or preprocessing steps?

I would greatly appreciate any details you can share.
Thank you again!

[rui-qian]

Hi, thank you for your impressive work and for sharing the code.

I have a question regarding the RefCOCO results reported in your paper. Could you please clarify how the model was trained to obtain those results?

For example:

• Did you fine-tune the model specifically on RefCOCO?
• What was the training setup?
• Were there any special tricks or preprocessing steps?

I would greatly appreciate any details you can share. Thank you again!

Sorry for the late reply. We are currently building a systematic framework that will be released soon — it may help address your performance concerns.

1. We did not fine-tune the model specifically on RefCOCO. Instead, it is trained with multiple datasets: LLaVA VQA, RefCOCO*, R-RefCOCO*, and FP-RefCOCO*. In fact, we trained the model using different mixing ratios by modifying:

   DATASET_REASONSEG="sem_seg||refer_seg||correct_refer_seg||vqa||neg_refer_seg||reason_seg"
   SAMPLE_RATES_REASONSEG="10,10,2,3,1,1"

   You may consider training exclusively on RefCOCO* data if the current performance does not meet your expectations — this might help.

2. Regarding the training setup, all configuration details are preserved in the following archive:
   https://huggingface.co/datasets/rui-qian/misc/blob/main/READ-13B.zip

3. As for tricks or special preprocessing steps — as far as we know, question templates and the reasoning ability of the LLM (i.e., the version of LLaVA used) greatly affect performance. For example, upgrading from LLaVA 1.5 to LLaVA 1.6 brings significant improvements.

   Also, as mentioned on the GitHub page:

   "We found that the hyperparameters: lora_r, lr, dice_loss_weight, num_points (in SESAME.py), and SAMPLE_RATES_REASONSEG (in train_sesame.sh) have a significant impact on model performance."

Feel free to reach out if you need help!

[rui-qian]

Dear authors,

Thank you for your great work on this project.

I have been training the model under the same environment described in the paper, but I am not getting consistent results. While I have slightly reduced the training scale (e.g., fewer samples), the performance is significantly lower than expected.

Specifically, I am training from scratch using llava-v1.5-7b, but the current results are as follows:

CIoU: 0.074

GIoU: 0.095

This is quite different from what I observed when training the LISA model from scratch in the past, where I obtained:

CIoU: 0.519

GIoU: 0.414

Is there a chance that I might have misconfigured something? Or is such low performance expected in the early stages of training?

Any insights or suggestions would be greatly appreciated. Thank you again for your valuable work!

Best regards,

def parse_args(args):
    parser = argparse.ArgumentParser(description="READ Model Training")
    parser.add_argument("--local_rank", default=0, type=int, help="node rank")
    parser.add_argument(
        "--version", default="liuhaotian/llava-v1.5-7b"
    )
    parser.add_argument(
        "--precision",
        default="bf16",
        type=str,
        choices=["fp32", "bf16", "fp16"],
        help="precision for inference",
    )
    parser.add_argument("--image_size", default=1024, type=int, help="image size")
    parser.add_argument("--model_max_length", default=512, type=int)
    parser.add_argument("--lora_r", default=8, type=int)
    parser.add_argument(
        "--vision_tower", default="openai/clip-vit-large-patch14-336", type=str
    )
    parser.add_argument("--load_in_8bit", action="store_true", default=False)
    parser.add_argument("--load_in_4bit", action="store_true", default=False)

    parser.add_argument("--dataset", default="refer_seg||correct_refer_seg||vqa||neg_refer_seg", type=str)
    parser.add_argument("--sample_rates", default="9,3,3", type=str)
    parser.add_argument(
        "--sem_seg_data",
        default="ade20k||cocostuff||pascal_part||paco_lvis||mapillary",
        type=str,
    )
    parser.add_argument(
        "--refer_seg_data", default="refclef||refcoco||refcoco+||refcocog", type=str
    )
    parser.add_argument(
        "--neg_refer_seg_data", default="R-refcocog||R-refcoco||R-refcoco+", type=str
    )
    parser.add_argument(
        "--correct_refer_seg_data",
        default="fprefcocog||fprefcoco||fprefcoco+",
        type=str,
    )
    parser.add_argument("--vqa_data", default="llava_instruct_150k", type=str)
    parser.add_argument("--reason_seg_data", default="ReasonSeg|train", type=str)
    parser.add_argument("--val_dataset", default="ReasonSeg", type=str)
    parser.add_argument("--val_split", default="val", type=str)
    parser.add_argument("--dataset_dir", default="./dataset", type=str)
    parser.add_argument("--log_base_dir", default="./runs", type=str)
    parser.add_argument("--exp_name", default="read_referseg", type=str)
    parser.add_argument("--epochs", default=20, type=int)
    parser.add_argument("--steps_per_epoch", default=50, type=int)    ###
    parser.add_argument(
        "--batch_size", default=12, type=int, help="batch size per device per step"
    )
    parser.add_argument(
        "--grad_accumulation_steps",
        default=1,
        type=int,
    )
    parser.add_argument("--val_batch_size", default=1, type=int)
    parser.add_argument("--workers", default=4, type=int)
    parser.add_argument("--lr", default=0.0003, type=float)    ###
    parser.add_argument("--ce_loss_weight", default=1.0, type=float)
    parser.add_argument("--dice_loss_weight", default=0.5, type=float)
    parser.add_argument("--bce_loss_weight", default=2.0, type=float)
    parser.add_argument("--lora_alpha", default=16, type=int)
    parser.add_argument("--lora_dropout", default=0.05, type=float)
    parser.add_argument("--lora_target_modules", default="q_proj,v_proj", type=str)
    parser.add_argument("--beta1", default=0.9, type=float)
    parser.add_argument("--beta2", default=0.95, type=float)
    parser.add_argument("--num_classes_per_sample", default=1, type=int)
    parser.add_argument("--no_eval", action="store_true", default=False)
    parser.add_argument("--eval_only", action="store_true", default=False)
    parser.add_argument("--vision_pretrained", default="PATH_TO_SAM_ViT-H", type=str)
    parser.add_argument("--out_dim", default=256, type=int)
    parser.add_argument("--resume", default="", type=str)
    parser.add_argument("--print_freq", default=3, type=int)
    parser.add_argument("--start_epoch", default=0, type=int)
    parser.add_argument("--gradient_checkpointing", action="store_true", default=True)
    parser.add_argument("--train_mask_decoder", action="store_true", default=True)
    parser.add_argument("--use_mm_start_end", action="store_true", default=True)
    parser.add_argument("--auto_resume", action="store_true", default=False)
    parser.add_argument("--use_wandb", action="store_true", default=False)
    parser.add_argument("--use_released_param", action="store_true", default=False)   ###
    parser.add_argument(
        "--conv_type",
        default="llava_v1",
        type=str,
        choices=["llava_v1", "llava_llama_2"],
    )

VERSION="../dataset/llava-v1.5-7b"
VISION_TOWER="../dataset/clip-vit-large-patch14-336"
DATASET_DIR="../dataset"
MODEL_MAX_LENGTH=2048
EXP_NAME="READ-LLaVA-v1.5-7B"
BATCH_SIZE=2     
GRAD_ACCUMULATION_STEPS=10
NUM_CLASSES_PER_SAMPLE=2

We recently released a high-performance framework and recommend using it for training: UGround.