Results differ when running multiple tasks

[pumetu]

Hello, thank you for your work! I'm encountering a problem when evaluating tinyllava on multiple tasks, where the results differ compared to running each task on its own. The commands are exactly the same between the two runs, only differing in the amount of tasks ran. I have also attached the results from each as an example.

Commands:
Two tasks command:

python3 -m accelerate.commands.launch \
    --num_processes=4 \
    -m lmms_eval \
    --model tinyllava \
    --model_args pretrained="$PATH,conv_mode=$conv_mode" \
    --tasks mme,mmstar \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $suffix \
    --output_path outputs/ \
    --verbosity=DEBUG

Multiple tasks command:

python3 -m accelerate.commands.launch \
    --num_processes=4 \
    -m lmms_eval \
    --model tinyllava \
    --model_args pretrained="$PATH,conv_mode=$conv_mode" \
    --tasks mme,mmstar,mmvet,pope,scienceqa,textvqa,gqa,vizwiz_vqa,seedbench_2_plus \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix $suffix \
    --output_path outputs/ \
    --verbosity=DEBUG

Results:
Two tasks:

Multiple tasks:

Here we see that the mme and mmstar results are completely different compared to the earlier image when we run just two tasks (mme and mmstar) with mmstar getting 0% in the multiple tasks command compared to 23% when we run just mme and mmstar (the same story happens for mme).

[kcz358]

Hi @pumetu , thank you for the report. This seems to be an important issue. Have you checked the logs that the result are consistent for mme and mmstar under different settings? I will try to reproduce this result recently.

[pumetu]

Hi @kcz358, I've found that running them individually is fine but I forgot to log other combinations (some were fine some failed) I can test a few more and get back to you. A side note, I found that some benchmarks received a very low score compared to other implementations (ScienceQA and textVQA), I tested the official tinyllava implementation for scienceQA and got around 60-80% but with the same model on lmms it falls to 20-35%.

[kcz358]

Thanks and sorry for the late reply, I forget to reproduce this error recently. I will assign this to myself and check if I can find out the source of error.

[MichelDucartier]

I know this is old, but this might be an issue related to tinyllava in particular, especially this line.

If the gen_kwargs differ from one task to another, tinyllava will take the wrong gen_kwargs for some tasks (as it uses the same gen_kwargs no matter the task)