How to use opencompass to evaluate LLaVA's LLM on text-only tasks.

[yash0307]

In instruction-tuning, LLaVA trains the LLM as well. I want to evaluate the LLM after the instruction tuning for text-only tasks such as MMLU. How to do this? From what I understand, LLaVA saves the projection layer together with the LLM, which is failing the evaluation attempts. Any workarounds?

[bittersweet1999]

Thanks for your attention, we move multi-modality model evaluation to this repo:
https://github.com/open-compass/MMBench
You can try to find in it

[yash0307]

@bittersweet1999, thank you for a prompt response. I am looking for a way to evaluate LLaVA on text-only benchmarks, not on multi-modal evaluations.

[bittersweet1999]

If you intend to use OpenCompass to evaluate LLaVA for text-only tasks, you might consider trying the configuration found at https://github.com/open-compass/opencompass/tree/main/configs/multimodal/llava
They may still be usable.