Is there any typo in the model https://huggingface.co/listen2you002/ChartLlama-13b/tree/main?

[yanchuqiao]

Hello. Thank you for releasing all these amazing work.
I am encountering a ClipVision issue. When I try to load the Lora weigths from ChartLlama-13b model, the "mm_vision_tower": "/mnt/share_1227775/yandali/multimodal/models/ft_local/clip-vit-large-patch14-336/" was n't accessiable. I changed it to the openai/clip-vit-large-patch14-336, but then it has another problem of generating texts.

I am wondering is there any typo or issues here? Maybe I am wrong. Canyou help to fix it?

Thank you so much

[sbY99]

I faced the same issue. Did you resolve it?

[moured]

How I fixed the issue:

1. First, download the listen2you002/ChartLlama-13 model from Hugging Face locally.
2. Next, download the liuhaotian/llava-v1.5-13b model from Hugging Face locally.
3. Then, download the openai/clip-vit-large-patch14-336 model from Hugging Face locally.
4. Navigate to the folder where ChartLlama-13 was downloaded, and in the config.json, replace "mm_vision_tower" with the full path of clip-vit-large-patch14-336.
5. For inference, pass the local paths of both liuhaotian/llava-v1.5-13b and listen2you002/ChartLlama-13.

It was a mismatch of paths. Hope this helps!