Add support for LLaVA model

[youssefadr]

🚀 The feature, motivation and pitch

LLaVA seems to be currently a strong open-source competitor to GPT4-V, it doesn't seem to be supported by the library. Do you plan on adding it? If yes, is there something I could contribute with to help?

Alternatives

No response

Additional context

No response

[ebsmothers]

Hi @youssefadr, thanks for opening this issue. LLaVA is definitely something we're interested in adding and we would be happy to have you contribute. Is there a specific portion of the model you're especially interested in helping out with?

[youssefadr]

Thanks for your answer @ebsmothers, I would like to add the model to torchmultimodal/models first.

[ebsmothers]

That sounds reasonable to me. We already have CLIP visual encoders in the library here, so feel free to reuse those. Then the bulk of the work for the model should be to add the LLM. A couple pointers to help with that: TransformerDecoderLayer, RMSNorm. We also have an open PR for rotary positional embeddings (#450) that might be useful. Let me know if this makes sense, happy to provide more details as needed.

[youssefadr]

Nice ! I'll come back to you with more questions later, not sure I'll start working on it this week.

[theadamsabra]

@youssefadr have you worked on this to any capacity? i'm interested in picking this up if not

[ebsmothers]

@theadamsabra if not, you are more than welcome to take it up

[theadamsabra]

@ebsmothers thanks! If I don't get a response by tomorrow I'll just pick it up myself