Support for VLM Conversion with Encoder-Decoder Architecture

[sagnik-charlie]

Description of the bug:

Currently, the litert_torch library supports decoder-only model conversion, even for VLMs like qwen_vl, paligemma or smolvlm2 those mentioned in litert_torch.generative.examples. However, the export feature in litert_torch.genenerative.export_hf.export file mentions something about 'task'. By mentioning the task as 'image_text_to_text' the VLM will be loaded as pytorch model. But again from the exportable_module in litert_torch.genenerative.export_hf.core it uses the decoder only model conversion script. Using both generative examples and export_hf files, the litert_torch library still supports a decoder only model conversion, which is task like 'llm_chat' and 'llm_prompt_lab' for litert-lm engine runtime. The litert-lm engine expects 'TF_LITE_VISION_ENCODER' and 'TF_LITE_AUDIO_ENCODER_HW' for image and audio support along with the decoder, the code for which is still missing. Can you please look into this problem, and provide a solution for this?

Actual vs expected behavior:

VLM Conversion, expects both the encoder as well as the decoder to be exported. But however for all the mentioned VLMs, the VLM model returns a decoder with valid signatures. Due to this, only 'llm_chat' and 'llm_prompt_lab' tasks are supported in the Google AI Edge Gallery App, whereas the tasks like 'llm_ask_image' and 'llm_ask_audio' expects a 'TF_LITE_VISION_ENCODER' and 'TF_LITE_AUDIO_ENCODER_HW' respectively inside the .litertlm bundle. The conversion for which is still not supported using the litert_torch library.

Any other information you'd like to share?

No response

[sourcelite]

Hi @sagnik-charlie, welcome here, thanks for this feature request - Native support for disaggregated VLM export. If you feeding litert-torch a Vision_Language_Model VLM then it is treating it like a text generator model, by missing the encoder vision part.

Your suggestion: export_hf tool for auto detect and split of the model into two distinct LiteRT subgraphs -

1. TF_LITE_VISION_ENCODER: image processing tower (e.g., CLIP/SigLIP)
2. TF_LITE_LM_DECODER: text generation tower

Impact of this feature: It will allow LiteRT LM runtime to orchestrate the two parts more efficiently - even good for apps like AI Edge Gallery, maybe run the vision once and cache it and then run text generation.

https://github.com/google-ai-edge/litert-torch/tree/main/litert_torch/generative/examples/gemma3 this and many examples here shows that we will like to have VLM supported, but now we need to add split-export utility inside it. Thanks for raising this issue.

[sagnik-charlie]

Hi @sourcelite,

Thank you for your response.
I found that currently LiteRT-LM runtime is looking for Vision Encoder and Vision Adapter model types inside the bundled .litertlm with specific model signatures. So we have to make separate tflite models for both vision encoder and vision adapter for SigLIP encoder of gemma-3-4b. However in https://github.com/google-ai-edge/litert-torch/tree/main/litert_torch/generative/examples/gemma3, the Gemma3MM model class is trying to pass both pixel values and text tokens, but the flow of the logic implemented here is somehow breaking the LiteRT-LM runtime flow of acknowledging Vision-Language Models. As you said splitting will allow LiteRT LM runtime to orchestrate the two parts more efficiently, so we don't need to bundle different tflite models together. Also, the spilt-export utility you talked about sounds interesting. Looking forward to it.

Thanks.

[pfarinha91]

I also want to raise awareness for this issue. After a lot of work trying to convert a VLM model (MedGemma 4B IT) based on the available Gemma3 scripts (only compatible with text-to-text 270M and 1B variants) and on the PaliGemma example, I was able to convert it successfully to tflite, bundle it with both litertlm and MediaPipe task bundler, but was unable to test it on AI Edge Gallery due to the engine expecting 'TF_LITE_VISION_ENCODER' for any vision enabled model.