Steps to Convert A LLM model ( for example Function Gemma ) to litertlm format for Qualcomm NPU

[chandan2300]

Description of the bug:

Hi LiteRT team currently do we have any steps to convert Gemma Model or any LLM model to .litertlm format for Qualcomm NPU? I was not able to find any resources.

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response

[aravindmurali22]

Hi @chandan2300 , thanks for filing this issue.

@sourcelite, could you please look into this? Thanks.

[sourcelite]

Hi @chandan2300, welcome here. Thanks @aravindmurali22.

The following steps are involved to get a gemma model running on Qualcomm NPUs. Also I given this link below will be helpful to you - Link: Run LLMs on NPU using LiteRT-LM https://ai.google.dev/edge/litert/next/litert_lm_npu

Simply briefing the steps for your reference purpose - Use the ai edge torch generative api key for converting pytorch model into tflite format. After that go for quantization - apply the w8a16 which is 8 bit weight and 16 bit activation. These specs are specially optimised for Qualcomm Snapdragon NPUs. Lastly you need to bundle or combine that tflite model and tokenjser into a .litertlm bundle ( it can even be in .task format). Finally AOT compilation for max performance on qcomm pls use the litert Qualcomm ai engine direct QNN accelerator for pre compile of model for any specific SoC chip.

Moreover you can also check the LiteRT GenAI Model Zoo on Hugging Face for quicker start link - There you will see few preconverted .litertlm files for different different Gemma models.

I hope this info + link helps you. Thanks

[sourcelite]

@chandan2300 I found this info on huggingface page -

pls find these ready-to-use .litertlm files for Gemma here following links: you will like to use them -

Gemma 2 2B: this will be found in huggingface path - huggingface.co/litert-community/Gemma2-2B-IT
Gemma 3 1B: this will be found in huggingface path - huggingface.co/litert-community/Gemma3-1B-IT

I feel referring to the provided link will remain more relevant and also provide the info you are looking for - Run LLMs on NPU using LiteRT-LM https://ai.google.dev/edge/litert/next/litert_lm_npu

Thanks, anything more you may want, please keep posted here.

[chandan2300]

Hi @sourcelite , thanks a lot for the response. I already have the sample app setup provided here (https://ai.google.dev/edge/litert/next/litert_lm_npu). I'm seeing on how i can generate .literlm for Qualcomm NPU for function Gemma model (currently in HF repo we have CPU model and safetensors).

I have 2 questions -

1. I understand the steps to convert the model to .tflite model using AOT, but how to bundle it into .litertlm format? (Could you please share example scripts and documentation I did not find any)

2. How to compile the model for specific Qualcomm SOC (Again looking for some resources and sample scripts to do the same)

I understand that there are some pre-generated models I'm looking for steps which can be used for any custom model.

Thanks for your time and looking forward to your response.

[sourcelite]

Hi @chandan2300, sry for the delay response. Please take these official resources which are document - Python APIs for Bundling and AOT Compilation.

1. Documentation for Model Bundling (.litertlm files)

I too searched public docs, at present there is no standalone "bundler" page you looking for, but instructions are the part of the MediaPipe LLM Inference Guide, specified under the "PyTorch Model Conversion" section. This guide document is the mediapipe.tasks.python.genai.bundler API. Link is MediaPipe LLM Inference > PyTorch Model Conversion You need to go to - section named "Create a Task Bundle" section, which details the BundleConfig parameters such as start_token and stop_tokens.

2. Documentation for Qualcomm AOT Compilation

The Python API for AOT compilation (ai_edge_litert.aot) is documented in at NPU Acceleration Guide.This link may help you NPU Acceleration with LiteRT > AOT Compilation. There you search for - The Python snippet using aot_compile and qnn_target.

Qualcomm Specifics: Qualcomm NPU (AI Engine Direct) Guide. This can be also helpful - the list of supported SocModel enums (e.g., SM8650, SM8550) to ensure you passing an correct target to the compiler.

3. Fast way to move ahead for you is this example scripts (which are extracted from docs)

A. Bundling Script (Matches MediaPipe Docs)

# Reference: https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference#pytorch_model_conversion
from mediapipe.tasks.python.genai import bundler

config = bundler.BundleConfig(
    tflite_model="gemma_function_calling.tflite",
    tokenizer_model="tokenizer.model",
    start_token="<bos>",
    stop_tokens=["<eos>", "<end_of_turn>"],
    output_filename="gemma_function_calling.litertlm"
)
bundler.create_bundle(config)

B. AOT Compilation Script (Matches NPU Docs)

# Reference link is: https://ai.google.dev/edge/litert/next/npu
from ai_edge_litert.aot import aot_compile as aot_lib
from ai_edge_litert.aot.vendors.qualcomm import target as qnn_target

# Sample Target the Snapdragon 8 Gen 3 (SM8650)
target_soc = qnn_target.Target(qnn_target.SocModel.SM8650)

compiled_models = aot_lib.aot_compile(
    tflite_model_path="gemma_function_calling_int8.tflite",
    targets=[target_soc]
)

with open("gemma_function_calling_qnn.tflite", "wb") as f:
    f.write(compiled_models[target_soc])

I hoping this google links + script will help you unblock and move ahead. Thanks!

[chandan2300]

@sourcelite thanks for the response, but Step B is still generating .tflite model rite? so are u suggesting i first do AOT compile and then bundle it to litertlm format using mediapipe bundler? Will this .litertlm model generated be compatible to Qualcomm NPU though im using mediapipe bundler?

[sourcelite]

Hi @chandan2300, sry for delay. You are correct, let me fix and respond again. Previously I treated generating the TFLite as one step, but actually in NPU it is actually two distinct TFLite generations. This is what missed by my side last time, (just anaology here) the distinction that bundle specially requires the second QNN TFLite file to actually work on hardware. We cannot (bundle) it before we polish (AOT) it, else NPU will receive the wrong version.

AOT first then Mediapine Bundler packaging step.
Additional info: Android device actually needs to match the SoC (example - SM8650 for Gen 3), else if we run SM8650 compiled bundle on SM8550 (Gen 2) it will fail to load

AOT compile optimisation step:

from ai_edge_litert.aot import aot_compile as aot_lib
from ai_edge_litert.aot.vendors.qualcomm import target as qnn_target

target_soc = qnn_target.Target(qnn_target.SocModel.SM8650)

compiled_models = aot_lib.aot_compile(
    tflite_model_path="gemma_function_calling_int8.tflite",
    targets=[target_soc]
)

qnn_model_path = "gemma_function_calling_qnn.tflite"
with open(qnn_model_path, "wb") as f:
    f.write(compiled_models[target_soc])

Bundling packaging step.

from mediapipe.tasks.python.genai import bundler

config = bundler.BundleConfig(
    # IMPORTANT: Use the QNN model from Phase 1 here!
    tflite_model="gemma_function_calling_qnn.tflite",
    tokenizer_model="tokenizer.model",
    start_token="<bos>",
    stop_tokens=["<eos>", "<end_of_turn>"],
    output_filename="gemma_qnn_function_calling.litertlm"
)
bundler.create_bundle(config)

You were correct, I suggest you now to try it. Please let us know. Thanks!

[github-actions]

Marking this issue as stale since it has been open for 7 days with no activity. This issue will be closed if no further activity occurs.