Support Granite 4 models

[Christian24]

Description of the bug:

Hello,

The Granite 4 models by IBM seem to be especially great at low parameter counts. It would be great to have them supported in the LiteRT ecosystem.

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response

[sourcelite]

Hi @Christian24, welcome here and thanks for this feature request. True, both the Granite 3.0 and 4.0 series are great coz they are all rounder and also great function calling, maybe make a great model for on-device accelerator. Let us check the possibility.

just curious - maybe this info be helpful to us.
1.] what Granite model series (traditional or Hybrid H model series) you using ? Granite has few very well known flavors. Granite-4.0-H-350M, H-1B, or the H-Small MoE using any Hybrid Mamba-2 layers ?, these are traditional Granite-4.0-Micro 3B). these are standard dense transformers. traditional are better for NPU acceleration, hybrid needs to map new mamba 2 kernal to NPU.
2.] what is your target hardware ? android/iOS/web
general info I have is -
Android (Qualcomm/Mediatek NPU): requires 4-bit/8-bit quantisation and AOT compilation
iOS (Metal/CoreML): requires specific FP16 optimization
Web (WebGPU/ML Drift): requires the new LiteRT WebGPU backend

@aravindmurali22 can we mark this as feature request and any further steps ?

[Christian24]

1. The normal Granite 4 model in 350M Parameters. Currently mostly technology demo. I might switch to 1B parameters later, but so far 350 M seems to work great.
2. Web / WebAI, so browser

[Christian24]

@sourcelite I did submit a feature request, but the template looks more like a bug template. Maybe the configuration is wrong? I just checked. It seems the template for feature requests is indeed a bug template.

[aravindmurali22]

Hi @g-davidmassoud could you please take a look into this? Thanks!

[g-davidmassoud]

We have Granite 4 on the roadmap but I can't share any timelines for completion.

[aravindmurali22]

@g-davidmassoud , thanks for the response. In this case we can keep this issue open to track this feature request. Thanks.

[protobird-git]

@Christian24 have you tried export_hf with ibm-granite/granite-4.0-h-350m? It would be great to see if it works out of box to export Granite 4 models for your use cases.

[Christian24]

@protobird-git I just tried it. This is my output:

============== Export Configuration ==============
auto_model_override    : None
batch_size             : 1
bundle_litert_lm       : True
cache_implementation   : 'LiteRTLMCache'
cache_length           : 4096
cache_length_dim       : None
enable_dynamic_shape   : False
experimental_lightweight_conversion : False
experimental_use_mixed_precision : False
export_vision_encoder  : False
externalize_embedder   : False
externalize_rope       : False
extra_kwargs           : {}
k_ts_idx               : 2
keep_temporary_files   : False
litert_lm_llm_metadata_override : None
litert_lm_model_type_override : None
model                  : 'ibm-granite/granite-4.0-h-350m'
output_dir             : './tmp/exported-model'
prefill_length_dim     : None
prefill_lengths        : [256]
quantization_recipe    : 'int8_weight_only'
single_token_embedder  : False
split_cache            : False
task                   : 'text_generation'
trust_remote_code      : False
use_jinja_template     : True
v_ts_idx               : 3
vision_encoder_quantization_recipe : 'weight_only_wi8_afp32'
work_dir               : '/Users/christian/Documents/workspace/litert-torch/tmp/exported-model/tmpntwi3_nl'
==================================================
(00:00) [START] LiteRT GenAI Export
(00:00) [START] LiteRT GenAI Export > Load source model

                   .
                  / \
                 / ! \
                /  .  \
               /_______\
              /  ERROR  \
             /___________\
            /_____________\
           /_______________\


Model is not fully compilable.
(00:37) [ DONE] LiteRT GenAI Export > Load source model (+00:37)
(00:37) [START] LiteRT GenAI Export > Export text prefill-decode model
(00:37) [START] LiteRT GenAI Export > Export text prefill-decode model > LiteRT-Torch 
Convert
(00:37) [START] LiteRT GenAI Export > Export text prefill-decode model > LiteRT-Torch 
Convert > Torch Export: prefill_256
(00:38) [ FAIL] LiteRT GenAI Export > Export text prefill-decode model > LiteRT-Torch 
Convert > Torch Export: prefill_256
(00:38) [ FAIL] LiteRT GenAI Export > Export text prefill-decode model > LiteRT-Torch 
Convert
(00:38) [ FAIL] LiteRT GenAI Export > Export text prefill-decode model
(00:38) [ FAIL] LiteRT GenAI Export

The command I ran was litert-torch export_hf \ --model ibm-granite/granite-4.0-h-350m \ --output_dir ./tmp/exported-model \ --task text_generation \ --quantization_recipe int8_weight_only.

Btw is it just me or are the nightly builds not up to date? I only saw a single 0.8.0 build on pip. Hence why I installed from source.

[protobird-git]

@Christian24 Thanks for the retry. Really helpful and appreciated.

@sirakiin for visibility, especially for nightly build