Questions about text_generator_main.cc

[liamsun2019]

Hi authors,
I have several questions about text_generator_main.cc.

1. The calculation of kv_cache_max_size
   int kv_cache_max_size = kv_cache_k_0->dims->data[1];
   For gemma3-1b, kv_cache_k_0->dims->data[1] is always 1. The following logic of setting decode_steps:
   int decode_steps =
   std::min(max_decode_steps, kv_cache_max_size - prefill_seq_size);
   will result in a negative value of decode_steps and the MINIMAL_CHECK(decode_steps > 0) fails.

2. I just simplly set kv_cache_max_size = kv_cache_k_0->dims->data[2] to get through decode_steps check. Then I executed following command:
   ./text_generator_main --tflite_model=gemma3-1b_q8_ekv1280.tflite --sentencepiece_model=tokenizer.model --prompt="What is Tensorflow?" --max_decode_steps=256 --start_token="<bos>" --stop_token="<eos>" --num_threads=2
   where, gemma3-1b_q8_ekv1280.tflite was generated by ai_torch_edge with "full_int8_dynamic_recipe" quantized. The results look normal.
   As a comparison, I also tested gemma3-1B-it-int4.tflite which was downloaded from https://www.kaggle.com/models/google/gemma-3/tfLite
   The command was similar:
   ./text_generator_main --tflite_model=gemma3-1B-it-int4.tflite --sentencepiece_model=tokenizer.model --prompt="What is Tensorflow?" --max_decode_steps=256 --start_token="<bos>" --stop_token="<eos>" --num_threads=2
   The results looked abnormal:
   100010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010010
   One note that I must point out, the original bazel building of text_generator_main does not support int4 quantization due to the tensorflow version used in the building. Therefore, I build text_generator_main combined with tensorflow 2.20.0 by cmake.

Any suggestions are appreciated. Big thanks.

[gaikwadrahul8]

Hi, @liamsun2019
I apologize for the delayed response, I was trying to replicate same behavior from my end so at the moment I tried gemma3-1B-it-int4.tflite and I'm getting the EMBEDDING_LOOKUP error so it seems like compatibility issue with the INT4 quantized model. The model expects UInt8 or Int8 tensors but is getting a different type.

(ai-edge-env) gaikwadrahul@gaikwadrahul:~/ai-edge-torch-test$ ./text_generator_main \
    --tflite_model=models/gemma3-1B-it-int4.tflite \
    --sentencepiece_model=models/tokenizer.model \
    --prompt="What is Tensorflow?" \
    --max_decode_steps=256 \
    --start_token="<bos>" \
    --stop_token="<eos>" \
    --num_threads=2
normalizer.cc(52) LOG(INFO) precompiled_charsmap is empty. use identity normalization.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
ERROR: external/org_tensorflow/tensorflow/lite/kernels/embedding_lookup.cc:72 value->type == kTfLiteUInt8 || value->type == kTfLiteInt8 was not true.
ERROR: Node number 1 (EMBEDDING_LOOKUP) failed to prepare.
Error at ai_edge_torch/generative/examples/cpp/text_generator_main.cc:171
(ai-edge-env) gaikwadrahul@gaikwadrahul:~/ai-edge-torch-test$ 

To confirm, you followed this official documentation to get gemma3-1b_q8_ekv1280.tflite if so could you please help me with exact steps which you followed so replicate same behavior from our end? Thank you for your cooperation and patience.

[liamsun2019]

Hi @gaikwadrahul8 ,
Thanks for your response. The errors you encountered came from the tflite library which did not support int4 quantization. I conducted my tests based on tensorflow 2.20.0 where int4 quantization was fully supported. Meanwhile, I generated gemma3-1b_q4_block32_ekv1280.tflite and tested it. The results looked normal. So I have no idea what's the problem with gemma3-1B-it-int4.tflite on https://www.kaggle.com/models/google/gemma-3/tfLite. Maybe it needs a different way to do inference? I am not sure.

BTW, do you have any plan to support gemma3-4b with multimodal?
Thanks for your time.

[github-actions]

Marking this issue as stale since it has been open for 7 days with no activity. This issue will be closed if no further activity occurs.

[github-actions]

This issue was closed because it has been inactive for 14 days. Please post a new issue if you need further assistance. Thanks!

[protobird-git]

@liamsun2019 Sorry for the delayed reply for a follow-up question.

We support multimodality for Gemma3n-2B/4B with LiteRT-LM, https://github.com/google-ai-edge/LiteRT-LM?tab=readme-ov-file#multimodal