[BUG] Error on submit job llama 3.2 3B ai - hub

[PhamDangNguyen]

Hello, first of all, I would like to thank my team for their contributions to the AI community.
I am trying to export the LLaMA 3.2 3B model on QC AI Hub.
However, when I submit the job llama_v3_2_3b_instruct_prompt_2_of_3 and I encounter an error:

Additional Information from the Compile Log
Error occurred due to op : _model_model_layers_0_mlp_gate_proj_MatMul;
op_type : FullyConnected;
reason : BQ is not supported
Failed to validate op _model_model_layers_0_mlp_gate_proj_MatMul with error 0xc26 : QNN_OP_PACKAGE_ERROR_VALIDATION_FAILURE
For additional information please refer to the QNN HTP Backend Op Definition Supplement : https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-10/HtpOpDefSupplement.html ```   



This is the command I ran on the host machine (Windows):
``` python -m qai_hub_models.models.llama_v3_2_3b_instruct.export ^
 --chipset qualcomm-sa8295p ^
 --context-length 1024 ^
 --checkpoint DEFAULT_W4 ^
 --skip-profiling ^
 --skip-inferencing ^
 --output-dir output_bin  ```

Please let me know what error I am encountering and how to resolve it!

[shreyajn]

We no longer support this device for Llama models. Earlier to support the device, we had switched some mammals to convs. Recently we found that caused some quality issues on-device. So we reverted the change. Given that, this model no longer converts for SA8295P.

[PhamDangNguyen]

Thank you, @shreyajn. I understand your explanation.

However, when I changed DEFAULT_W4 to DEFAULT_W4A16 and commented out the code that checks DEFAULT_W4A16 with SA8295P, the model was converted successfully by QAI Hub.

Could you please advise whether it is safe or recommended to use this converted model?
Additionally, how can I convert the model to a .bin format?

I would appreciate your response at your earliest convenience.

[shreyajn]

DEFAULT_W4A16 doesn't give good results on SA8295P. It is unfortunate but at the moment no LLMs work on this device. We have internally opened an issue to get this resolved. I don't have a timeline as of now.