Recipes: Custom AI Assistant kit - The Speech Recognition Distil-Whisper model is very inaccurate. Help!

[LeonMatch]

Hello,
In custom_ai_assistant kit (recipes directory), to which directory do I need to download llama?
This step is missing in the instructions!

Update:
The model have to be downloaded to the root (custom_ai_assistant) directory.

I made it run but having another problem: The Speech Recognition Distil-Whisper model is very inaccurate!
It does not recognize simple words. Completely misses it.
How do I improve it's quality?
I am using Meta-Llama-3-8B-Instr from HuggingFace downloaded into root (custom_ai_assistant) directory.

All details are in the following Issue:
https://github.com/openvinotoolkit/openvino_notebooks/issues/2195

Thanks,
Leon

[brmarkus]

Which notebook under "https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks" are you referring to?

[LeonMatch]

Hello,
I am referring to recipes, custom_ai_assistant.
I actually passed that error. It was due to venv.
However I'm still having issues with this solution. The Speech Recognition Distil-Whisper model is very inaccurate.
It does not recognize simple words. Completely misses it.
How do I improve it's quality?
I am using Meta-Llama-3-8B-Instr from HuggingFace downloaded into root (custom_ai_assistant) directory.

The link to the recipe:
https://github.com/openvinotoolkit/openvino_notebooks/tree/recipes/recipes/custom_ai_assistant

Steps to reproduce:

git clone -b recipes https://github.com/openvinotoolkit/openvino_notebooks.git openvino_notebooks
cd openvino_notebooks/recipes/custom_ai_assistant
conda create -n venv python=3.11 -y
conda activate venv
python -m pip install --upgrade pip
pip install -r requirements.txt
huggingface-cli login
git lfs install
git clone https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct meta-llama/Meta-Llama-3-8B-Instruct
python convert_and_optimize_asr.py --precision int8
python convert_and_optimize_chat.py --chat_model_type llama3-8B --precision int4
python app.py --asr_model_dir model/distil-whisper-large-v2-INT8 --chat_model_dir model/llama3-8B-INT4
access UI at provided url, eg: http://127.0.0.1:xxxx/ and test the model.

[adrianboguszewski]

Please see the answer in #13

[raymondlo84Fork]

I think we have addressed those in our new demo. And one thing I noticed is the use of GPU + INT8 is problematic. I have documented these in the README and hope that will help.

https://github.com/openvinotoolkit/openvino_build_deploy/tree/master/demos/virtual_ai_assistant_demo