RAG Performance & Fairness Evaluation Toolkit (OpenVINO + LangChain) #3114
Conversation
…. The toolkit computes standard metrics (BERT, BLEU, ROUGE, perplexity score) and a racial-bias indicator, and it is implemented using Optimum-Intel’s OVModelForCausalLM with the OpenVINO backend and LangChain for orchestration.
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
Hi @pkhara31 , the notebook https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-rag-langchain/llm-rag-langchain.ipynb seems to cover the same tasks you described in comment. Could you please check ? Does this match your idea of the notebook you wanted to add ? |
|
Hi @sbalandi , This NB covers the methodology to evaluate the performance of RAG pipeline by computing the BERT, BLEU, ROGUE, perplexity, racial bias scores. |
|
Thanks for your contribution @pkhara31 Do you think if we can add this NB to RAG notebook, as a separate |
|
Another idea is to change this |
|
The model loading redundancy can be avoided, but I think the evaluation and the ability to apply RAG pipeline on web docs (scraping) should still remain in the helper. |
|
May I know the latest update on this? |
Hi, sorry for the delay, I'm in the process of reviewing notebook, I'll write comments within a week |
| langchain_community | ||
| msoffcrypto-tool | ||
| docx2txt | ||
| urllib |
There was a problem hiding this comment.
I don't think we need urllib , os , typing here as os/typing are build in modules
also, please, move requiremetns to notebook and use pip_install(), please, check example in https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-rag-langchain/llm-rag-langchain.ipynb , Prerequisites section
| @@ -0,0 +1,662 @@ | |||
| { | |||
There was a problem hiding this comment.
Why these modules needed ?
Please, add openvino, "optimum[openvino,nncf,onnxruntime]"
Reply via ReviewNB
| @@ -0,0 +1,662 @@ | |||
| { | |||
There was a problem hiding this comment.
we can use pre-converted model here https://huggingface.co/OpenVINO/Phi-3-mini-4k-instruct-int4-ov
also, please, save model via model.save_pretrained("ov_model")
Reply via ReviewNB
| @@ -0,0 +1,662 @@ | |||
| { | |||
There was a problem hiding this comment.
embedding_model_name = "BAAI/bge-small-en-v1.5"
It would be also great to save the model so you don't have to download it every time
embedding = OpenVINOBgeEmbeddings( model_name_or_path=embedding_model_name, model_kwargs=embedding_model_kwargs, encode_kwargs=encode_kwargs, )
Reply via ReviewNB
| @@ -0,0 +1,662 @@ | |||
| { | |||
There was a problem hiding this comment.
Line #11. response = urlopen(req)
I haven't managed to load it on linux, but looks like on windows works well. Did you check it on Linux ?
Reply via ReviewNB
|
@sbalandi I have made the suggested changes, can you please have a look once? I did not get to check it on Linux. Please let me know. |
Hi @pkhara31 any feedback on this idea? Would you mind to consolidate this notebook into RAG notebook. Please let me know your concern. |
Yes, I think it should be OK. |
|
Any updates here? @sbalandi |
|
This PR will be closed in a week because of 2 weeks of no activity. |
|
Do we have any updates on this PR/next steps ? @sbalandi |
|
Hi, @pkhara31 thank you for the changes ! they look good for me, but further checks and review are transferred to @openvino-dev-samples , please, contact him for any questions |
|
hi @pkhara31 thanks for your update, and how about moving this notebook into [notebooks/llm-rag-langchain] instead of [notebooks/llm-rag-ov-langchain]. Then you can also merge the README into existing one. |
|
Hi @openvino-dev-samples , I am already aligned to move this notebook into llm-rag-langchain, but may I know if that is something you would help out? Thanks. |
Thanks. It generally looks good to me, but I would like to test it with my local environment if you could help to move it, then I can give you detailed feedback. |
Added a notebook (.ipynb) code to evaluate the effectiveness of the RAG by computing standard metrics to evaluate the RAG performance.
Update README.md with the steps to execute the RAG evaluation notebook.
| @@ -0,0 +1,1597 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,1597 @@ | |||
| { | |||
There was a problem hiding this comment.
Line #31. "openvino-genai==2025.4.0.0",
I think openvino-genai will not be used in this notebook ? Please help to remove the unused dependencies.
Reply via ReviewNB
| @@ -0,0 +1,1597 @@ | |||
| { | |||
There was a problem hiding this comment.
Line #9. device="GPU" # Specify GPU for inference, can also be "CPU"
Please let user to select the device. Otherwise this notebook will be unavailable for platform without GPU. You can refer copy this part from other notebooks.
Reply via ReviewNB
| @@ -0,0 +1,1597 @@ | |||
| { | |||
There was a problem hiding this comment.
Line #31. model_id = "meta-llama/Llama-2-7b-hf"
As Llama2 will require an agreement from Meta for model downloading, there would extra step for users. Would you mind to change to another model, e.g Phi4 Qwen3 ?
Reply via ReviewNB
| @@ -0,0 +1,1597 @@ | |||
| { | |||
There was a problem hiding this comment.
Line #303. device="GPU", # or "GPU" for Intel integrated graphics
Please let user to select device
Reply via ReviewNB
| "\n", | ||
| "pip_install(\"--pre\", \"-U\", \"openvino==2025.4.0\", \"--extra-index-url\", \"https://storage.openvinotoolkit.org/simple/wheels/nightly\")\n", | ||
| "pip_install(\"--pre\", \"-U\", \"openvino-tokenizers==2025.4.0.0\", \"--extra-index-url\", \"https://storage.openvinotoolkit.org/simple/wheels/nightly\")\n", | ||
| "pip_install(\n", |
There was a problem hiding this comment.
| "pip_install(\n", | |
| "pip_install(\n", | |
| " \"--q\",\n", |
There was a problem hiding this comment.
remove the long output
| "\n", | ||
| "#Usage Example:\n", | ||
| "loader = LocalDocumentLoader(directory_path=\"content\")\n", | ||
| "docs = loader.load()\n", |
There was a problem hiding this comment.
I got this error when running this line of code, could you help to check ?
There was a problem hiding this comment.
The user needs to replace the "content" directory with their local directory name.
There was a problem hiding this comment.
Please set a default folder in this case, and do not let user manually create it
|
@openvino-dev-samples Can you please take a look again, I have addressed the above feedback. |
3 participants
This toolkit enables developers to build, evaluate, and optimize Retrieval-Augmented Generation (RAG) applications with comprehensive quality metrics including accuracy, bias detection, and perplexity analysis plus a racial-bias indicator. This uses RAG pipeline optimized with Intel OpenVINO for enhanced performance on CPU, GPU, and NPU. The pipeline leverages:
Optimum-Intel’s OVModelForCausalLM with the OpenVINO backend for efficient inference.
LangChain for orchestration of document loading, chunking, embedding, retrieval, reranking, and generation.
Goal: Provide a portable notebook-driven workflow for rapid experimentation, model comparison, and validation of RAG systems on custom/private corpora.