How to do streaming with RetrievalQA and huggingface model - Falcon 7b

[NajiAboo]

Hi ,

I am using falcon 7b instruct as my llm and I am using RetrievalQA to query against the document. But its taking a minute to return the data. I am trying to stream the result, But it not working as i expected. I saw that its working fine with OpenAIchat.
Please help me to figure out this problem.

Thanks.

[NajiAboo]

Below is my code. Please help me with it. Streaming is not working in this code.
Using chroma db here.

model = "tiiuae/falcon-7b-instruct"
tokenizer = AutoTokenizer.from_pretrained(model)
pipe = pipeline(
task="text-generation",
model=model,
tokenizer=tokenizer,
pad_token_id = tokenizer.eos_token_id,
top_p=0.95,
repetition_penalty = 1.2,
trust_remote_code=True,
max_new_tokens=100,
model_kwargs={
"device_map": "auto",
"load_in_8bit": False,
"max_length": 512,
"temperature": 0.01,
},

    )

llm = HuggingFacePipeline(pipeline=pipe)
streaming_llm = HuggingFacePipeline(pipeline=pipe, callbacks=[StreamingStdOutCallbackHandler()],)

question_generator = LLMChain(llm=llm, prompt=CONDENSE_QUESTION_PROMPT)
doc_chain = load_qa_chain(streaming_llm, chain_type="stuff", prompt=QA_PROMPT)

qa = ConversationalRetrievalChain(
retriever=retriever,
combine_docs_chain=doc_chain,
question_generator=question_generator,
callbacks=[StreamingStdOutCallbackHandler()]
)

chat_history = []
query = "here will be the question?"
print(qa({"question": query, "chat_history": chat_history})["answer"])