AI is streaming incorrect grammar and wrongly formed sentences when using ConversationalRetrievalChain.

[eRuaro]

I'm trying to implement HTTPS streaming in the ConversationalRetrievalChain via FastAPI but am stumbling into some issues. Mainly the streaming text is full of bad writing.

"TheForFocusedThis F-mode isocused thinking example the, is Mode focused— essential mode forA  studying T of math  thinking andight.\n Pin scienceThe.ball close It Machine bumpers of involves a direct the approach focused to mode solving mean problems that you can using rational, more sequential you can use the focused mode to multiply numbers—if you already know how to multiply, that is. easily, think a analytical precise approaches If you. thought. The Basically’re studying a language focused,, mode you the might is associated use the focused mode focused is used to with the concentrating mode abilities to concentrate of on the become more brain something that’ fluent’s s with already pre the tightly Spanish verb connected in conjugation your you mindfrontal cortex,, located right often because behind learned your last you week forehead.. are familiar and If you’re a swim comfortable withmer the, underlying you concepts might. use If the you look focused closely mode at the to upper analyze part your of the breast focusedstroke-mode thought pattern, you as you practice staying low to allow more’ll see energy a wider, to “ gowell into your-t forward motionro.dden When” you part focus of on the something line,. the That consciously broader attentive path pre showsfront howal the cortex automatically sends out signals along focused neural-mode thought is following pathways along. a These signals route you link different’ve already areas of your brain practiced or experienced. related to what you’re thinking about. This process is a little like an octopus that sends its tentacles to different areas of its surroundings to fiddle with whatever it’s working on. The octopus has only so many tentacles to make connections, just as your working memory has only so many things it can hold at once."

Here's an example of a streamed response ^.

I already had an issue on streaming gibberish before: #9592 (comment) but I've seemed to have resolved that already - in the sense that I'm no longer streaming the model thought process.

However, the issue now is the wrong grammar being sent. Here's the code:

async def send_message(query: str, history: list):
    api_key = os.environ.get("OPENAI_API_KEY")
    embedding = OpenAIEmbeddings()
    collection_name = "name-of-collection"
    connection_result = check_if_connection_exists(collection_name)

    if connection_result == False:
        raise Exception("Collection does not exist")

    store = PGVector(
        connection_string=connection_string,
        embedding_function=embedding,
        collection_name=collection_name,
        distance_strategy=DistanceStrategy.COSINE
    )

    chat_memory = ChatMemory.from_pg_conversation(
        response_json=history)

    retriever = store.as_retriever()
    callback_handler = AsyncIteratorCallbackHandler()
    callback_manager = AsyncCallbackManager([callback_handler])

    llm = ChatOpenAI(
        streaming=False,
        openai_api_key=api_key,
        temperature=0,
        max_tokens=1500,
        verbose=True,
    )
    streaming_llm = ChatOpenAI(
        streaming=True,
        openai_api_key=api_key,
        temperature=0,
        max_tokens=1500,
        verbose=True,
        callback_manager=callback_manager,
    )

    question_generator = LLMChain(llm=llm, prompt=CONDENSE_QUESTION_PROMPT, verbose=False)
    doc_chain = load_qa_with_sources_chain(
        llm=streaming_llm,
        chain_type="map_reduce",
        verbose=False,
    )

    chain = ConversationalRetrievalChain(
        retriever=retriever,
        question_generator=question_generator,
        combine_docs_chain=doc_chain,
    )

    run = asyncio.create_task(
        chain.acall(
            {"question": query, "chat_history": chat_memory.history})
        )

       # Yield the tokens as they come in
    try: 
        async for token in callback_handler.aiter():
            yield token
    except Exception as e:
        print(f"exception: {e}")
        # Await the chain run
    result = await run

    print(f"{result['answer']}")


@router.post("/document/qa/demo/stream", tags=["document"])
async def qa_demo_stream(document: DemoDocument):
    try:
        generator = send_message(document.query, document.history)

        response = StreamingResponse(
            generator,
            media_type="text/event-stream",
            status_code=200
        )
        
        return response
    except Exception as e:
        return HTTPException(status_code=404, detail="There was an error in streaming your response.")
    except Exception as e:
        raise HTTPException(status_code=404, detail=str(e))

However, the streamed response is different from result['answer']. How can I stream the texts in result['answer']? Or at least have the streamed response similar to the one returned in result['answer']?

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out and providing a detailed explanation of your issue. It seems like the problem you're facing is related to the quality of the responses generated by the ConversationalRetrievalChain class.

The ConversationalRetrievalChain class in LangChain is responsible for generating the response to a given question. The response is generated in the _call and _acall methods of the BaseConversationalRetrievalChain class, which is the parent class of ConversationalRetrievalChain.

The response is generated by the combine_docs_chain object, which is an instance of a class that inherits from BaseCombineDocumentsChain. This object is responsible for combining the retrieved documents into a final response. The quality of the response depends on the implementation of the run or arun method of the combine_docs_chain object.

If you're not satisfied with the quality of the responses, you might want to modify the run or arun method of the combine_docs_chain object. You could use a more sophisticated language model or a different strategy for combining the documents.

Here is the part of the code where the response is generated:

answer = self.combine_docs_chain.run(
    input_documents=docs, callbacks=_run_manager.get_child(), **new_inputs
)

And here is the part of the code where the combine_docs_chain object is defined:

combine_docs_chain = StuffDocumentsChain(...)

You could replace StuffDocumentsChain with a different class that inherits from BaseCombineDocumentsChain and provides a better implementation of the run or arun method.

As for the difference between the streamed response and result['answer'], it's possible that the streamed response is showing the intermediate outputs of the ConversationalRetrievalChain while it's running, whereas result['answer'] is the final output after the chain has finished running. If you want to stream the final output, you might need to modify the AsyncIteratorCallbackHandler class to yield the final output instead of the intermediate outputs.

I hope this helps! If you have any further questions, please don't hesitate to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/chains/conversational_retrieval/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.