Can someone point me to the streaming files.

[d1pankarmedhi]

I want to learn how streaming is implemented in haystack but I'm unable to local where exactly the code is. Would appreciate some help on locating that.
Thanks in advance :)

[sandangel]

I think streaming has not been implemented yet. The only place it's done is in openai chat streaming callback. But I believe they are working on a more general version of streaming that can be used in haystack v2 for everything. @anakin87 can you leave a comment?

[sandangel]

@anakin87 how about AsyncGenerator, will it be considered as well?

[anakin87]

IDK the details but we should provide similar functionality...

[sandangel]

The AsyncGenerator will yield token from LLM as it's streaming from LLM output. That way we can use async for to loop through tokens and update the UI like you can see the code snippet:

 async for chunk in response:
        await response_message.stream_token(token=chunk["replies"]) # chunk["replies"] can contain partial LLM outputs instead of the complete final answer.

[d1pankarmedhi]

Doc https://docs.haystack.deepset.ai/docs/prompt_node#with-streaming

I am trying to understand how model_kwargs={"stream":True} is implemented.

from haystack.nodes.prompt import PromptNode

pn = PromptNode("gpt-3.5-turbo", api_key="<api_key_goes_here>", model_kwargs={"stream":True})
prompt = "What are the three most interesting things about Berlin? Be elaborate and use numbered list"
pn(prompt)

[Phlasse]

Are there any news, about when this feature is planned? Looking forward to it very much :)