[Community CTransformers] Ability to use stream and astream #19262
QuentinBriand
announced in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Checked
Feature request
The feature is quite simple. Actually the CTransformers class doesn't integrate the methods _stream and _astream, what it does actually is switching silently (it seems weird for me that there isn't a proper log on the base LLM class to warn for such switches) to the invoke method.
Motivation
For business purpose i would want to be able to load gguf files with CTransformers while still being able to stream the responses via Langserve automatically or via custom SSE's and not get only a huge block at the end.
Proposal (If applicable)
I already added both of the functions into my own scripts but i don't know if i should post the code embed here as is.
Beta Was this translation helpful? Give feedback.
All reactions