Skip to content

Commit 18d3877

Browse files
authored
feat: optimize xinference stream (#989)
1 parent 53e83d8 commit 18d3877

File tree

1 file changed

+3
-3
lines changed

1 file changed

+3
-3
lines changed

api/core/third_party/langchain/llms/xinference_llm.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -108,12 +108,12 @@ def _stream_generate(
108108
Yields:
109109
A string token.
110110
"""
111-
if isinstance(model, RESTfulGenerateModelHandle):
112-
streaming_response = model.generate(
111+
if isinstance(model, (RESTfulChatModelHandle, RESTfulChatglmCppChatModelHandle)):
112+
streaming_response = model.chat(
113113
prompt=prompt, generate_config=generate_config
114114
)
115115
else:
116-
streaming_response = model.chat(
116+
streaming_response = model.generate(
117117
prompt=prompt, generate_config=generate_config
118118
)
119119

0 commit comments

Comments
 (0)