First message to custom endpoint does not retrieve final chunk

[krasimir100ev]

What happened?

I use a custom API integrated in libreChat. For using it I have configured a custom Endpoint. When I start a conversation with the my API, the first assistant prompt (the first response) hangs on the UI and I have to click the "stop" button to continue. All other responses are fine, only the first one hangs.

In the network tab I see that the first response does not have the final chunk received.

I debugged my API and I am sure that my API returns the last chunk (data:[DONE])

Steps to Reproduce

1. Activate librechat.yaml for custom endpoints
2. Setup a custom endpoint to an OpenAI Compatible API
3. Select the new API for a new conversation
4. Ask it a question (invoke /chat/completions endpoint)
5. Observe the UI when answer is streamed - it hangs waiting for other chunks where no other chunks are delivered from the custom API

What browsers are you seeing the problem on?

No response

Relevant log output

No response

Screenshots

After I click the stop button next to the prompt field, the conversation continues normally for all other messages:

Screenshots of the Network tab: the first message does not receive the last part with chunk marked as finel

After the first message, all others continue normally and receive their final = true message

Code of Conduct

• I agree to follow this project's Code of Conduct

[krasimir100ev]

I now saw that when the end of the first response occurs, there is an error in the librechat log file:

{"level":"error","message":"There was an uncaught error: stream ended without producing a ChatCompletionMessage with role=assistant","stack":"Error: stream ended without producing a ChatCompletionMessage with role=assistant\n at ChatCompletionStream._AbstractChatCompletionRunner_getFinalMessage (/app/api/node_modules/openai/lib/AbstractChatCompletionRunner.js:322:11)\n at ChatCompletionStream._emitFinal (/app/api/node_modules/openai/lib/AbstractChatCompletionRunner.js:103:152)\n at /app/api/node_modules/openai/lib/EventStream.js:52:22\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"} {"level":"error","message":"\n\nAn Uncaught OpenAIErrorerror may be due to your reverse-proxy setup or stream configuration, or a bug in theopenai` node package."}

On the next responses in the same conversation there is no such error.

This is the python code that returns the response into chunks:

`async def _resp_async_generator(text_resp: str):
# let's pretend every word is a token and return it over time
tokens = text_resp.split(" ")

for i, token in enumerate(tokens):
    chunk = {
        "id": i,
        "object": "chat.completion.chunk",
        "created": time.time(),
        "model": "jarvis-1.0",
        "choices": [{"delta": {"content": token + " "}}],
    }
    # Log the chunk
    print(f"Chunk {i}: {json.dumps(chunk)}")
    yield f"data: {json.dumps(chunk)}\n\n"
    await asyncio.sleep(0.1)
# Log the end
print("Stream completed: [DONE]")
yield "data:[DONE]"`

[linkedlist771]

I now saw that when the end of the first response occurs, there is an error in the librechat log file:

{"level":"error","message":"There was an uncaught error: stream ended without producing a ChatCompletionMessage with role=assistant","stack":"Error: stream ended without producing a ChatCompletionMessage with role=assistant\n at ChatCompletionStream._AbstractChatCompletionRunner_getFinalMessage (/app/api/node_modules/openai/lib/AbstractChatCompletionRunner.js:322:11)\n at ChatCompletionStream._emitFinal (/app/api/node_modules/openai/lib/AbstractChatCompletionRunner.js:103:152)\n at /app/api/node_modules/openai/lib/EventStream.js:52:22\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"} {"level":"error","message":"\n\nAn Uncaught OpenAIErrorerror may be due to your reverse-proxy setup or stream configuration, or a bug in theopenai` node package."}

On the next responses in the same conversation there is no such error.

This is the python code that returns the response into chunks:

`async def _resp_async_generator(text_resp: str): # let's pretend every word is a token and return it over time tokens = text_resp.split(" ")

for i, token in enumerate(tokens):
    chunk = {
        "id": i,
        "object": "chat.completion.chunk",
        "created": time.time(),
        "model": "jarvis-1.0",
        "choices": [{"delta": {"content": token + " "}}],
    }
    # Log the chunk
    print(f"Chunk {i}: {json.dumps(chunk)}")
    yield f"data: {json.dumps(chunk)}\n\n"
    await asyncio.sleep(0.1)
# Log the end
print("Stream completed: [DONE]")
yield "data:[DONE]"`

did you solve it? what cause its?

[danny-avila]

Try testing it with the openai-node package using the same version as librechat, it could make the issue more apparent in an encapsulated script.

[harrisonhxy]

It's a pity that I have the same problem, but I'm a computer novice and don't understand your method. Could you please tell me in detail how to solve the problem? thank you a lot!

[7ojo]

@harrisonhxy This is basically API problem and problem came on using/developing custom OpenAI API implementation. On custom API implementation we need to make sure that last message from API is that message above so OpenAI client can work with that.

[harrisonhxy]

@harrisonhxy This is basically API problem and problem came on using/developing custom OpenAI API implementation. On custom API implementation we need to make sure that last message from API is that message above so OpenAI client can work with that.

It is indeed an API issue.Since I'm using the reverse API and that reverse API proxy was not built by me, I don't know how to fix it.
The way I can barely use it at the moment is that I remove ["stream"] via the dropParams parameter.But the consequence of this is that all the output is not streaming, which affects my experience of using it.😣

[7ojo]

Yep, there is no easy fix on that situation or I don't know any 😞

[nimobeeren]

The error message is very confusing, as the issue apparently has nothing to do with the role. Just adding "index": 0 to the final message fixes the error.

[linkedlist771]

did you guys solve it?

[nimobeeren]

See @krasimir100ev's reply in the thread above.

[linkedlist771]

Ok, I have solved it. Thanks!

[frenzybiscuit]

What backend/middle are you using?

I cannot replicate this on Llamacpp, exllamav2 or Aphrodite.