Chat response stream hangup

[thingersoft]

What happened?

Chat response stream sometimes gets stuck logging multiple "Unexpected non-whitespace character after JSON ..." messages in console.
Most often it happens with lenghty responses and structured content.

The problem resides in /packages/data-provider/src/sse.js

The code listens to the progress event of XMLHttpRequest to extract the chunks from the response and produce a custom "message" event for each json chunk.

/client/src/hooks/SSE/useSSE.ts defines a listener for the "message" event that crashes as it tries to parse the (expected) json from the "message" event data.

When the first error occurs console log shows that the "message" event data field contains two concatenated json chunks rather than a single one.
After that the data field keeps growing, potentially leading to browser crashes (probably related to #4417 also).

Don't know why the client is listening on the progress event of XMLHttpRequest rather than using the native EventSource API, doing a substring on the whole response body at each step isn't great from a performance point of view.
Also the whole event dispatching logic looks pretty brittle to me, concatenating strings to get the "registered" listener to invoke.

@danny-avila
I would like to produce a PR but rather than simply fixing the bug I would replace the client side streaming logic to leverage the EventSource API.

Steps to Reproduce

1. create a chat
2. make a request wich implies a lengthy, structured response

What browsers are you seeing the problem on?

No response

Relevant log output

No response

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

Hi @thingersoft

I moved this to discussions because I can't reproduce it, and I'm using the app daily. Would need more information like endpoint/provider/model and other relevant chat settings used to help reproduce.

The SyntaxError is to be expected when there's a malformed stream chunk from the API.

The SSE code is by Maxime Petazzoni and used in place of EventSource for these reasons:

sse.js is a flexible EventSource replacement for JavaScript designed to consume Server-Sent Events (SSE) streams with more control and options than the standard EventSource. The main limitations of EventSource are that it only supports no-payload GET requests, and does not support specifying additional custom headers to the HTTP request.

This package is designed to provide a usable replacement to EventSource that makes all of this possible: SSE. It is a fully compatible EventSource polyfill so you should be able to do this if you want/need to

I have not updated it since it was first implemented (at the very start of the project). At the time, there was no other way to initiate server-sent events via POST request that I knew how via the native EventSource method.

Maybe we can do that if some of these issues are then resolved, and if not, they can be addressed here: https://github.com/mpetazzoni/sse.js/

What I have in https://github.com/danny-avila/LibreChat/blob/main/packages/data-provider/src/sse.js
is essentially just copied from: https://github.com/mpetazzoni/sse.js/blob/main/lib/sse.js

I forget why I chose to do it this way instead of including it as a package.

[danny-avila]

More info: https://stackoverflow.com/questions/34261928/can-server-sent-events-sse-with-eventsource-pass-parameter-by-post

[thingersoft]

Hello @danny-avila

I think EventSource API doesn's support POST on purpose since clients shouldn't expect a stream of events in response to server mutations.

Using the right http verbs has security implications and the reason behind choosing POST, PUT PATCH, etc. rather than GET shouldn't be "my query string is too long".

If for some reason one absolutely needs to use POST for a query than there's probably something wrong with the API design in first place.

Having said that, in the case of LibreChat I guess that sending chat messages actually causes server mutations, so using POST is the right choice indeed.

Imho the sending message request should be decoupled from the progress/stream request, the first one being a POST and the second one a GET.

Anyway looking at the sse.js code currently in use by LibreChat I see at least two points of failure.

1. if the text returned by the llm contains a sequence of \n\n or similar the chunks splitting logic doesn't work
2. if for some reason the message listener throws an error the chunk variable isn't blanked causing the event data field to keep growing till the end of the stream

By a quick look at the mpetazzoni repository these aspects haven't been fixed yet so I'll open an issue there.

Anyway I'm still trying to figure out how to reproduce the scenario of two chunks collapsed into one, I'll get back as soon as I can.

[thingersoft]

1. if the text returned by the llm contains a sequence of \n\n or similar the chunks splitting logic doesn't work

I see this isn't a problem since the server is doing a JSON.stringify before writing to the response so \n sequences are escaped.

2. if for some reason the message listener throws an error the chunk variable isn't blanked causing the event data field to keep growing till the end of the stream

This could be the culprit, but still investigating.

[thingersoft]

@danny-avila
Finally got it, the root double chunk issue is caused by this:
mpetazzoni/sse.js#36
When the code has been embedded into LibreChat the issue wasn't fixed yet.

It's a pretty edge case, anyway it sounds strange that you you never experienced it, may it's due to different network characteristics?

The cleanest solution would be to replace the embedded version of sse.js with the external up-to-date dependency.
But I see that you you applied some customization to resume the streaming when auth token expires so that should be refactored too.

I'll try to come up with a PR asap, in the meantime I think the discussion can be converted back into an issue.