Strange buffering on client when action returns a stream

[acavestro-at-halueio]

Hi people, I want to show you an interesting issue I’m having.

High level, I've an action which returns a stream. It is created originally from openai-node but I'm not sure how it is relevant. The client side which made the request using fetch parse the streamed response.

Speaking of code, the action code is this:

const stream = openai.beta.threads.runs
    .stream(threadId, {
      assistant_id: assistantId,
      include: ['step_details.tool_calls[*].file_search.results[*].content'],
    })
    .on('messageDelta', (message) => {
      console.log('messageDelta', JSON.stringify(message));
    })
   .toReadableStream();

  return new Response(stream, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      Connection: 'keep-alive',
      Thread: threadId,
    },
  });

The client part is

const response = await fetch(localizedChatAPIURL, {
    method: 'POST',
    body: thePayload,
  });
const reader = response.body
          .pipeThrough(new TextDecoderStream())
          .pipeThrough(new TextLineDecoderStream())
          .pipeThrough<AssistantStreamEvent>(new JSONDecoderStream())
          .getReader();

The strange issue is that messages appears to be different from server to client.

The output of that console.log on messageDelta, server side is:

messageDelta {"content":[{"index":0,"type":"text","text":{"value":"E","annotations":[]}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":"ddy"}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":" Mer"}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":"ck"}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":"x"}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":","}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":" noto"}}]}
messageDelta {"content":[{"index":0,"type":"text","text":{"value":" come"}}]}

While the network inspector on the browser side logs these messages:

{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":"Eddy Merckx, noto","annotations":[]}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":"ddy"}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":" Mer"}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":"ck"}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":"x"}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":","}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":" noto"}}]}}}
{"event":"thread.message.delta","data":{"id":"msg_lb6hMW0y8YccR8wnkPtfkScL","object":"thread.message.delta","delta":{"content":[{"index":0,"type":"text","text":{"value":" come"}}]}}}

The issue here is that this inconsistency is causing malformations of the first sentences. The example is in italian, I’m sorry, but the beginning of that sentence should be “Eddy Merkx , noto come”. Instead, the client part prints “Eddy Merckx, notoddy Merckx, noto come”.

I noticed the first message on the client side is longer than the first message on the server side so I’m suspecting some buffering happening somewhere.

Do you people have already encounter this issue? Do you have feedbacks about how to handle this and avoid the malformations?

The Remix version here is 2.5.1. I had a look at the issues and the PRs to find if maybe it is a fixed issue on a later version. In case, do you have suggestions on how I can workaround this without an upgrade?

Thank you

[acavestro-at-halueio]

Closing, it's related to openai-node.