Error handling when completion_tokens is 0 and missing finish_reason for choice 0

[jameslamine]

What happened?

I'm running into an intermittent issue where the endpoint incorrectly returns 0 tokens and HTTP status 200. It responds with completion_tokens: 0 and content: "".

When this happens, LibreChat shows a blank reply with that blank content:

I think we could improve the user experience for how LibreChat handles this error.

We can see from the LibreChat logs that it (correctly) doesn't consider the response to be valid:

  "level":          "error",
  "message":        "[OpenAIClient] Known OpenAI error: Error: missing finish_reason for choice 0",
  "stack":          "Error: Error: missing finish_reason for choice 0
    at OpenAIClient.chatCompletion (/app/api/app/clients/OpenAIClient.js:1410:15)
    at async OpenAIClient.sendCompletion (/app/api/app/clients/OpenAIClient.js:659:15)
    at async OpenAIClient.sendMessage...",

Ideally, the LibreChat UI should treat this as a failed response instead of showing the chat bot sending a blank response. That way it's obvious to the user that there was an error and they can click the retry button.

Here's the endpoint response itself:

{
  "status": 200,
  "headers": {
    "connection": "keep-alive",
    "content-type": "text/event-stream",
    "date": "Wed, 29 Jan 2025 19:33:40 GMT",
  },
  "body": {
    "object": "chat.completion.chunk",
    "created": 1738179223,
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": ""
        }
      }
    ],
    "model": "",
    "usage": {
      "completion_tokens": 0
    }
  },
  "responseTime": 72,
}

This might be related to #1352

Steps to Reproduce

You can reproduce this by mocking a response which looks like this:

{
  "status": 200,
  "headers": {
    "connection": "keep-alive",
    "content-type": "text/event-stream",
    "date": "Wed, 29 Jan 2025 19:33:40 GMT",
  },
  "body": {
    "object": "chat.completion.chunk",
    "created": 1738179223,
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": ""
        }
      }
    ],
    "model": "",
    "usage": {
      "completion_tokens": 0
    }
  },
  "responseTime": 72,
}

What browsers are you seeing the problem on?

Chrome

Relevant log output

"level":          "error",
  "message":        "[OpenAIClient] Known OpenAI error: Error: missing finish_reason for choice 0",
  "stack":          "Error: Error: missing finish_reason for choice 0
    at OpenAIClient.chatCompletion (/app/api/app/clients/OpenAIClient.js:1410:15)
    at async OpenAIClient.sendCompletion (/app/api/app/clients/OpenAIClient.js:659:15)
    at async OpenAIClient.sendMessage...",

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

This is an issue with your custom endpoint, it's not completely compatible with openai as the upstream SDK is strict about this.

https://github.com/openai/openai-node/blob/6f89573f9b334960195b074e17ad70df32329e8e/src/lib/ChatCompletionStream.ts#L618

[jameslamine]

Thanks! I do agree it's an issue with the custom endpoint. We're hitting AWS bedrock through PortKey. Unfortunately, we don't have control over the format of their responses.

I'm totally happy to patch it to handle this more gracefully. I have a clarifying question though.

The response isn't sending any tokens. I think there are two possible reasons:

Possibility 1: LibreChat is terminating the request before the server has a chance to stream tokens

Maybe the server does actually have tokens to send, but LibreChat is terminating the request early when it encounters the unexpected missing finish_reason for choice 0 data. This would mean that if I patch LibreChat to ignore that error, it will continue streaming the request and receive tokens.

Possibility 2: The endpoint is encountering an error and doesn't have any tokens to send

If this is the case, the fix is probably for me to patch LibreChat so that it's still treated as an error, but it more gracefully handles how the error is reported / displayed

Would really appreciate it if you can share your thoughts here!

[danny-avila]

Either case, I'm happy to review and test other endpoints with the patch.

Whatever works for you especially since I can't easily replicate what the PortKey setup. If it's just Anthropic Key + PortKey, I'll set it up when I can.

[jameslamine]

Makes sense. I was mostly wondering if you had an intuition as to which of those two possibilities is more likely.

It's AWS Bedrock + Portkey talking to Claude 3.5. The issue is it's intermittent (perhaps 10% of the time) so hard to reproduce. I'll try both approaches and see.

[jameslamine]

Ok, I think there are two fixes we should do:

1. Improve error handling of known openAI errors with empty intermediate response. PR here: 🔧 fix: handle known OpenAI errors with empty intermediate reply #5562

2. Add finish_reason: null to any chunk that's missing finish_reason

From PortKey customer support:

For Bedrock, we map the finish reason equivalent in the response. But we are not setting it as null if it’s not present. It should still be present for the last chunk. We will add it in the next build to make it fully aligned OpenAI spec. Bedrock only sends it for last chunk so all other chunks are getting mapped with undefined finish reason instead of null

So it does seem like AWS Bedrock is out of spec. PortKey will open a PR to patch it on their end. But for compatibility with Bedrock directly, we should probably implement my proposal above to add finish_reason: null to any chunk missing it.

[jameslamine]

Pr for the second change is here: #5563

[mkagit]

Are you still seeing missing finish_reason for choice 0 from Portkey?