Why do I always get "finish_reason":"length" when calling GPT4All-API chat/completions endpoint?

[ghevge]

I've set up a GPT4All-API container and loaded the openhermes-2.5-mistral-7b.Q8_0.gguf model. All good here but when I try to send a chat completion request using curl, I always get a no message response, with "finish_reason":"length" set. From what I was reading
this flag is usually set when there is a problem with the max tokens limit. So I've manually increased that limit, but still I am getting the
same behavior.

Any ideas?

In GPT4ALL UI for the same input and the same model I get: "After the letter "C" in the English alphabet, we have the letter "D"."

Thanks

Request:

 curl http://localhost:4891/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "openhermes-2.5-mistral-7b.Q8_0.gguf",
 "response_format": { "type": "json_object" },
 "max_tokens": 500,
 "temperature": 0.28,
 "top_p": 0.95,
 "n": 1,
 "echo": true,
 "stream": false,
 "messages": [
        {"role": "system", "content": "You are a helpful assistant designed to output JSON."},
        {"role": "user", "content": "What comes after C?"}] 
 }'

Response:

{"id":"237cdee0-96ea-49a2-a7ad-fcd1daf1e1bd","object":"text_completion","created":1706562736,"model":"openhermes-2.5-mistral-7b.Q8_0.gguf","choices":[{"message":{"role":"system","content":"Echo: What comes after C?"},"index":0,"logprobs":-1.0,"finish_reason":"length"}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}

[ghevge]

Is GPT4All-API implementation completed? From what I can see here:

gpt4all/gpt4all-api/gpt4all_api/app/api_v1/routes/chat.py

Line 59 in f549d5a

response_choice = ChatCompletionChoice(

The response of /chat/completions endpoint is hardcoded ....

[ghevge]

Why exactly was this issue closed? May I get some clear explanations at least ? Thanks!

[cebtenzzre]

I closed this discussion because it seems like you already figured out that it should have been an issue instead, and opened one.