How to properly implement streaming output through WebAPI endpoint?

[piotrek-k]

I'm trying to create intermediary microservice that is calling GPT API using this library, gets output and re-streams it to web frontend.

Here is how I'm trying to make it:

[HttpGet]
public async IAsyncEnumerable<string> StreamAnswer(string message)
{   
    await foreach (var part in _openAiManager.StreamAnswer(message))
    {
        yield return part;
    }
}

StreamAnswer looks like this:

public async IAsyncEnumerable<string> StreamAnswer(string userInputText)
{
    var completionResult = _service.ChatCompletion.CreateCompletionAsStream(new ChatCompletionCreateRequest
    {
        Messages = new List<ChatMessage>
        {
            new(StaticValues.ChatMessageRoles.System, "You are a helpful assistant."),
            new(StaticValues.ChatMessageRoles.User, userInputText)
        },
        Model = Models.Gpt_3_5_Turbo,
        MaxTokens = 150//optional
    });

    await foreach (var completion in completionResult)
    {
        if (completion.Successful)
        {
            Console.Write(completion.Choices.First().Message.Content);

            yield return completion.Choices.First().Message.Content;
        }
        else
        {
            if (completion.Error == null)
            {
                throw new Exception("Unknown Error");
            }

            Console.WriteLine($"{completion.Error.Code}: {completion.Error.Message}");
        }
    }
}

For now, my endpoint returns a proper message generated by GPT but it does not stream it. I mean: my endpoint waits until GPT returns entire message and then returns all of the text instead of displaying words as they arrive.

As you can see, I also print chunks of message using Console.Write and I can see in Console window that indeed answers arrive word by word. But for some reason webapi endpoint then buffers it and returns text when GPT finishes streaming.

[DEAN-Cherry]

I got this problem as well, but I found that if u pass more value just than completion.Choices.First().Message.Content, it would work well.
For example you can pass completion.ToJson(); in yield return or just something else. I don't know the reason why in this situation that the message only returns when GPT finishes streaming.