[BUG] Streaming does not work with Spring AI and Azure OpenAI

[bruno-oliveira]

Describe the bug
When streaming a completion, the results are still all aggregated and arrive sequentially instead of "streamed".

Exception or Stack Trace
No stacktrace

To Reproduce
Make a streaming call with Spring AI version 1.0.0 and observe the nature of the call is wrong, as the chunks are "blocked" somewhere deep in the Azure SDK code.

Code Snippet

 @ServiceMethod(returns = ReturnType.COLLECTION)
    public IterableStream<ChatCompletions> getChatCompletionsStream(String deploymentOrModelName,
        ChatCompletionsOptions chatCompletionsOptions) {
        chatCompletionsOptions.setStream(true);
        RequestOptions requestOptions = new RequestOptions();
        Flux<ByteBuffer> responseStream = getChatCompletionsWithResponse(deploymentOrModelName,
            BinaryData.fromObject(chatCompletionsOptions), requestOptions).getValue().toFluxByteBuffer();
        OpenAIServerSentEvents<ChatCompletions> chatCompletionsStream
            = new OpenAIServerSentEvents<>(responseStream, ChatCompletions.class);
        return new IterableStream<>(chatCompletionsStream.getEvents());
    }

Class: OpenAIClient.java

Expected behavior
The response needs to be streamed instead of arriving all at once

Setup (please complete the following information):

• OS: MacOS
• IDE: IntelliJ IDEA 2024.1.2 (Ultimate Edition)
• Library/Libraries: azure-ai-openai-1.0.0-beta.8.jar
• Java version: 21
• App Server/Environment: Standard Springboot setup
• Frameworks: Spring Boot

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [ X ] Bug Description Added
• [ X ] Repro Steps Added
• [ X ] Setup information Added

[ventaglio]

Same issue here, even using the last available version (1.0.0-beta.9) inside a simple Java class.

[bruno-oliveira]

Thanks for confirming this additionally as well! I initially thought it was an issue with the Spring AI project but after careful debugging realized that it was nested deeper into the SDK code, hopefully this can be picked up soon-ish.

PS: I did try to checkout the code myself and adapt the method from inside my fork of the Spring AI project but I got too lost in it and I couldn't devise a possible solution avenue

[nazarenodefrancescomaize]

I have the same problem but I think is a matter of deploy configuration.
Are you using personalized content filter without asynchronous streaming mode?
In which region is your deployment?

[bruno-oliveira]

@nazarenodefrancescomaize No, essentially I'm just using OpenAI API as it comes out of the box, nothing else is configured anywhere. Essentially, I use a test prompt for this:

Give me a recipe from Portugal

Then I need to see the answer being streamed if the mode is set to stream, and not streamed if not.
I have no filters anywhere, region is within EU.

[nazarenodefrancescomaize]

@nazarenodefrancescomaize No, essentially I'm just using OpenAI API as it comes out of the box, nothing else is configured anywhere. Essentially, I use a test prompt for this:

Give me a recipe from Portugal

Then I need to see the answer being streamed if the mode is set to stream, and not streamed if not. I have no filters anywhere, region is within EU.

Ok thanks. So I think it is a different problem, beacuase we solved our problem and was related to a custom content filter applied on that Azure deployment without the Asynchrounous filtering enabled. In default mode in fact the filter waits the generation completion, even in streaming mode.

[ifangng]

I also encountered the same problem

[bruno-oliveira]

bump

[timostark]

I can also confirm that issue in combination with springai. It is also not related to my GPT model / azure deployment. Using the npm package (@azure/openai) the streaming works without issues.