Skip to content

Conversation

svilupp
Copy link
Contributor

@svilupp svilupp commented Aug 10, 2025

Closes #65

This PR:

  • adds StreamCallbacks.jl dependency, replacing the existing streaming implementation
  • implements configure_callback! helper and extend create_chat to support custom providers and StreamCallbacks
  • Adds the explanation to docs, README and a dedicated example to showcase different streaming use cases

Other minor things:

  • replaced model to gpt-5-mini to be more current
  • re-formatted with the latest JuliaFormatter, CI was failing otherwise(that's why most files seem "changed")

Tests pass and examples execute as expected.

@svilupp
Copy link
Contributor Author

svilupp commented Aug 10, 2025

@roryl23 what do you think?

@roryl23
Copy link
Collaborator

roryl23 commented Aug 17, 2025

@svilupp I'll take a look! Sorry about the delay

@roryl23
Copy link
Collaborator

roryl23 commented Aug 23, 2025

I keep getting

chatcompletion - streaming: Error During Test at /home/rory/src/OpenAI.jl/test/chatcompletion.jl:24
  Got exception outside of a @test
  HTTP.RequestError:
  HTTP.Request:
  HTTP.Messages.Request:
  """
  POST /v1/chat/completions HTTP/1.1
  Authorization: ******
  Content-Type: text/event-stream
  Host: api.openai.com
  Accept: */*
  User-Agent: HTTP.jl/1.11.6
  Accept-Encoding: gzip
  Cookie: ******
  Transfer-Encoding: chunked
  
  [Message Body was streamed]"""Underlying error:
  AssertionError: Content-Type header should include the type text/event-stream.
  Received type: application/json; charset=utf-8
  Status code: 400
  Response headers:
   - Date: Sat, 23 Aug 2025 20:29:01 GMT
   - Content-Type: application/json; charset=utf-8
   - Content-Length: 167
   - Connection: keep-alive
   - vary: Origin
   - x-request-id: req_7cdabade8598440e95b75793d187049f
   - x-envoy-upstream-service-time: 4
   - cf-cache-status: DYNAMIC
   - Set-Cookie: __cf_bm=1H4OhS98wgFaOhcl0dEerdIImGlGCd8zBDc0i.Z31TU-1755980941-1.0.1.1-xs.XCwb6PGfdyaZK7oq1K6GbvAeGlCGwknrvyFhVAC9BAQuehLnxxc5VzsTbCy5RR8m5llsetSIfRao7tDHFDGyM8osxoyGfrI5GiC92wIU; path=/; expires=Sat, 23-Aug-25 20:59:01 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None
   - Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
   - X-Content-Type-Options: nosniff
   - Server: cloudflare
   - CF-RAY: 973d5c166b406c81-DFW
   - alt-svc: h3=":443"; ma=86400
  Response body: 
  Please check the model you are using and that you set `stream=true`.

even when I've added changes to override the Content-Type header, have you seen this while running tests?

@svilupp
Copy link
Contributor Author

svilupp commented Aug 24, 2025

I'll look at it tonight.

It basically says that the response from the server is not SSE/not parseable, so I rather fail.

It's usually driven by:

  • not setting stream=true in the api kwargs, so server isn't streaming
  • in the context of Streamcallbacks, it happens when user providers unconfigured callback (it doesn't know which backend and expects user to configure it fully). This should be avoided for normal users by automatically the configure_callback under the hood, which injects stream=true on users behalf.
  • The other reason is that the API refuses to stream and sends some error message -- maybe I should handle that more gracefully and capture that message better in the assertion! (think rate-limiting or unsupported params in the request)

I'm not sure why all tests passed fine for me and not for you though.

EDIT: Now I see that the error message explains that at the bottom -- I added that because it was a common error.

@svilupp
Copy link
Contributor Author

svilupp commented Aug 25, 2025

Thanks, but gpt-5-mini supports streaming without a problem ->
image

Is there something special about your API key / OpenAI org?
I tried to step through the examples/streamcallbacks.jl and it all works as expected.

For debugging, the callbacks have a parameter verbose=true (echoes all received data pre-processing), but it won't help in this case, because we're already printing all info for easier debugging of this special issue anyway-- the error you hit is this line:
https://github.com/svilupp/StreamCallbacks.jl/blob/d482e6a0e4290aff930caefb50419c24b4c85add/src/shared_methods.jl#L234
That prints all the headers and response.body by default, but it seems that the error you got had no response body, which is puzzling.

Can you perhaps try again if it was an OpenAI glitch?

@roryl23 roryl23 self-requested a review August 28, 2025 22:48
@roryl23 roryl23 merged commit fabf099 into JuliaML:main Aug 28, 2025
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Move streaming to StreamCallbacks.jl
3 participants