Is streaming results possible? (Not in a Jupyter notebook context)

[andysalerno]

Hi there,

I'm having a blast prototyping with this library. It's really excellent!

One thing I haven't been able to figure out- is it possible to enable async streaming of tokens from the model?

I've seen the tutorials and API that show stream=True, but this seems to only have an effect on Jupyter notebook hosted scenarios? That's how it seems, anyway.

I'm wondering, if I wanted to stream tokens in realtime to my own client, is there a supported way to do this?

In my own toy projects, I'm always showing token generation in realtime, since it's a much better user experience. But after switching to Guidance, I'm missing out on that. Everything else is 100x better with Guidance though :)

[slundberg]

I am working on exposing a nice API for that. Will post in discussion when it lands :)

The current plan to is do something like:

for p in program(stream=True):
    # do stuff with p, which is the partially executed program

and an equivalent async version. Any comments on the API are welcome. The upside of what is above is that p has a nice clean structure, the downside is that if you just want what is new (the next chunk) then you need to track the current position yourself. I think this is better though than defining a whole new "diff" format people then have to learn