What's the intended usage pattern for model.stream()?

[nchammas]

I don't understand how to use model.stream(), and the behavior I'm seeing for it is just weird.

Here's the test script I'm running using guidance b278873:

from guidance import models, gen, user, assistant
from guidance.chat import Phi3MiniChatTemplate


if __name__ == "__main__":
    model_stream = (
        models.Transformers(
            "microsoft/Phi-3.5-mini-instruct",
            chat_template=Phi3MiniChatTemplate,
            enable_monitoring=False,
            echo=False,
            torch_dtype="auto",
            device_map="auto",
        )
        .stream()
    )
    with user():
        model_stream += "Hello. How are you?\n"
    with assistant():
        model_stream += gen(name="response", stop="\n")

    print("response" in model_stream)
    print("response" in model_stream.model)

    for part in model_stream:
        print(part)

This is the output I get:

$ python test-stream.py 
Loading checkpoint shards: 100%|████████| 2/2 [00:05<00:00,  2.88s/it]
False
False
Hello. How are you?

Hello. How are you?

Hello. How are you?

This leads me to the following questions:

1. How is someone supposed to capture named model output when using streams? From what I can see, response is not available in either the ModelStream object or in the underlying Model object.
2. Why is the user prompt printed three times? And where is the model's response?

Additionally, if I make the following change to my test script and rerun it:

    with user():
        model_stream += "Hello. How are you?"  # remove newline at the end

I get completely different output:

$ python test-stream.py 
Loading checkpoint shards: 100%|████████| 2/2 [00:06<00:00,  3.23s/it]
False
False
Hello. How are you
Hello. How are you?
Hello. How are you? I
Hello. How are you? I'
Hello. How are you? I'm
Hello. How are you? I'm doing
Hello. How are you? I'm doing well
Hello. How are you? I'm doing well,
Hello. How are you? I'm doing well, thank
Hello. How are you? I'm doing well, thank you
Hello. How are you? I'm doing well, thank you for
Hello. How are you? I'm doing well, thank you for asking
Hello. How are you? I'm doing well, thank you for asking.
Hello. How are you? I'm doing well, thank you for asking. I
Hello. How are you? I'm doing well, thank you for asking. I'
<hundreds of lines of output snipped>

What's going on? Am I completely misusing this API, or is there a bug somewhere?

[hudson-ai]

@nchammas the model stream interface is kind of a leftover from another time -- it's semantics need a full review and possible rewrite, and we need to document this all.

Current behavior (afaik):

1. Lazy execution, only actually running the model when the stream is iterated over (as opposed to the "eager" execution you're probably used to with normal model objects)
   • Lazy execution actually doesn't work with role blocks at the present moment, so you're seeing some weird interaction happening as a result of adding a gen with stop='\n' directly after a trailing newline.
2. Iterating over the stream yields model/lm objects at intermediate stages of execution.

We don't really have a concept of a "chunk" or "delta" at the moment, and maybe that needs to change. Open to suggestions if you have an idea of what streams should actually do :)

[nchammas]

• Lazy execution actually doesn't work with role blocks at the present moment, so you're seeing some weird interaction happening as a result of adding a gen with stop='\n' directly after a trailing newline.

Just FYI I'm seeing similar behavior (with and without the trailing newline) even without the role blocks.

We don't really have a concept of a "chunk" or "delta" at the moment, and maybe that needs to change. Open to suggestions if you have an idea of what streams should actually do :)

I am tinkering with a potential API to layer on top of Guidance for my own purposes, but it would be great if ModelStream could be refactored to work something like this:

@guidance
def respond(model, question):
    with user():
        model += question
        yield model
    with assistant():
        model += "Response: "
        model += gen(stop="\n", name="response")
        yield model


def stream_response(model_stream):
    for part in model_stream:
        print(part, end="")
    print(model_stream["response"])


if __name__ == "__main__":
    model_stream = model.stream()
    model_stream += respond("How are you?")
    stream_response(model_stream)
    model_stream += respond("How's the weather?")
    stream_response(model_stream)

I haven't worked much with generators, so perhaps this idea doesn't work. But roughly, what I'm looking for is a ModelStream API that:

1. Works more or less like the usual Model API with support for +, gen, and all the other usual guidance functions.
2. yields a ModelStream after every generated token.
3. Tracks named generations.

That would mean that stream_response would print, word by word as the tokens are generated, both the user question and assistant response, followed by the captured named response by itself. We would see the two user questions followed by the model responses printed exactly once, since a generator "consumes" output it returns. The calls to += respond(...) would return instantly as none of the work happens there; they just build the plan of work to be executed when the user iterates over the stream.

This kind of API would make it easier to use Guidance in an interactive application where it's important to show the user that things are working without long delays for the model to finish generating a complete response. I am using Guidance with Streamlit, for example, and there is a dedicated write_stream() API that would work very well with this kind of refactored ModelStream.

I don't know how practical this would be to implement, but from a design perspective do you see any problems with this idea?

[nchammas]

for part in model_stream: may need to instead be something like for part in model_stream.state_iter(): to separate the stream object from the model's text state. I'd have to try implementing this to get clearer in my head about how this should work, but hopefully you get the gist of the idea for now.

[nchammas]

I am tinkering with a potential API to layer on top of Guidance for my own purposes

Just FYI, this is the technique I'm planning to use in my Streamlit application:

from datetime import datetime
from guidance import models, gen, user, assistant
from guidance.chat import Phi3MiniChatTemplate


class LazyAssistant:
    def __init__(self):
        self.model = models.Transformers(
            "microsoft/Phi-3.5-mini-instruct",
            chat_template=Phi3MiniChatTemplate,
            enable_monitoring=False,
            echo=False,
            torch_dtype="auto",
            device_map="auto",
        )

    def ask(self, question):
        with user():
            self.model += question
            yield "User: " + question + "\n"
        with assistant():
            yield "Response: "
            for _ in range(50):
                self.model += gen(max_tokens=1, name="response")
                # something goes wrong if I use `stop`
                # self.model += gen(stop="\n", name="response")
                yield self.model["response"]
            yield "\n"


if __name__ == "__main__":
    lazy_assistant = LazyAssistant()
    ask_start = datetime.now()
    response1 = lazy_assistant.ask("Write me a poem.")
    response2 = lazy_assistant.ask("What's the biggest thing that happened in 1989?")
    response3 = lazy_assistant.ask("How large is the Earth?")
    ask_end = datetime.now()
    print("ask time:", ask_end - ask_start)
    for response in [response1, response2, response3]:
        for part in response:
            print(part, end="")
        print()

It works as I would expect:

Screen.Recording.2025-04-23.at.4.09.32.PM.mov

I would guess that something would go very wrong if I jumped back and forth between the various response iterators, but since I don't need to support concurrent conversations this works fine. If I did want to support that use case, I'd perhaps need to "branch" the model's state for each question and merge those branches back after the question was fully answered (i.e. the response was fully iterated through).

I like this approach because it's very simple and doesn't get me into multithreading, multiprocessing, or asyncio. Haven't thought about whether it could be adapted to Guidance itself, but just wanted to share.