Skip to content

Commit 24797e3

Browse files
chr-hertelchr-hertel
committed
feature #118 [Demo] Add an example with streaming (valtzu)
This PR was merged into the main branch. Discussion ---------- [Demo] Add an example with streaming | Q | A | ------------- | --- | Bug fix? | no | New feature? | no | Docs? | no | License | MIT I'm proposing to add an example which uses streaming via Turbo Streams / SSE since `symfony/ux-turbo` is already included, yet no demo app is using streaming. This demo does not use a custom Stimulus controller at all, but solely relies on vendor-provided `live` & `turbo_stream` controllers, making it ideal for those afraid of JS :) ```mermaid sequenceDiagram participant frontend participant backend participant platform frontend->>backend: Live action: submit with message activate backend backend-->>backend: Store the message<br>but do nothing else backend-->>frontend: Response with placeholder message element + <turbo-stream-source id="stream" src="/stream/assistant-reply"> deactivate backend frontend->>backend: turbo_stream_controller.js starts consuming /stream/assistant-reply as SSE source activate backend backend->>platform: Make platform call with stream: true activate platform platform-->>backend: first chunk received Note over backend,frontend: Clear out "Thinking..." after receiving the first chunk backend-->>frontend: data: <turbo-stream action="update" target="placeholder"></turbo-stream> backend-->>frontend: data: <turbo-stream action="append" target="placeholder">{{ chunk }}</turbo-stream> loop until end of stream platform-->>backend: nth chunk received backend-->>frontend: data: <turbo-stream action="append" target="placeholder">{{ chunk }}</turbo-stream> end deactivate platform backend-->>frontend: data: <turbo-stream action="remove" target="stream"></turbo-stream> frontend-->>backend: turbo_stream_controller.js closes the stream since the <turbo-stream-source> was removed deactivate backend ``` --- A couple of observations made while doing this: 1. `Symfony\AI\Agent\Chat` is not compatible at all with async/streaming flow. 2. As I recall, Message IDs were designed/added "for the frontend". If that's still true, then it'd nice if you could pass the message id from the outside, because it could be needed in the frontend to render the placeholder message ("thinking...") which happens before we have an instance of `AssistantMessage`. Or alternatively, `AssistantMessage` would need to be non-readonly (or have `withContent`) so that the message object could be created earlier and appended with content later. At the end, I didn't end up needing the IDs here since they didn't solve the "duplicate messages" (one from turbo, another from live component) problem I had at some point. Commits ------- 4ef719b [Demo] Add an example with streaming
2 parents d15c375 + 4ef719b commit 24797e3

File tree

12 files changed

+297
-7
lines changed

12 files changed

+297
-7
lines changed

‎demo/assets/app.js

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import 'bootstrap/dist/css/bootstrap.min.css';
33
import './styles/app.css';
44
import './styles/audio.css';
55
import './styles/blog.css';
6+
import './styles/stream.css';
67
import './styles/youtube.css';
78
import './styles/video.css';
89
import './styles/wikipedia.css';

‎demo/assets/styles/stream.css

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
.stream {
2+
body&, .card-img-top {
3+
background: #ececec;
4+
background: linear-gradient(0deg, #d26f15 0%, #ff9f4c 100%);
5+
}
6+
7+
&.chat {
8+
.user-message {
9+
background: #ffffff;
10+
}
11+
12+
.bot-message {
13+
background: #ffffff;
14+
15+
a {
16+
color: #3e2926;
17+
}
18+
}
19+
20+
.avatar {
21+
&.bot, &.user {
22+
outline: 1px solid #eaeaea;
23+
background: #eaeaea;
24+
}
25+
}
26+
27+
footer {
28+
&, & a {
29+
color: var(--bs-light);
30+
}
31+
}
32+
}
33+
}

‎demo/config/packages/ai.yaml

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,16 @@ ai:
1414
name: 'clock'
1515
description: 'Provides the current date and time.'
1616
method: 'now'
17+
stream:
18+
model:
19+
class: 'Symfony\AI\Platform\Bridge\OpenAI\GPT'
20+
name: !php/const Symfony\AI\Platform\Bridge\OpenAI\GPT::GPT_4O_MINI
21+
system_prompt: |
22+
You are an example chat application where messages from the LLM are streamed to the user using
23+
Server-Sent Events via `symfony/ux-turbo` / Turbo Streams. This example does not use any custom
24+
javascript and solely relies on the built-in `live` & `turbo_stream` Stimulus controllers.
25+
Whatever the user asks, tell them about the application & used technologies.
26+
tools: false
1727
youtube:
1828
model:
1929
class: 'Symfony\AI\Platform\Bridge\OpenAI\GPT'
@@ -62,4 +72,3 @@ services:
6272
Symfony\AI\Agent\Toolbox\Tool\Wikipedia: ~
6373
Symfony\AI\Agent\Toolbox\Tool\SimilaritySearch:
6474
$model: '@ai.indexer.default.model'
65-

‎demo/config/routes.yaml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,3 +38,14 @@ wikipedia:
3838
defaults:
3939
template: 'chat.html.twig'
4040
context: { chat: 'wikipedia' }
41+
42+
stream:
43+
path: '/stream'
44+
controller: 'Symfony\Bundle\FrameworkBundle\Controller\TemplateController'
45+
defaults:
46+
template: 'chat.html.twig'
47+
context: { chat: 'stream' }
48+
49+
stream_assistant_reply:
50+
path: '/stream/assistant-reply'
51+
controller: 'App\Stream\TwigComponent::streamContent'

‎demo/demo.png

8.07 KB
Loading

‎demo/src/Stream/Chat.php

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
<?php
2+
3+
/*
4+
* This file is part of the Symfony package.
5+
*
6+
* (c) Fabien Potencier <[email protected]>
7+
*
8+
* For the full copyright and license information, please view the LICENSE
9+
* file that was distributed with this source code.
10+
*/
11+
12+
namespace App\Stream;
13+
14+
use Symfony\AI\Agent\AgentInterface;
15+
use Symfony\AI\Platform\Message\AssistantMessage;
16+
use Symfony\AI\Platform\Message\Message;
17+
use Symfony\AI\Platform\Message\MessageBag;
18+
use Symfony\AI\Platform\Message\UserMessage;
19+
use Symfony\Component\DependencyInjection\Attribute\Autowire;
20+
use Symfony\Component\HttpFoundation\RequestStack;
21+
22+
final class Chat
23+
{
24+
private const SESSION_KEY = 'stream-chat';
25+
26+
public function __construct(
27+
private readonly RequestStack $requestStack,
28+
#[Autowire(service: 'ai.agent.stream')]
29+
private readonly AgentInterface $agent,
30+
) {
31+
}
32+
33+
public function loadMessages(): MessageBag
34+
{
35+
return $this->requestStack->getSession()->get(self::SESSION_KEY, new MessageBag());
36+
}
37+
38+
public function submitMessage(string $message): UserMessage
39+
{
40+
$messages = $this->loadMessages();
41+
42+
$userMessage = Message::ofUser($message);
43+
$messages->add($userMessage);
44+
45+
$this->saveMessages($messages);
46+
47+
return $userMessage;
48+
}
49+
50+
/**
51+
* @return \Generator<int, string, void, AssistantMessage>
52+
*/
53+
public function getAssistantResponse(MessageBag $messages): \Generator
54+
{
55+
$stream = $this->agent->call($messages, ['stream' => true])->getContent();
56+
\assert(is_iterable($stream));
57+
58+
$response = '';
59+
foreach ($stream as $chunk) {
60+
yield $chunk;
61+
$response .= $chunk;
62+
}
63+
64+
$assistantMessage = Message::ofAssistant($response);
65+
$messages->add($assistantMessage);
66+
$this->saveMessages($messages);
67+
68+
return $assistantMessage;
69+
}
70+
71+
public function reset(): void
72+
{
73+
$this->requestStack->getSession()->remove(self::SESSION_KEY);
74+
}
75+
76+
private function saveMessages(MessageBag $messages): void
77+
{
78+
$this->requestStack->getSession()->set(self::SESSION_KEY, $messages);
79+
}
80+
}

‎demo/src/Stream/TwigComponent.php

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
<?php
2+
3+
/*
4+
* This file is part of the Symfony package.
5+
*
6+
* (c) Fabien Potencier <[email protected]>
7+
*
8+
* For the full copyright and license information, please view the LICENSE
9+
* file that was distributed with this source code.
10+
*/
11+
12+
namespace App\Stream;
13+
14+
use Symfony\AI\Platform\Message\MessageInterface;
15+
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
16+
use Symfony\Component\HttpFoundation\EventStreamResponse;
17+
use Symfony\Component\HttpFoundation\Request;
18+
use Symfony\Component\HttpFoundation\ServerEvent;
19+
use Symfony\Component\HttpFoundation\Session\Session;
20+
use Symfony\Component\HttpFoundation\Session\Storage\MockArraySessionStorage;
21+
use Symfony\UX\LiveComponent\Attribute\AsLiveComponent;
22+
use Symfony\UX\LiveComponent\Attribute\LiveAction;
23+
use Symfony\UX\LiveComponent\Attribute\LiveProp;
24+
use Symfony\UX\LiveComponent\DefaultActionTrait;
25+
26+
#[AsLiveComponent('stream')]
27+
final class TwigComponent extends AbstractController
28+
{
29+
use DefaultActionTrait;
30+
31+
#[LiveProp(writable: true)]
32+
public ?string $message = null;
33+
public bool $stream = false;
34+
35+
public function __construct(
36+
private readonly Chat $chat,
37+
) {
38+
}
39+
40+
/**
41+
* @return MessageInterface[]
42+
*/
43+
public function getMessages(): array
44+
{
45+
return $this->chat->loadMessages()->withoutSystemMessage()->getMessages();
46+
}
47+
48+
#[LiveAction]
49+
public function submit(): void
50+
{
51+
if (!$this->message) {
52+
return;
53+
}
54+
55+
$this->chat->submitMessage($this->message);
56+
$this->message = null;
57+
$this->stream = true;
58+
}
59+
60+
#[LiveAction]
61+
public function reset(): void
62+
{
63+
$this->chat->reset();
64+
}
65+
66+
public function streamContent(Request $request): EventStreamResponse
67+
{
68+
$messages = $this->chat->loadMessages();
69+
70+
$actualSession = $request->getSession();
71+
72+
// Overriding session will prevent the framework calling save() on the actual session.
73+
// This fixes "Failed to start the session because headers have already been sent" error.
74+
$request->setSession(new Session(new MockArraySessionStorage()));
75+
76+
return new EventStreamResponse(function () use ($request, $actualSession, $messages) {
77+
$request->setSession($actualSession);
78+
$response = $this->chat->getAssistantResponse($messages);
79+
80+
$thinking = true;
81+
foreach ($response as $chunk) {
82+
// Remove "Thinking..." when we receive something
83+
if ($thinking && trim($chunk)) {
84+
$thinking = false;
85+
yield new ServerEvent(explode("\n", $this->renderBlockView('_stream.html.twig', 'start')));
86+
}
87+
88+
yield new ServerEvent(explode("\n", $this->renderBlockView('_stream.html.twig', 'partial', ['part' => $chunk])));
89+
}
90+
91+
yield new ServerEvent(explode("\n", $this->renderBlockView('_stream.html.twig', 'end', ['message' => $response->getReturn()])));
92+
});
93+
}
94+
}

‎demo/templates/_message.html.twig

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@
44
{{ _self.user(message.content) }}
55
{% endif %}
66

7-
{% macro bot(content, loading = false, latest = false) %}
8-
<div class="d-flex align-items-baseline mb-4">
7+
{% macro bot(content, loading = false, latest = false, contentId = null, messageId = null) %}
8+
<div class="d-flex align-items-baseline mb-4"{% if messageId %} id="{{ messageId }}"{% endif %}>
99
<div class="bot avatar rounded-3 shadow-sm">
1010
{{ ux_icon('fluent:bot-24-filled', { height: '45px', width: '45px' }) }}
1111
</div>
@@ -16,7 +16,7 @@
1616
<i>{{ content }}</i>
1717
</div>
1818
{% else %}
19-
<div class="bot-message d-inline-block p-2 px-3 m-1 border border-light-subtle shadow-sm">
19+
<div class="bot-message d-inline-block p-2 px-3 m-1 border border-light-subtle shadow-sm"{% if contentId %} id="{{ contentId }}"{% endif %}>
2020
{% if latest and app.request.xmlHttpRequest %}
2121
<span
2222
data-controller="symfony--ux-typed"

‎demo/templates/_stream.html.twig

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{% block start %}
2+
<twig:Turbo:Stream:Update target="#bot-message-streamed"/>
3+
{% endblock %}
4+
5+
{% block partial %}
6+
<twig:Turbo:Stream:Append target="#bot-message-streamed">{{ part }}</twig:Turbo:Stream:Append>
7+
{% endblock %}
8+
9+
{% block end %}
10+
<twig:Turbo:Stream:Remove target="#bot-message-stream"/>
11+
{% endblock %}

‎demo/templates/base.html.twig

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
<strong>Symfony AI</strong> Demo
2222
</a>
2323
<div class="collapse navbar-collapse">
24-
<ul class="navbar-nav ms-auto me-2 mb-0">
24+
<ul class="navbar-nav ms-auto me-2 mb-0 small">
2525
<li class="nav-item">
2626
<a class="nav-link" href="{{ path('blog') }}">{{ ux_icon('mdi:symfony', { height: '20px', width: '20px' }) }} Symfony Blog Bot</a>
2727
</li>
@@ -37,6 +37,9 @@
3737
<li class="nav-item">
3838
<a class="nav-link" href="{{ path('video') }}">{{ ux_icon('tabler:video-filled', { height: '20px', width: '20px' }) }} Video Bot</a>
3939
</li>
40+
<li class="nav-item">
41+
<a class="nav-link" href="{{ path('stream') }}">{{ ux_icon('mdi:car-turbocharger', { height: '20px', width: '20px' }) }} Turbo Stream Bot</a>
42+
</li>
4043
<li class="nav-item"><span class="nav-link">|</span></li>
4144
<li class="nav-item">
4245
<a class="nav-link" href="https://github.com/symfony/ai" target="_blank">{{ ux_icon('mdi:github', { height: '20px', width: '20px' }) }} GitHub</a>

0 commit comments

Comments
 (0)