Add limerick example (#41)

* Add limerick example

* Remove redundant var

* Add interruptions to agents and update poet example

* Update dockerfile

* Update dockerfile

* Update openapi

* Update tests

* Update README

* Remove litestar from deps

* Fix readme format

* Fix typo in README

* Handle OpenAI errors

* Print raw OpenAI server errors

* Remove errors

* Use dotenv

* Make note more clear

Author: AHGIJMKLKKZNPJKQR

examples/poet_chat/.env.example

@@ -0,0 +1,6 @@
+# Your Fishjam envs, which you can get at https://fishjam.io/app
+FISHJAM_ID="your-fishjam-id"
+FISHJAM_MANAGEMENT_TOKEN="your-management-token"
+
+# An API key for the OpenAI Realtime API. You can generate one at https://platform.openai.com/api-keys
+OPENAI_API_KEY="your-api-key"

examples/poet_chat/README.md

@@ -0,0 +1,30 @@
+# Limerick Poet demo
+
+This directory contains an example implementation of a real-time chat agent using the [Fishjam](https://fishjam.io) and [OpenAI Agents](https://github.com/openai/openai-agents-python) Python SDKs.
+
+The agent introduces itself when the user joins and creates limericks based on what the user says.
+The agent handles interruptions from users to ensure a natural conversation flow.
+
+## Running
+
+> [!IMPORTANT]
+> All commands should be run from the `examples/poet_chat` directory
+
+Make sure to [install uv](https://docs.astral.sh/uv/getting-started/installation/) if you don't have it already.
+Once you have `uv` installed, fetch the dependencies with
+
+```bash
+uv sync
+```
+
+To run the app, first copy [`.env.example`](./.env.example) to `.env` and populate your environment variables.
+
+Once you have populated `.env`, you can run the demo with
+
+```bash
+uv run ./main.py
+```
+
+A peer token should be generated and printed to standard output.
+You can then use the [minimal-react](https://github.com/fishjam-cloud/web-client-sdk/tree/main/examples/react-client)
+demo app to connect with this token and talk with the agent!

examples/poet_chat/main.py

@@ -0,0 +1,64 @@
+import asyncio
+
+from poet_chat.agent import poet_runner
+from poet_chat.config import (
+    AGENT_OPTIONS,
+    fishjam_client,
+)
+from poet_chat.notifier import make_notifier
+
+from fishjam.agent import OutgoingAudioTrackOptions
+
+
+async def main():
+    room = fishjam_client.create_room()
+    _, token = fishjam_client.create_peer(room.id)
+    print(f"Join the chat with the following token: {token}")
+
+    agent = fishjam_client.create_agent(room.id, AGENT_OPTIONS)
+    async with (
+        agent.connect() as fishjam_session,
+        await poet_runner.run() as openai_session,
+    ):
+        track = await fishjam_session.add_track(
+            OutgoingAudioTrackOptions(
+                sample_rate=24000, metadata={"type": "microphone"}
+            )
+        )
+
+        async def _openai_recv():
+            msg = ""
+            async for event in openai_session:
+                if event.type == "audio":
+                    audio = event.audio.data
+                    await track.send_chunk(audio)
+                elif event.type == "audio_interrupted":
+                    await track.interrupt()
+                elif event.type == "raw_model_event":
+                    if event.data.type == "input_audio_transcription_completed":
+                        print(f"Peer said:\n{event.data.transcript}\n")
+                    elif event.data.type == "transcript_delta":
+                        msg += event.data.delta
+                    elif event.data.type == "turn_ended":
+                        print(f"Agent said:\n{msg}\n")
+                        msg = ""
+                    elif event.data.type == "exception":
+                        raise event.data.exception
+                    elif event.data.type == "raw_server_event":
+                        match event.data.data:
+                            case {"response": {"status": "failed"}}:
+                                print(event.data.data)
+                                raise RuntimeError("Raw server error from OpenAI API!")
+
+        async def _fishjam_recv():
+            async for event in fishjam_session.receive():
+                await openai_session.send_audio(event.data)
+
+        async with asyncio.TaskGroup() as tg:
+            tg.create_task(make_notifier(openai_session).connect())
+            tg.create_task(_openai_recv())
+            tg.create_task(_fishjam_recv())
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/poet_chat/poet_chat/__init__.py

@@ -0,0 +1,19 @@
+from agents.realtime import RealtimeAgent, RealtimeRunner
+
+from poet_chat.config import OPENAI_INSTRUCTIONS
+
+poet = RealtimeAgent(name="Poet", instructions=OPENAI_INSTRUCTIONS)
+poet_runner = RealtimeRunner(
+    starting_agent=poet,
+    config={
+        "model_settings": {
+            "voice": "alloy",
+            "modalities": ["audio", "text"],
+            "input_audio_format": "pcm16",
+            "output_audio_format": "pcm16",
+            "turn_detection": {
+                "type": "semantic_vad",
+            },
+        }
+    },
+)

examples/poet_chat/poet_chat/agent.py

@@ -0,0 +1,29 @@
+import os
+from pathlib import Path
+
+import dotenv
+
+from fishjam import AgentOptions, AgentOutputOptions, FishjamClient
+
+dotenv.load_dotenv()
+
+FISHJAM_ID = os.environ["FISHJAM_ID"]
+FISHJAM_TOKEN = os.environ["FISHJAM_MANAGEMENT_TOKEN"]
+FISHJAM_URL = os.getenv("FISHJAM_URL")
+
+AGENT_OPTIONS = AgentOptions(output=AgentOutputOptions(audio_sample_rate=24000))
+
+OPENAI_MODEL = "gpt-realtime"
+
+PROMPT_DIR = Path(__file__).parents[1] / "prompts"
+
+INSTRUCTION_PATH = PROMPT_DIR / "instructions.md"
+GREET_PATH = PROMPT_DIR / "greet.md"
+
+with open(INSTRUCTION_PATH) as prompt:
+    OPENAI_INSTRUCTIONS = prompt.read()
+
+with open(GREET_PATH) as prompt:
+    OPENAI_GREET = prompt.read()
+
+fishjam_client = FishjamClient(FISHJAM_ID, FISHJAM_TOKEN, fishjam_url=FISHJAM_URL)

examples/poet_chat/poet_chat/config.py

@@ -0,0 +1,27 @@
+from agents.realtime import RealtimeSession
+
+from fishjam import FishjamNotifier
+from fishjam.events import (
+    ServerMessagePeerConnected,
+    ServerMessagePeerType,
+)
+from fishjam.events.allowed_notifications import AllowedNotification
+
+from .config import FISHJAM_ID, FISHJAM_TOKEN, FISHJAM_URL, OPENAI_GREET
+
+
+def make_notifier(poet: RealtimeSession) -> FishjamNotifier:
+    notifier = FishjamNotifier(FISHJAM_ID, FISHJAM_TOKEN, fishjam_url=FISHJAM_URL)
+
+    @notifier.on_server_notification
+    async def _(notification: AllowedNotification):
+        match notification:
+            case ServerMessagePeerConnected(
+                peer_type=ServerMessagePeerType.PEER_TYPE_WEBRTC,
+            ):
+                await handle_peer_connected()
+
+    async def handle_peer_connected():
+        await poet.send_message(OPENAI_GREET)
+
+    return notifier

examples/poet_chat/poet_chat/notifier.py

@@ -0,0 +1,19 @@
+**Objective:** Welcome users to the real-time limerick generation experience by introducing yourself as their poetic guide and requesting a list of words for personalized poetry creation.
+
+**Instructions:**
+
+1. **Introduction:**
+
+   - Begin with a warm and engaging introduction, presenting yourself as a creative and friendly poet eager to craft delightful limericks.
+   - Briefly explain the purpose of the session: to create fun and custom limericks based on the user's input.
+
+2. **Request for Words:**
+
+   - Invite the user to share a list of words that capture their current mood, interests, or thoughts triggered by listening to the audio.
+   - Encourage them to be imaginative and creative in their word selection to make the limerick more personalized and engaging.
+
+3. **Set Expectations:**
+   - Assure the user that each limerick will be crafted uniquely with their provided words, promising a lighthearted and entertaining poetic experience.
+   - Mention the traditional limerick structure (AABBA) that will guide the creation process.
+
+By following these instructions, create a welcoming and interactive atmosphere that encourages user participation and sets the stage for crafting personalized limericks.

examples/poet_chat/prompts/greet.md

@@ -0,0 +1,19 @@
+**Objective:** Craft limericks in real-time, using input audio clips or live audio feeds to prompt user interaction. The generated poetry should be based solely on a list of words provided by the user in response to the audio.
+
+**Instructions:**
+
+1. **User Interaction:**
+
+   - Prompt the user to listen to the audio and provide a list of words that capture their response or impression from it.
+
+2. **Limerick Construction:**
+
+   - Use the user-provided list of words to guide the theme and imagery of the limerick.
+   - Ensure the limerick adheres to the traditional AABBA rhyme scheme and rhythm, typically with lines 1, 2, and 5 containing three metrical feet and lines 3 and 4 containing two metrical feet.
+   - Creativity is key—incorporate humor or cleverness where possible to enhance the impact.
+
+3. **Language and Performance:**
+   - Ensure all poetry is crafted and recited in English.
+   - Use expressive language to vividly depict themes inspired by the provided words, aiming to entertain and engage the listener.
+
+By following these instructions, generate limericks that transform user-provided words inspired by audio inputs into lively and entertaining poetic pieces.

examples/poet_chat/prompts/instructions.md

@@ -0,0 +1,10 @@
+[project]
+name = "poet_chat"
+version = "0.1.0"
+description = "Fishjam voice agent example"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = ["fishjam-server-sdk", "openai-agents[voice]>=0.2.11"]
+
+[tool.uv.sources]
+fishjam-server-sdk = { workspace = true }