Enhance Realtime Prompting Guide with new tips and model updates

- Updated the Realtime prompting guide to reflect the new gpt-realtime model, emphasizing improved instruction following and voice quality.
- Added a new section with general tips for effective prompting techniques.
- Revised existing content for clarity and consistency, including formatting changes and updated audio file references.
- Introduced a new audio file for better demonstration of concepts.

Author: minh-hoque

examples/Realtime_prompting_guide.ipynb

@@ -18,13 +18,14 @@
     "Speech-to-speech systems are essential for enabling voice as a core AI interface. Today’s release enhances robustness and usability, giving enterprises the confidence to deploy mission-critical voice agents at scale.\n",
     "\n",
     "\n",
-    "The new gpt-4o-realtime-08-11 model delivers stronger instruction following, more reliable tool calling, noticeably better voice quality, and an overall smoother feel. These gains make it practical to move from chained approaches to true realtime experiences, cutting latency while keeping a consistent conversational tone.\n",
+    "The new gpt-realtime model delivers stronger instruction following, more reliable tool calling, noticeably better voice quality, and an overall smoother feel. These gains make it practical to move from chained approaches to true realtime experiences, cutting latency, carrying over more nuance from spoken input, and producing responses that sound more natural and expressive.\n",
     "\n",
     "Realtime model benefit from different prompting techniques that wouldn't directly apply to text based models. This prompting guide starts with a simple prompt skeleton, then walks through each part with practical tips, small patterns you can copy, and examples you can adapt to your use case.\n",
     "\n",
     "# Table of Contents\n",
     "\n",
     "- [Realtime Prompting Guide](#realtime-prompting-guide)\n",
+    "- [General Tips](#general-tips)\n",
     "- [Prompt Structure](#prompt-structure)\n",
     "- [Role and Objective](#role-and-objective)\n",
     "- [Personality and Tone](#personality-and-tone)\n",
@@ -65,6 +66,21 @@
     "from IPython.display import Audio, display"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "cfec8f54",
+   "metadata": {},
+   "source": [
+    "# General Tips\n",
+    "- **Iterate relentlessly**: Small wording changes can make or break behavior.\n",
+    "  - Example: swapping “inaudible” → “intelligible” boosted noisy input handling.\n",
+    "- **Prefer bullets over paragraphs**: Clear, short bullets outperform long paragraphs.\n",
+    "- **Guide with examples**: The model strongly follows onto sample phrases.\n",
+    "- **Be precise**: Ambiguity or conflicting instructions = degraded performance similar to GPT-5.\n",
+    "- **Control language**: Pin output to a target language if you see drift.\n",
+    "- **Fight repetition**: Add a Variety rule to reduce robotic phrasing.\n"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "629a2289",
@@ -136,7 +152,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
+   "execution_count": 14,
    "id": "093a6b4f",
    "metadata": {},
    "outputs": [
@@ -154,13 +170,13 @@
        "<IPython.lib.display.Audio object>"
       ]
      },
-     "execution_count": 26,
+     "execution_count": 14,
      "metadata": {},
      "output_type": "execute_result"
     }
    ],
    "source": [
-    "Audio(\"./data/audio/obj_o6.wav\")"
+    "Audio(\"./data/audio/obj_06.wav\")"
    ]
   },
   {
@@ -319,7 +335,7 @@
     "Friendly, calm and approachable expert customer service assistant.\n",
     "\n",
     "## Tone\n",
-    "Tone: Warm, concise, confident, never fawning.\n",
+    "Warm, concise, confident, never fawning.\n",
     "\n",
     "## Length\n",
     "2–3 sentences per turn.\n",
@@ -342,7 +358,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": null,
    "id": "6dce17d5",
    "metadata": {},
    "outputs": [
@@ -366,14 +382,6 @@
     }
    ],
    "source": [
-    "import soundfile as sf\n",
-    "from IPython.display import Audio, display\n",
-    "# Load AIFC\n",
-    "data, samplerate = sf.read(\"/Users/minh.hoque/work/github/openai-cookbook/examples/data/audio/multi-emotion.aifc\")\n",
-    "\n",
-    "# Save as WAV for better browser compatibility\n",
-    "sf.write(\"/Users/minh.hoque/work/github/openai-cookbook/examples/data/audio/multi-emotion.wav\", data, samplerate)\n",
-    "\n",
     "Audio(\"/Users/minh.hoque/work/github/openai-cookbook/examples/data/audio/multi-emotion.wav\")"
    ]
   },
@@ -410,13 +418,14 @@
     "Friendly, calm and approachable expert customer service assistant.\n",
     "\n",
     "## Tone\n",
-    "Tone: Warm, concise, confident, never fawning.\n",
+    "Warm, concise, confident, never fawning.\n",
     "\n",
     "## Length\n",
     "2–3 sentences per turn.\n",
     "\n",
     "## Pacing\n",
-    "Deliver your audio response fast, but do not sound rushed. Do not modify the content of your response, only increase speaking speed for the same response.\n",
+    "- Deliver your audio response fast, but do not sound rushed.\n",
+    "- Do not modify the content of your response, only increase speaking speed for the same response.\n",
     "```"
    ]
   },
@@ -430,7 +439,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 15,
    "id": "9754730f",
    "metadata": {},
    "outputs": [
@@ -454,14 +463,6 @@
     }
    ],
    "source": [
-    "import soundfile as sf\n",
-    "\n",
-    "# Load AIFC\n",
-    "data, samplerate = sf.read(\"/Users/minh.hoque/work/github/openai-cookbook-internal/examples/data/audio/pace_06.wav.aif\")\n",
-    "\n",
-    "# Save as WAV for better browser compatibility\n",
-    "sf.write(\"/Users/minh.hoque/work/github/openai-cookbook-internal/examples/data/audio/pace_06.wav\", data, samplerate)\n",
-    "\n",
     "Audio(\"./data/audio/pace_06.wav\")"
    ]
   },
@@ -534,13 +535,15 @@
     "Friendly, calm and approachable expert customer service assistant.\n",
     "\n",
     "## Tone\n",
-    "Tone: Warm, concise, confident, never fawning.\n",
+    "Warm, concise, confident, never fawning.\n",
     "\n",
     "## Length\n",
     "2–3 sentences per turn.\n",
     "\n",
     "## Language\n",
-    "The conversation will only be in English. Do not respond in any other language even if the user asks for it, only respond in english.\n",
+    "- The conversation will be only in English.\n",
+    "- Do not respond in any other language even if the user asks.\n",
+    "- If the user speaks another language, politely explain that support is limited to English.\n",
     "```"
    ]
   },
@@ -575,7 +578,7 @@
     "Friendly, calm and approachable expert voice tutor.\n",
     "\n",
     "## Tone\n",
-    "Tone: Warm, concise, confident, never fawning.\n",
+    "Warm, concise, confident, never fawning.\n",
     "\n",
     "## Length\n",
     "2–3 sentences per turn.\n",
@@ -629,16 +632,19 @@
     "Friendly, calm and approachable expert customer service assistant.\n",
     "\n",
     "## Tone\n",
-    "Tone: Warm, concise, confident, never fawning.\n",
+    "Warm, concise, confident, never fawning.\n",
     "\n",
     "## Lenght\n",
     "2–3 sentences per turn.\n",
     "\n",
     "## Language\n",
-    "The conversation will only be in English. Do not respond in any other language even if the user asks for it, only respond in english.\n",
+    "- The conversation will be only in English.\n",
+    "- Do not respond in any other language even if the user asks.\n",
+    "- If the user speaks another language, politely explain that support is limited to English.\n",
     "\n",
     "## Variety\n",
-    "Variety: Do not repeat the same sentence twice. Vary your responses so it doesn't sound robotic.\n",
+    "- Do not repeat the same sentence twice.\n",
+    "- Vary your responses so it doesn't sound robotic.\n",
     "```"
    ]
   },
@@ -798,7 +804,8 @@
     "### Example\n",
     "```\n",
     "# Instructions/Rules\n",
-    "- When reading numbers or codes, speak each character separately, separated by hyphens (e.g., 4-1-5). Repeat EXACTLY the provided number, do not forget any.\n",
+    "- When reading numbers or codes, speak each character separately, separated by hyphens (e.g., 4-1-5). \n",
+    "- Repeat EXACTLY the provided number, do not forget any.\n",
     "```"
    ]
   },
@@ -943,16 +950,10 @@
     "...\n",
     "\n",
     "\n",
-    "## Unclear audio\n",
-    "- Always respond in the same language the user is speaking in, if intelligible. (optional)\n",
-    "- Only respond to clear speech or text.\n",
-    "- If the user's audio is not clear (e.g. background noise/inaudible/silent/unintelligible) or if you did not fully hear or understand the user, ask for clarification using English phrases such as “I didn’t catch that—mind repeating?”. Vary the phrases.'\n",
-    "\n",
-    "\n",
-    "## Preferred Response Language\n",
-    "- Always respond in the same language the user is speaking. This is the preferred language for the session.\n",
-    "- If you cannot clearly determine the user's language (e.g., due to background noise, inaudible, silent, unintelligible, or ambiguous input), do not guess or fabricate. Instead, politely ask the user to repeat or clarify.\n",
-    "- If you cannot determine the user's preferred language, ask for clarification in English.\n",
+    "## Unclear audio \n",
+    "- Always respond in the same language the user is speaking in, if intelligible.\n",
+    "- Only respond to clear audio or text. \n",
+    "- If the user's audio is not clear (e.g. ambiguous input/background noise/silent/unintelligible) or if you did not fully hear or understand the user, ask for clarification using {preferred_language} phrases.\n",
     "```"
    ]
   },
@@ -1030,7 +1031,7 @@
     "...\n",
     "```\n",
     "\n",
-    "We need to ensure the tool list has the same availability tools and do not contradict each other:\n",
+    "We need to ensure the tool list has the same availability tools and **the descriptions do not contradict each other**:\n",
     "\n",
     "```json\n",
     "[\n",
@@ -1104,7 +1105,7 @@
    "id": "a752a4a6",
    "metadata": {},
    "source": [
-    "### Example\n",
+    "#### Example\n",
     "```python\n",
     "tools = [\n",
     "  {\n",
@@ -1182,7 +1183,7 @@
    "id": "30ccc1d9",
    "metadata": {},
    "source": [
-    "*Note: If you notice the model disregarding instructions or constraints about your tool call, there’s a chance this might be due to the fact you are asking it to be very proactive.*"
+    "*Tip: If a tool call can fail unpredictably, add clear failure-handling instructions so the model responds gracefully.*"
    ]
   },
   {
@@ -1232,6 +1233,14 @@
     "```"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "24579f54",
+   "metadata": {},
+   "source": [
+    "*Tip: If your tool call can fail in unpredictablr ways, try adding instructions on how to handle tool call outputs failures so taht the model can behave*"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "edebafe2",
@@ -1610,8 +1619,8 @@
     "As use case complexity grows, you need maintainable structure without overloading the model.\n",
     "\n",
     "Two patterns for complex scenarios: \n",
-    "1. State machine conversation flow\n",
-    "2. Dynamic flow via session updates.\n"
+    "1. Conversation Flow as State Machine\n",
+    "2. Dynamic Conversation Flow via session.updates\n"
    ]
   },
   {
@@ -1628,7 +1637,7 @@
    "id": "b77ddc02",
    "metadata": {},
    "source": [
-    "### Example\n",
+    "#### Example\n",
     "```json\n",
     "# Conversation States\n",
     "[\n",
@@ -1824,24 +1833,21 @@
     "### Example\n",
     "```\n",
     "# Safety & Escalation\n",
-    "**When to escalate (no extra troubleshooting):**\n",
-    "- Safety risk (self-harm, threats, harassment)\n",
-    "- User explicitly asks for a human\n",
-    "- Severe dissatisfaction (e.g., “extremely frustrated,” repeated complaints, profanity)\n",
-    "- **2** failed tool attempts on the same task **or** **3** consecutive no-match/no-input events\n",
-    "- Out-of-scope or restricted (e.g., real-time news, financial/legal/medical advice)\n",
-    "\n",
-    "\n",
-    "**Examples of what to say (Mandatory phrase before handoff):**\n",
-    "- “I'm sorry for the trouble — I'm transferring you to a specialist now. **.”\n",
-    "\n",
+    "- When to escalate (no extra troubleshooting):\n",
+    "  - Safety risk (self-harm, threats, harassment)\n",
+    "  - User explicitly asks for a human\n",
+    "  - Severe dissatisfaction (e.g., “extremely frustrated,” repeated complaints, profanity)\n",
+    "  - **2** failed tool attempts on the same task **or** **3** consecutive no-match/no-input events\n",
+    "  - Out-of-scope or restricted (e.g., real-time news, financial/legal/medical advice)\n",
     "\n",
-    "**Then call the tool:** `escalate_to_human`\n",
+    "- Examples of what to say (Mandatory phrase before handoff):\n",
+    "  - “I'm sorry for the trouble — I'm transferring you to a specialist now. **.”\n",
     "\n",
+    "- Then call the tool: `escalate_to_human`\n",
     "\n",
-    "Examples that would require escalation:\n",
-    "- “This is the third time the reset didn’t work. Just get me a person.”\n",
-    "- “I am extremely frustrated!”\n",
+    "- Examples that would require escalation:\n",
+    "  - “This is the third time the reset didn’t work. Just get me a person.”\n",
+    "  - “I am extremely frustrated!”\n",
     "```"
    ]
   },