Skip to content

Commit bdeff55

Browse files
minh-hoqueminh-hoque
committed
Fixed resampling to 24 kHz
1 parent 14844d4 commit bdeff55

File tree

1 file changed

+75
-80
lines changed

1 file changed

+75
-80
lines changed

‎examples/Speech_transcription_methods.ipynb‎

Lines changed: 75 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,9 @@
55
"id": "26a10eea",
66
"metadata": {},
77
"source": [
8-
"# 🗣️ Methods of Speech-to-Text using OpenAI API\n",
8+
"# 🗣️ Methods of Speech-to-Text using OpenAI API & Agents SDK\n",
99
"\n",
10-
"**Updated : April 27 2025** \n",
10+
"**Updated : April 29 2025** \n",
1111
"This notebook provides a clear, hands-on guide for beginners to quickly get started with Speech-to-Text (STT) using the OpenAI API. You'll explore multiple practical methods, their use cases, and considerations.\n",
1212
"\n",
1313
"Assumption: For simplicity and ease of use, this notebook uses WAV audio files. Real-time microphone streaming (e.g., from web apps or microphones) is not utilized."
@@ -22,7 +22,7 @@
2222
"| Mode | Latency to **first token** | Best for (real examples) | What you still need to handle / key limits |\n",
2323
"|--------------------------------|-------------------------|--------------------------------------------------------------|-----------------------------------------------------------|\n",
2424
"| File upload + `stream=False` (blocking) | > 1 s | Voicemail, meeting recordings | • No partial results, users see nothing until file finishes <br>• Max 25 MB per request (you must chunk long audio) |\n",
25-
"| File upload + `stream=True` | < 1 s | voice memos in mobile apps | • Still requires a completed file <br>• You implement progress bars / chunked uploads |\n",
25+
"| File upload + `stream=True` | < 1 s | Voice memos in mobile apps | • Still requires a completed file <br>• You implement progress bars / chunked uploads |\n",
2626
"| Realtime WebSocket | < 1 s | Live captions in webinars | • Audio must be pcm16, g711_ulaw, or g711_alaw <br>• Session ≤ 30 min, reconnect & stitch <br>• You handle speaker-turn formatting to build the full transcript |\n",
2727
"| Agents SDK VoicePipeline | < 1 s | Internal help-desk assistant | • Python-only beta <br>• API surface may change, limited customisation |"
2828
]
@@ -68,7 +68,7 @@
6868
},
6969
{
7070
"cell_type": "code",
71-
"execution_count": 13,
71+
"execution_count": 6,
7272
"id": "e4078915",
7373
"metadata": {},
7474
"outputs": [
@@ -83,6 +83,7 @@
8383
"source": [
8484
"import os, nest_asyncio, asyncio\n",
8585
"from openai import OpenAI\n",
86+
"import time\n",
8687
"nest_asyncio.apply()\n",
8788
"\n",
8889
"# ✏️ Put your key in an env-var or just replace the call below.\n",
@@ -145,7 +146,7 @@
145146
},
146147
{
147148
"cell_type": "code",
148-
"execution_count": 14,
149+
"execution_count": 2,
149150
"id": "ab545e4c",
150151
"metadata": {},
151152
"outputs": [
@@ -179,7 +180,7 @@
179180
},
180181
{
181182
"cell_type": "code",
182-
"execution_count": 15,
183+
"execution_count": 10,
183184
"id": "7ae4af8d",
184185
"metadata": {},
185186
"outputs": [
@@ -255,67 +256,16 @@
255256
},
256257
{
257258
"cell_type": "code",
258-
"execution_count": 16,
259+
"execution_count": null,
259260
"id": "d027fdb9",
260261
"metadata": {},
261262
"outputs": [
262263
{
263264
"name": "stdout",
264265
"output_type": "stream",
265266
"text": [
266-
"TranscriptionTextDeltaEvent(delta='The', type='transcript.text.delta', logprobs=None)\n",
267-
"TranscriptionTextDeltaEvent(delta=' stale', type='transcript.text.delta', logprobs=None)\n",
268-
"TranscriptionTextDeltaEvent(delta=' smell', type='transcript.text.delta', logprobs=None)\n",
269-
"TranscriptionTextDeltaEvent(delta=' of', type='transcript.text.delta', logprobs=None)\n",
270-
"TranscriptionTextDeltaEvent(delta=' old', type='transcript.text.delta', logprobs=None)\n",
271-
"TranscriptionTextDeltaEvent(delta=' beer', type='transcript.text.delta', logprobs=None)\n",
272-
"TranscriptionTextDeltaEvent(delta=' l', type='transcript.text.delta', logprobs=None)\n",
273-
"TranscriptionTextDeltaEvent(delta='ingers', type='transcript.text.delta', logprobs=None)\n",
274-
"TranscriptionTextDeltaEvent(delta='.', type='transcript.text.delta', logprobs=None)\n",
275-
"TranscriptionTextDeltaEvent(delta=' It', type='transcript.text.delta', logprobs=None)\n",
276-
"TranscriptionTextDeltaEvent(delta=' takes', type='transcript.text.delta', logprobs=None)\n",
277-
"TranscriptionTextDeltaEvent(delta=' heat', type='transcript.text.delta', logprobs=None)\n",
278-
"TranscriptionTextDeltaEvent(delta=' to', type='transcript.text.delta', logprobs=None)\n",
279-
"TranscriptionTextDeltaEvent(delta=' bring', type='transcript.text.delta', logprobs=None)\n",
280-
"TranscriptionTextDeltaEvent(delta=' out', type='transcript.text.delta', logprobs=None)\n",
281-
"TranscriptionTextDeltaEvent(delta=' the', type='transcript.text.delta', logprobs=None)\n",
282-
"TranscriptionTextDeltaEvent(delta=' odor', type='transcript.text.delta', logprobs=None)\n",
283-
"TranscriptionTextDeltaEvent(delta='.', type='transcript.text.delta', logprobs=None)\n",
284-
"TranscriptionTextDeltaEvent(delta=' A', type='transcript.text.delta', logprobs=None)\n",
285-
"TranscriptionTextDeltaEvent(delta=' cold', type='transcript.text.delta', logprobs=None)\n",
286-
"TranscriptionTextDeltaEvent(delta=' dip', type='transcript.text.delta', logprobs=None)\n",
287-
"TranscriptionTextDeltaEvent(delta=' restores', type='transcript.text.delta', logprobs=None)\n",
288-
"TranscriptionTextDeltaEvent(delta=' health', type='transcript.text.delta', logprobs=None)\n",
289-
"TranscriptionTextDeltaEvent(delta=' and', type='transcript.text.delta', logprobs=None)\n",
290-
"TranscriptionTextDeltaEvent(delta=' zest', type='transcript.text.delta', logprobs=None)\n",
291-
"TranscriptionTextDeltaEvent(delta='.', type='transcript.text.delta', logprobs=None)\n",
292-
"TranscriptionTextDeltaEvent(delta=' A', type='transcript.text.delta', logprobs=None)\n",
293-
"TranscriptionTextDeltaEvent(delta=' salt', type='transcript.text.delta', logprobs=None)\n",
294-
"TranscriptionTextDeltaEvent(delta=' pickle', type='transcript.text.delta', logprobs=None)\n",
295-
"TranscriptionTextDeltaEvent(delta=' tastes', type='transcript.text.delta', logprobs=None)\n",
296-
"TranscriptionTextDeltaEvent(delta=' fine', type='transcript.text.delta', logprobs=None)\n",
297-
"TranscriptionTextDeltaEvent(delta=' with', type='transcript.text.delta', logprobs=None)\n",
298-
"TranscriptionTextDeltaEvent(delta=' ham', type='transcript.text.delta', logprobs=None)\n",
299-
"TranscriptionTextDeltaEvent(delta='.', type='transcript.text.delta', logprobs=None)\n",
300-
"TranscriptionTextDeltaEvent(delta=' T', type='transcript.text.delta', logprobs=None)\n",
301-
"TranscriptionTextDeltaEvent(delta='acos', type='transcript.text.delta', logprobs=None)\n",
302-
"TranscriptionTextDeltaEvent(delta=' al', type='transcript.text.delta', logprobs=None)\n",
303-
"TranscriptionTextDeltaEvent(delta=' pastor', type='transcript.text.delta', logprobs=None)\n",
304-
"TranscriptionTextDeltaEvent(delta=' are', type='transcript.text.delta', logprobs=None)\n",
305-
"TranscriptionTextDeltaEvent(delta=' my', type='transcript.text.delta', logprobs=None)\n",
306-
"TranscriptionTextDeltaEvent(delta=' favorite', type='transcript.text.delta', logprobs=None)\n",
307-
"TranscriptionTextDeltaEvent(delta='.', type='transcript.text.delta', logprobs=None)\n",
308-
"TranscriptionTextDeltaEvent(delta=' A', type='transcript.text.delta', logprobs=None)\n",
309-
"TranscriptionTextDeltaEvent(delta=' zest', type='transcript.text.delta', logprobs=None)\n",
310-
"TranscriptionTextDeltaEvent(delta='ful', type='transcript.text.delta', logprobs=None)\n",
311-
"TranscriptionTextDeltaEvent(delta=' food', type='transcript.text.delta', logprobs=None)\n",
312-
"TranscriptionTextDeltaEvent(delta=' is', type='transcript.text.delta', logprobs=None)\n",
313-
"TranscriptionTextDeltaEvent(delta=' the', type='transcript.text.delta', logprobs=None)\n",
314-
"TranscriptionTextDeltaEvent(delta=' hot', type='transcript.text.delta', logprobs=None)\n",
315-
"TranscriptionTextDeltaEvent(delta='-cross', type='transcript.text.delta', logprobs=None)\n",
316-
"TranscriptionTextDeltaEvent(delta=' bun', type='transcript.text.delta', logprobs=None)\n",
317-
"TranscriptionTextDeltaEvent(delta='.', type='transcript.text.delta', logprobs=None)\n",
318-
"TranscriptionTextDoneEvent(text='The stale smell of old beer lingers. It takes heat to bring out the odor. A cold dip restores health and zest. A salt pickle tastes fine with ham. Tacos al pastor are my favorite. A zestful food is the hot-cross bun.', type='transcript.text.done', logprobs=None)\n"
267+
"The stale smell of old beer lingers.\n",
268+
"The stale smell of old beer lingers.\n"
319269
]
320270
}
321271
],
@@ -330,7 +280,13 @@
330280
")\n",
331281
"\n",
332282
"for event in stream:\n",
333-
" print(event)"
283+
" # If this is an incremental update …\n",
284+
" if getattr(event, \"delta\", None): \n",
285+
" print(event.delta, end=\"\", flush=True)\n",
286+
" time.sleep(0.05) # simulate real-time pacing\n",
287+
" # … otherwise it’s the final transcript chunk\n",
288+
" elif getattr(event, \"text\", None):\n",
289+
" print(\"\\n\" + event.text)"
334290
]
335291
},
336292
{
@@ -375,7 +331,7 @@
375331
"#### Limitations\n",
376332
"- **Complex integration:** Requires managing WebSockets, Base64 encoding, and robust error handling. \n",
377333
"- **Session constraints:** Limited to 30-minute sessions. \n",
378-
"- **Restricted formats:** Accepts only raw PCM (no MP3 or Opus); bandwidth ≈ 1.5 Mbit/s for 16-kHz mono audio. "
334+
"- **Restricted formats:** Accepts only raw PCM (no MP3 or Opus); For pcm16, input audio must be 16-bit PCM at a 24kHz sample rate, single channel (mono), and little-endian byte order."
379335
]
380336
},
381337
{
@@ -396,8 +352,9 @@
396352
"import websockets # asyncio-based WebSocket client\n",
397353
"\n",
398354
"\n",
399-
"TARGET_SR = 16_000 # GPT-4o-Transcribe expects 16 kHz mono input\n",
400-
"PCM_SCALE = 32_767 # float32 (-1…1) → int16 conversion factor\n",
355+
"TARGET_SR = 24_000 # Realtime STT expects 24-kHz mono input\n",
356+
"PCM_SCALE = 32_767 # float32 (−1…1) → int16 conversion factor\n",
357+
"DEFAULT_CHUNK = 3_072 # 3 072 / 24 000 ≈ 128 ms (server-VAD sweet-spot)\n",
401358
"REALTIME_URL = \"wss://api.openai.com/v1/realtime?intent=transcription\"\n",
402359
"\n",
403360
"def load_and_resample(path: str, target_sr: int = TARGET_SR) -> bytes:\n",
@@ -433,13 +390,11 @@
433390
" # WebSocket payload must be JSON → encode the binary chunk as base64\n",
434391
" b64 = base64.b64encode(chunk).decode()\n",
435392
" try:\n",
436-
" await ws.send({\n",
437-
" \"type\": \"input_audio_buffer.append\",\n",
438-
" \"audio\": b64,\n",
439-
" } | {} if False else json.dumps({ # keep original semantics\n",
440-
" \"type\": \"input_audio_buffer.append\",\n",
441-
" \"audio\": b64,\n",
442-
" }))\n",
393+
" payload = {\n",
394+
" \"type\": \"input_audio_buffer.append\",\n",
395+
" \"audio\": b64,\n",
396+
" }\n",
397+
" await ws.send(json.dumps(payload))\n",
443398
" except websockets.ConnectionClosed:\n",
444399
" # Receiver closed early (e.g., after final VAD turn) – just stop.\n",
445400
" break\n",
@@ -490,7 +445,7 @@
490445
" api_key: str,\n",
491446
" *,\n",
492447
" model: str = \"gpt-4o-transcribe\",\n",
493-
" chunk: int = 2_048, # ≈128 ms @16 kHz → good server-VAD latency\n",
448+
" chunk: int = DEFAULT_CHUNK, # now sized for 24 kHz\n",
494449
" debug: bool = False,\n",
495450
") -> str:\n",
496451
" \"\"\"\n",
@@ -519,7 +474,7 @@
519474
" await ws.send(json.dumps({\n",
520475
" \"type\": \"transcription_session.update\",\n",
521476
" \"session\": {\n",
522-
" \"input_audio_format\": \"pcm16\", # little-endian, mono\n",
477+
" \"input_audio_format\": \"pcm16\", # 24 kHz mono, little-endian\n",
523478
" \"turn_detection\": {\n",
524479
" \"type\": \"server_vad\", # server does VAD\n",
525480
" \"threshold\": 0.5 # adjust if needed\n",
@@ -545,13 +500,30 @@
545500
},
546501
{
547502
"cell_type": "code",
548-
"execution_count": null,
503+
"execution_count": 16,
549504
"id": "d90de5b9",
550505
"metadata": {},
551-
"outputs": [],
506+
"outputs": [
507+
{
508+
"name": "stdout",
509+
"output_type": "stream",
510+
"text": [
511+
"The stale smell of old beer lingers.It takes heat to bring out the odor.A cold dip restores health and zest.A salt pickle tastes fine with ham.Tacos al pastor are my favorite."
512+
]
513+
},
514+
{
515+
"data": {
516+
"text/plain": [
517+
"'The stale smell of old beer lingers. It takes heat to bring out the odor. A cold dip restores health and zest. A salt pickle tastes fine with ham. Tacos al pastor are my favorite.'"
518+
]
519+
},
520+
"execution_count": 16,
521+
"metadata": {},
522+
"output_type": "execute_result"
523+
}
524+
],
552525
"source": [
553-
"WAV_PATH = \"./data/sample_audio_files/18_sec_food_story.wav\" # any 16-bit PCM WAV file (mono or stereo OK)\n",
554-
"\n",
526+
"WAV_PATH = \"./data/sample_audio_files/18_sec_food_story.wav\"\n",
555527
"# IMPORTANT: this cell must be prefixed with `await`\n",
556528
"transcript = await transcribe_wav_async(WAV_PATH, OPENAI_API_KEY, debug=False)\n",
557529
"transcript"
@@ -600,10 +572,33 @@
600572
},
601573
{
602574
"cell_type": "code",
603-
"execution_count": 17,
575+
"execution_count": 13,
604576
"id": "754a846b",
605577
"metadata": {},
606-
"outputs": [],
578+
"outputs": [
579+
{
580+
"name": "stdout",
581+
"output_type": "stream",
582+
"text": [
583+
"\n",
584+
"[User]: The stale smell of old beer lingers.\n",
585+
"[Assistant]: L'odeur rance de la vieille bière persiste.\n",
586+
"[User]: A cold dip restores health and zest.\n",
587+
"[Assistant]: Un bain froid restaure la santé et l'énergie.\n",
588+
"[User]: Tacos al pastor are my favorite.\n",
589+
"[Assistant]: Les tacos al pastor sont mes préférés."
590+
]
591+
},
592+
{
593+
"name": "stdout",
594+
"output_type": "stream",
595+
"text": [
596+
"\n",
597+
"[User]: A zestful food is the hot cross bun.\n",
598+
"[Assistant]: Un aliment plein de dynamisme est le petit pain de Pâques."
599+
]
600+
}
601+
],
607602
"source": [
608603
"import asyncio, numpy as np, soundfile as sf, resampy\n",
609604
"from agents import Agent\n",
@@ -678,7 +673,7 @@
678673
"source": [
679674
"## Conclusion \n",
680675
"\n",
681-
"In this notebook you explored multiple ways to convert speech to text with the OpenAI API and the Agents SDK—ranging from simple file uploads to fully-interactive, real-time streaming. Each workflow shines in a different scenario, so pick the one that best matches your product’s needs.\n",
676+
"In this notebook you explored multiple ways to convert speech to text with the OpenAI API and the Agents SDK, ranging from simple file uploads to fully-interactive, real-time streaming. Each workflow shines in a different scenario, so pick the one that best matches your product’s needs.\n",
682677
"\n",
683678
"### Key takeaways\n",
684679
"- **Match the method to the use-case:** \n",

0 commit comments

Comments
 (0)