forked from kwindla/aiewf-eval
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathturns.py
More file actions
233 lines (233 loc) · 15.1 KB
/
turns.py
File metadata and controls
233 lines (233 loc) · 15.1 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
turns = [{'input': "I'm trying to decide whether to come for workshop day. When are the workshops?",
'golden_text': 'Workshop day is on Tuesday, June 3rd. There will be exclusive, hands-on '
'workshops across five different tracks, with topics for all experience levels.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_000.wav'},
{'input': 'Are there any workshops about Gemini?',
'golden_text': 'Yes, there are two workshops on that day that feature Gemini!\n'
'\n'
'First, at 10:40 AM, there\'s **"Building Voice Agents with Gemini and '
'Pipecat,"** where you can build and deploy your own voice agent.\n'
'\n'
'Later, at 1:00 PM, you can attend **"AI Engineering with the Google Gemini 2.5 '
'Model Family,"** which is a hands-on workshop where you\'ll learn to use Gemini '
'2.5 Pro with agentic tooling.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_001.wav'},
{'input': 'How many conference tracks are there?',
'golden_text': 'There will be 19 tracks of talks, panels, and demos during the main conference '
'days on June 4th and 5th.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_002.wav'},
{'input': 'Is there a track about MCP?',
'golden_text': 'Yes, there is an MCP track on June 4th. It focuses on the Model Context Protocol '
'and covers its origins, how to build with it, and its future possibilities for '
'agent-to-agent communication.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_003.wav'},
{'input': 'How many sessions are in that track?',
'golden_text': 'The MCP track on June 4th features a variety of sessions, including keynotes and '
'talks. There are also several workshops on June 3rd that are focused on building '
'with MCP. The sessions cover topics like the origins of MCP, its practical '
'implementation, and its role in the future of AI agents.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_004.wav'},
{'input': 'It seems like there are so many great sessions. Can you suggest some sessions for me?',
'golden_text': 'I agree, there are a lot of exciting sessions! To give you the best '
"recommendations, could you tell me what topics you're most interested in? For "
'example, are you into building agents, infrastructure, voice technology, or '
'something else?',
'required_function_call': None,
'audio_file': 'turns_audio/turn_005.wav'},
{'input': "I'm particularly interested in Voice AI and in techniques for building reliable "
'agents.',
'golden_text': 'Those are fantastic topics! To help me narrow down the best sessions for you, '
"could you let me know which day or days you'll be at the conference? And will "
'you be there all day, or just for the morning or afternoon?',
'required_function_call': None,
'audio_file': 'turns_audio/turn_006.wav'},
{'input': "I'll be at the conference on June 4th all day.",
'golden_text': 'Excellent! Based on your interests in Voice AI and building reliable agents, I '
'have two great suggestions for you for June 4th.\n'
'\n'
'First, I\'d recommend the session **"[Voice Keynote] Your realtime AI is ngmi"** '
"by Sean DuBois and Kwindla Kramer. It's on June 4th at 11:15 AM. They'll be "
'discussing why you need to design real-time AI systems from the network layer up '
'to be successful.\n'
'\n'
'Second, you might like **"12 Factor Agents - Principles of Reliable LLM '
'Applications"** by Dexter Horthy. That\'s on June 4th at 11:35 AM. He\'ll be '
'sharing principles for building LLM-powered software that is reliable enough for '
'production customers.\n'
'\n'
'I hope these suggestions are helpful for planning your day! Let me know if you '
'need anything else.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_007.wav'},
{'input': 'Actually, I made a mistake. I will only be at the conference on June 5th.',
'golden_text': "No problem at all! Let's find you some great sessions for June 5th. Based on "
'your interest in Voice AI and building reliable agents, here are two '
'recommendations for that day:\n'
'\n'
'First, for your interest in Voice AI, I suggest **"Serving Voice AI at Scale,"** '
"which is a fireside chat with Arjun Desai and Rohit Talluri. It's happening on "
"June 5th at 1:15 PM. They'll be doing a technical deep dive into the best "
'practices for building a state-of-the-art inference stack for voice AI.\n'
'\n'
'Second, to learn more about reliable agents, I\'d recommend **"Ship it! Building '
'Production-Ready Agents"** by Mike Chambers. This talk is on June 5th at 12:15 '
'PM and will cover the practical challenges and solutions for deploying AI agents '
'in real-world production environments.\n'
'\n'
'I hope these updated suggestions are helpful for planning your day on June 5th! '
'Let me know if you need anything else',
'required_function_call': None,
'audio_file': 'turns_audio/turn_008.wav'},
{'input': 'I heard there will be community organized hallway track sessions. Can I make a '
'suggestion for a hallway track session?',
'golden_text': 'Yes, I can help with that! I can submit a suggestion for a hallway track session '
'for you. What is your name and what is your suggestion?',
'required_function_call': None,
'audio_file': 'turns_audio/turn_009.wav'},
{'input': 'My name is Jennifer Smith.',
'golden_text': "Thanks, Jennifer! And what's your suggestion for the hallway track session?",
'required_function_call': None,
'audio_file': 'turns_audio/turn_010.wav'},
{'input': 'My suggestion is for a session about open telemetry tracing.',
'golden_text': "Thank you, Jennifer! I've submitted your suggestion for a session on open "
'telemetry tracing. Is there anything else I can help you with today?',
'required_function_call': {'name': 'submit_session_suggestion',
'args': {'name': 'Jennifer Smith',
'suggestion_text': 'A session about open telemetry '
'tracing.'}},
'function_call_response': {'status': 'success'},
'audio_file': 'turns_audio/turn_011.wav'},
{'input': 'Oh, one more suggestion. How about a session on state machine abstractions for complex '
'workflows?.',
'golden_text': "Thank you, Jennifer! I've submitted your suggestion for a session on state "
'machine abstractions for complex workflows. Is there anything else I can help '
'you with today?',
'required_function_call': {'name': 'submit_session_suggestion',
'args': {'suggestion_text': 'A session on state machine abstractions '
'for complex workflows.',
'name': 'Jennifer Smith'}},
'function_call_response': {'status': 'success'},
'audio_file': 'turns_audio/turn_012.wav'},
{'input': 'Thanks for submitting both session suggestions. Is there food at the conference?',
'golden_text': 'Yes, meals are provided throughout the conference. Continental breakfast and '
'lunch are available on all three days, June 3rd through the 5th. There are also '
'morning and afternoon breaks with refreshments, a Welcome Reception on the '
'evening of the 3rd, and an afterparty on the evening of the 4th.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_013.wav'},
{'input': 'Will there be vegan options?',
'golden_text': 'Of course! I can submit a dietary request to ensure there are vegan options '
'available for you. I have your name as Jennifer Smith. Is that correct?',
'required_function_call': None,
'audio_file': 'turns_audio/turn_014.wav'},
{'input': 'Yes.',
'golden_text': "Thank you, Jennifer. I've submitted your request for vegan meals. Is there "
'anything else I can help you with today?',
'required_function_call': {'name': 'submit_dietary_request',
'args': {'name': 'Jennifer Smith', 'dietary_preference': 'vegan'}},
'function_call_response': {'status': 'success'},
'audio_file': 'turns_audio/turn_015.wav'},
{'input': "Yes. I'm having trouble with the mobile app.",
'golden_text': "I'm sorry to hear you're having trouble. I can create a tech support ticket for "
'you. What seems to be the problem with the mobile app?',
'required_function_call': None,
'audio_file': 'turns_audio/turn_016.wav'},
{'input': "I can't access the location maps.",
'golden_text': "Thank you, Jennifer. I've created a tech support ticket for you regarding the "
'issue with accessing location maps on the mobile app. Someone from our tech team '
'will look into it. Is there anything else I can help you with?',
'required_function_call': {'name': 'request_tech_support',
'args': {'issue_description': 'Cannot access location maps on the '
'mobile app.',
'name': 'Jennifer Smith'}},
'function_call_response': {'status': 'success'},
'audio_file': 'turns_audio/turn_017.wav'},
{'input': "What's the weather supposed to be like in San Francisco tomorrow?",
'golden_text': "I'm sorry, I don't have weather information. I'm the voice assistant for the AI "
"Engineer World's Fair 2025 and can only help with questions about the event. Is "
'there anything about the conference I can help you with?',
'required_function_call': None,
'audio_file': 'benchmarks/_shared/audio/turn_018.wav'},
{'input': "Well, since I can't access the mobile maps, can you tell me how to get to Salon 2?",
'golden_text': "I understand you're looking for Salon 2 since the maps in the app aren't working "
"for you. Unfortunately, I don't have the ability to provide directions or "
'information about specific locations within the venue. The tech support team has '
'been notified about the app issue. Can I help you with any information about the '
'sessions or speakers instead?',
'required_function_call': None,
'audio_file': 'turns_audio/turn_019.wav'},
{'input': 'Is there a talk by Charles Frye?',
'golden_text': 'Yes, Charles Frye is speaking at the conference. He has two sessions on June '
'4th.\n'
'\n'
'First, at 11:15 AM, he\'s giving a workshop called **"What every AI engineer '
'needs to know about GPUs."**\n'
'\n'
'Then, at 12:45 PM, he\'s giving a talk titled **"How fast are LLM inference '
'engines anyway?"**\n'
'\n'
'The workshop is part of the Infrastructure track, and the talk is part of the AI '
'in Action track.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_020.wav'},
{'input': 'Where is the second one?',
'golden_text': 'That talk, "How fast are LLM inference engines anyway?", will be in the Juniper: '
'Expo Sessions room.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_021.wav'},
{'input': "I heard that there's some way to vote for our favorite talk of the conference. Is that "
'true?',
'golden_text': "Yes, that's correct! You can vote for your favorite sessions. I can help you "
"with that. I'll just need the Session ID for the talk you'd like to vote for. I "
'already have your name as Jennifer Smith.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_022.wav'},
{'input': "I don't know the session ID, but the talk was by Harald Kirschner.",
'golden_text': 'Of course. Harald Kirschner is involved in three sessions. Could you please tell '
"me which one you'd like to vote for?\n"
'\n'
'* First, there\'s the workshop on June 3rd titled: **"Real-World Development '
'with GitHub Copilot and VS Code."**\n'
'* Second, is the talk on June 4th titled: **"Full Spectrum MCP: Uncovering '
'Hidden Servers and Clients Capabilities."**\n'
'* And third, the talk on June 5th is titled: **"Vibe Coding at Scale: '
'Customizing AI Assistants for Enterprise Environments."**',
'required_function_call': None,
'audio_file': 'turns_audio/turn_023.wav'},
{'input': "I'd like to vote for the one about vibe coding.",
'golden_text': "Thank you, Jennifer! I've submitted your vote for the session on Vibe Coding at "
'Scale: Customizing AI Assistants for Enterprise Environments. Is there anything '
'else I can help you with today?',
'required_function_call': {'name': 'vote_for_session',
'args': {'session_id': '936902', 'name': 'Jennifer Smith'}},
'function_call_response': {'status': 'success'},
'audio_file': 'turns_audio/turn_024.wav'},
{'input': 'Yes — what time is lunch on June 5th?',
'golden_text': 'Lunch on June 5th is from 12:00 PM to 2:00 PM in Salons 9–15: Expo Hall.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_025.wav'},
{'input': "Could you also confirm if there's an afterparty on June 4th?",
'golden_text': 'Yes. The Toolbit Afterparty is on June 4th from 5:15 PM to 7:00 PM in Salons '
'9–15: Expo Hall.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_026.wav'},
{'input': 'Are there community meetups on the evening of June 5th?',
'golden_text': 'Yes. Community Meetups take place on June 5th from 5:30 PM to 9:00 PM in the '
'Atrium: Event Hub.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_027.wav'},
{'input': 'One last detail: when is continental breakfast on June 4th?',
'golden_text': 'On June 4th, continental breakfast is available from 7:15 AM to 9:55 AM in the '
'Grand Assembly.',
'required_function_call': None,
'audio_file': 'turns_audio/turn_028.wav'},
{'input': "I just wanted to say the conference was great. I don't have anything else.",
'golden_text': "Thank you for your feedback, Jennifer! I'm so glad you enjoyed the conference. "
'Have a great day',
'required_function_call': {'name': 'end_session', 'args': {}},
'audio_file': 'turns_audio/turn_029.wav'}]