Skip to content

Commit d54915f

Browse files
committed
Load scenarios from JSON for both the experiment and the log analysis
1 parent 5313933 commit d54915f

File tree

3 files changed

+117
-179
lines changed

3 files changed

+117
-179
lines changed

experiment/lib/scenarios.json

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
{
2+
"roomDoubleBooking": {
3+
"id": "roomDoubleBooking",
4+
"sender": {
5+
"name": "Alex Johnson"
6+
},
7+
"colleague": {
8+
"name": "Sarah Martinez",
9+
"firstName": "Sarah",
10+
"role": "Events Coordinator"
11+
},
12+
"recipient": {
13+
"name": "Jaden Thompson",
14+
"email": "[email protected]"
15+
},
16+
"taskInstructions": {
17+
"title": "Writing Task",
18+
"description": "You work as an event coordinator. Your colleague Sarah has messaged you about a scheduling conflict that needs to be resolved. You need to write an email to one of the panelists to address the situation.",
19+
"companyFraming": "You're representing the company in this communication. Consider how your message will reflect on the team."
20+
},
21+
"chat": {
22+
"initialMessages": [
23+
"Hey, remember that panel we scheduled with Jaden tomorrow?",
24+
"Turns out we double-booked the room! 😬 Sophia already posted that her panel is in room 12 at 1pm. She's more famous, we can't back out on her.",
25+
"Need you to send him an email sorting this out. Keep him happy, we can't afford to lose a client!"
26+
],
27+
"followUpMessage": "Let me know if you have any questions!",
28+
"systemPrompt": "You are Sarah Martinez, an Events Coordinator at a mid-sized company. You are currently dealing with a stressful room double-booking situation.\n\nSCENARIO CONTEXT:\n- Tomorrow there's a panel discussion with Jaden Thompson (a social media influencer)\n- The panel was originally scheduled for 1pm in Room 12\n- Room 12 was accidentally double-booked with Sophia Chen (a more famous influencer with 500K followers)\n- Sophia already publicly announced her panel at Room 12 at 1pm to her fans, so you can't move her\n- You need to move Jaden's panel to a different room/time\n- Room 14 is available, but the event before it ends at 1pm (so no setup time if scheduled at 1pm)\n- Room 14 would work fine at 1:30pm\n- Mike Chen handles facilities/room bookings\n- The user is a PR/communications person who needs to email Jaden about the change\n\nYOUR ROLE:\n- Answer questions about the facts of the situation\n- You're busy and stressed, typing quick messages on your phone\n- Keep responses SHORT - usually 1-2 sentences, sometimes just a few words\n- You can send multiple short messages in a row if that feels natural\n- You CANNOT and WILL NOT write the email for them or tell them exactly what to say - that's their job\n- You can give them facts, but not draft communications\n- If asked to write/draft anything, politely refuse (you're too busy, or it's their expertise)\n- You can make up reasonable details if needed, but keep them consistent with the scenario\n- Be natural and conversational, use occasional emoji when appropriate\n- Sometimes you might need to check with Mike or look something up - you can say you'll get back to them\n\nRESPONSE FORMAT:\nRespond with a JSON array of messages. Each message is a string. If you want to send multiple messages in quick succession (like someone texting), put them in separate array elements.\n\nExample: [\"1pm same room 😅\", \"can you email him?\"]\nOr: [\"Room 14 is free\", \"but the event before ends at 1 so no setup time\"]\n\nJust return the JSON array, nothing else."
29+
},
30+
"analysis": {
31+
"context": "A marketing/events coordinator needs to email a panelist (Jaden Thompson, a social media influencer) about a room double-booking situation.\n\nKEY FACTS:\n- Jaden's panel was originally scheduled for 1pm in Room 12 tomorrow\n- Room 12 was accidentally double-booked with a more famous influencer (Sophia Chen)\n- Sophia already publicly announced her panel, so she can't be moved\n- Jaden's panel needs to be moved to Room 14 at 1:30pm\n- Room 14 would work fine at 1:30pm (event before ends at 1pm, so 1pm wouldn't allow setup time)\n- Mike Chen handles facilities/room bookings\n- The colleague (Sarah Martinez, Events Coordinator) provided these details via chat\n\nRECIPIENT CONTEXT:\n- Jaden Thompson is a paying client / social media influencer\n- He may be frustrated or upset by the last-minute change\n- The company wants to maintain good relations and not lose him as a client",
32+
"keyFacts": [
33+
"The panel is being moved (not cancelled)",
34+
"New location: Room 14",
35+
"New time: 1:30pm (or acknowledgment of time change)",
36+
"Reason relates to scheduling conflict / double-booking"
37+
]
38+
}
39+
},
40+
"demoRescheduling": {
41+
"id": "demoRescheduling",
42+
"sender": {
43+
"name": "Alex Johnson"
44+
},
45+
"colleague": {
46+
"name": "Marcus Chen",
47+
"firstName": "Marcus",
48+
"role": "Solutions Engineer"
49+
},
50+
"recipient": {
51+
"name": "Dr. Lisa Patel",
52+
"email": "[email protected]"
53+
},
54+
"taskInstructions": {
55+
"title": "Writing Task",
56+
"description": "You work as a customer success manager. Your colleague Marcus has messaged you about a technical issue that requires rescheduling an important product demo. You need to write an email to the client to address the situation.",
57+
"companyFraming": "You're representing the company in this communication. Consider how your message will reflect on our professionalism and reliability."
58+
},
59+
"chat": {
60+
"initialMessages": [
61+
"Hey, we have a problem with tomorrow's MediCore demo 😓",
62+
"Found a critical bug in the reporting module this morning. Can't show it like this to a VP.",
63+
"Can you email Dr. Patel and reschedule? Need to keep her confident in us."
64+
],
65+
"followUpMessage": "Let me know if you need any details!",
66+
"systemPrompt": "You are Marcus Chen, a Solutions Engineer at a B2B SaaS company. You've discovered a critical bug right before an important product demo.\n\nSCENARIO CONTEXT:\n- Tomorrow (Tuesday) at 2pm you have a scheduled product demo with Dr. Lisa Patel, VP of IT at MediCore Health (a potential major client)\n- This morning you discovered a critical bug in the reporting module that causes incorrect data aggregation\n- The bug makes the product look unreliable and unprofessional - you absolutely cannot demo it in this state\n- Your engineering team needs 3-4 business days to fix and test it properly\n- Thursday afternoon and Friday morning next week are your available slots (you can check your calendar for exact times if asked)\n- This is the second meeting with MediCore - the first was an intro call last week where Dr. Patel expressed strong interest\n- The user is a customer success manager who handles client communications\n- Dr. Patel seems professional but busy - she mentioned having a tight timeline for vendor selection\n\nYOUR ROLE:\n- Answer questions about the technical issue and rescheduling options\n- You're concerned about maintaining client confidence but honest about technical issues\n- Keep responses SHORT - usually 1-2 sentences, sometimes just a few words\n- You can send multiple short messages in a row if that feels natural\n- You CANNOT and WILL NOT write the email for them or tell them exactly what to say - that's their job\n- You can give them facts about the bug, timeline, and available slots, but not draft communications\n- If asked to write/draft anything, politely refuse (it's their expertise in client relations)\n- You can make up reasonable technical details if needed, but keep them consistent\n- Be natural and conversational, use occasional emoji when appropriate\n- You might need to double-check your calendar or with engineering - you can say you'll get back to them\n\nRESPONSE FORMAT:\nRespond with a JSON array of messages. Each message is a string. If you want to send multiple messages in quick succession (like someone texting), put them in separate array elements.\n\nExample: [\"data aggregation bug\", \"makes us look bad\"]\nOr: [\"I have Thursday 2pm free\", \"or Friday morning\"]\n\nJust return the JSON array, nothing else."
67+
},
68+
"analysis": {
69+
"context": "A customer success manager needs to email a potential client (Dr. Lisa Patel, VP of IT at MediCore Health) about rescheduling a product demo due to a critical bug.\n\nKEY FACTS:\n- Demo was scheduled for Tuesday at 2pm\n- A critical bug was discovered in the reporting module (incorrect data aggregation)\n- Engineering needs 3-4 business days to fix and test\n- Available reschedule times: Thursday afternoon or Friday morning next week\n- This is a second meeting with MediCore - first was an intro call where she expressed strong interest\n- Dr. Patel has a tight timeline for vendor selection\n- The colleague (Marcus Chen, Solutions Engineer) provided these details via chat\n\nRECIPIENT CONTEXT:\n- Dr. Lisa Patel is VP of IT at a potential major client\n- She's busy and has mentioned tight vendor selection timeline\n- The company wants to maintain her confidence despite the delay",
70+
"keyFacts": [
71+
"The demo is being rescheduled (not cancelled)",
72+
"New time proposed (Thursday or Friday next week)",
73+
"Reason given (technical issue, ensuring quality)",
74+
"Commitment to follow through"
75+
]
76+
}
77+
}
78+
}

experiment/lib/studyConfig.ts

Lines changed: 16 additions & 122 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import type { ConditionCode, ConditionName } from '@/types/study';
2+
import scenariosData from './scenarios.json';
23

34
// Study wave identifier
45
export const WAVE = 'pilot-2';
@@ -75,128 +76,21 @@ export interface ScenarioConfig {
7576
};
7677
}
7778

78-
// Available scenarios
79-
export const SCENARIOS: Record<string, ScenarioConfig> = {
80-
roomDoubleBooking: {
81-
id: 'roomDoubleBooking',
82-
colleague: {
83-
name: 'Sarah Martinez',
84-
firstName: 'Sarah',
85-
role: 'Events Coordinator',
86-
},
87-
sender: {
88-
name: "Alex Johnson",
89-
},
90-
recipient: {
91-
name: 'Jaden Thompson',
92-
93-
},
94-
taskInstructions: {
95-
title: 'Writing Task',
96-
description: 'You work as an event coordinator. Your colleague Sarah has messaged you about a scheduling conflict that needs to be resolved. You need to write an email to one of the panelists to address the situation.',
97-
companyFraming: "You're representing the company in this communication. Consider how your message will reflect on the team.",
98-
},
99-
chat: {
100-
initialMessages: [
101-
"Hey, remember that panel we scheduled with Jaden tomorrow?",
102-
"Turns out we double-booked the room! 😬 Sophia already posted that her panel is in room 12 at 1pm. She's more famous, we can't back out on her.",
103-
"Need you to send him an email sorting this out. Keep him happy, we can't afford to lose a client!"
104-
],
105-
followUpMessage: "Let me know if you have any questions!",
106-
systemPrompt: `You are Sarah Martinez, an Events Coordinator at a mid-sized company. You are currently dealing with a stressful room double-booking situation.
107-
108-
SCENARIO CONTEXT:
109-
- Tomorrow there's a panel discussion with Jaden Thompson (a social media influencer)
110-
- The panel was originally scheduled for 1pm in Room 12
111-
- Room 12 was accidentally double-booked with Sophia Chen (a more famous influencer with 500K followers)
112-
- Sophia already publicly announced her panel at Room 12 at 1pm to her fans, so you can't move her
113-
- You need to move Jaden's panel to a different room/time
114-
- Room 14 is available, but the event before it ends at 1pm (so no setup time if scheduled at 1pm)
115-
- Room 14 would work fine at 1:30pm
116-
- Mike Chen handles facilities/room bookings
117-
- The user is a PR/communications person who needs to email Jaden about the change
118-
119-
YOUR ROLE:
120-
- Answer questions about the facts of the situation
121-
- You're busy and stressed, typing quick messages on your phone
122-
- Keep responses SHORT - usually 1-2 sentences, sometimes just a few words
123-
- You can send multiple short messages in a row if that feels natural
124-
- You CANNOT and WILL NOT write the email for them or tell them exactly what to say - that's their job
125-
- You can give them facts, but not draft communications
126-
- If asked to write/draft anything, politely refuse (you're too busy, or it's their expertise)
127-
- You can make up reasonable details if needed, but keep them consistent with the scenario
128-
- Be natural and conversational, use occasional emoji when appropriate
129-
- Sometimes you might need to check with Mike or look something up - you can say you'll get back to them
130-
131-
RESPONSE FORMAT:
132-
Respond with a JSON array of messages. Each message is a string. If you want to send multiple messages in quick succession (like someone texting), put them in separate array elements.
133-
134-
Example: ["1pm same room 😅", "can you email him?"]
135-
Or: ["Room 14 is free", "but the event before ends at 1 so no setup time"]
136-
137-
Just return the JSON array, nothing else.`,
138-
},
139-
},
140-
demoRescheduling: {
141-
id: 'demoRescheduling',
142-
colleague: {
143-
name: 'Marcus Chen',
144-
firstName: 'Marcus',
145-
role: 'Solutions Engineer',
146-
},
147-
sender: {
148-
name: "Alex Johnson",
149-
},
150-
recipient: {
151-
name: 'Dr. Lisa Patel',
152-
153-
},
154-
taskInstructions: {
155-
title: 'Writing Task',
156-
description: 'You work as a customer success manager. Your colleague Marcus has messaged you about a technical issue that requires rescheduling an important product demo. You need to write an email to the client to address the situation.',
157-
companyFraming: "You're representing the company in this communication. Consider how your message will reflect on our professionalism and reliability.",
158-
},
159-
chat: {
160-
initialMessages: [
161-
"Hey, we have a problem with tomorrow's MediCore demo 😓",
162-
"Found a critical bug in the reporting module this morning. Can't show it like this to a VP.",
163-
"Can you email Dr. Patel and reschedule? Need to keep her confident in us."
164-
],
165-
followUpMessage: "Let me know if you need any details!",
166-
systemPrompt: `You are Marcus Chen, a Solutions Engineer at a B2B SaaS company. You've discovered a critical bug right before an important product demo.
167-
168-
SCENARIO CONTEXT:
169-
- Tomorrow (Tuesday) at 2pm you have a scheduled product demo with Dr. Lisa Patel, VP of IT at MediCore Health (a potential major client)
170-
- This morning you discovered a critical bug in the reporting module that causes incorrect data aggregation
171-
- The bug makes the product look unreliable and unprofessional - you absolutely cannot demo it in this state
172-
- Your engineering team needs 3-4 business days to fix and test it properly
173-
- Thursday afternoon and Friday morning next week are your available slots (you can check your calendar for exact times if asked)
174-
- This is the second meeting with MediCore - the first was an intro call last week where Dr. Patel expressed strong interest
175-
- The user is a customer success manager who handles client communications
176-
- Dr. Patel seems professional but busy - she mentioned having a tight timeline for vendor selection
177-
178-
YOUR ROLE:
179-
- Answer questions about the technical issue and rescheduling options
180-
- You're concerned about maintaining client confidence but honest about technical issues
181-
- Keep responses SHORT - usually 1-2 sentences, sometimes just a few words
182-
- You can send multiple short messages in a row if that feels natural
183-
- You CANNOT and WILL NOT write the email for them or tell them exactly what to say - that's their job
184-
- You can give them facts about the bug, timeline, and available slots, but not draft communications
185-
- If asked to write/draft anything, politely refuse (it's their expertise in client relations)
186-
- You can make up reasonable technical details if needed, but keep them consistent
187-
- Be natural and conversational, use occasional emoji when appropriate
188-
- You might need to double-check your calendar or with engineering - you can say you'll get back to them
189-
190-
RESPONSE FORMAT:
191-
Respond with a JSON array of messages. Each message is a string. If you want to send multiple messages in quick succession (like someone texting), put them in separate array elements.
192-
193-
Example: ["data aggregation bug", "makes us look bad"]
194-
Or: ["I have Thursday 2pm free", "or Friday morning"]
195-
196-
Just return the JSON array, nothing else.`,
197-
},
198-
},
199-
};
79+
// Available scenarios (imported from JSON, cast to correct type)
80+
// The JSON includes an 'analysis' field for Python scripts that we exclude from the runtime type
81+
export const SCENARIOS: Record<string, ScenarioConfig> = Object.fromEntries(
82+
Object.entries(scenariosData).map(([key, value]) => [
83+
key,
84+
{
85+
id: value.id,
86+
sender: value.sender,
87+
colleague: value.colleague,
88+
recipient: value.recipient,
89+
taskInstructions: value.taskInstructions,
90+
chat: value.chat,
91+
} as ScenarioConfig,
92+
])
93+
);
20094

20195
// Default scenario
20296
export const DEFAULT_SCENARIO_ID = 'roomDoubleBooking';

0 commit comments

Comments
 (0)