You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: fern/test/test-suites.mdx
+5-4Lines changed: 5 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ slug: /test/test-suites
6
6
7
7
## Overview
8
8
9
-
**Test Suite** is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates a call from an AI tester that interacts with your voice agent by following a pre-defined call script. After the call, the transcript is sent to a language model (LLM) along with your evaluation rubric. The LLM then determines if the call met the defined objectives.
9
+
**Test Suite** is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates an AI tester that interacts with your voice agent by following a pre-defined script. After the interaction, the transcript is sent to a language model (LLM) along with your evaluation rubric. The LLM then determines if the interaction met the defined objectives.
10
10
11
11
## Creating a Test Suite
12
12
@@ -28,7 +28,8 @@ Begin by creating a **Test Suite** that organizes and executes multiple test cas
28
28
29
29
### Step 4: Configure Each Test Case
30
30
-**Script:** Define how the testing agent should behave, including a detailed multi-step prompt to simulate how the customer should behave on the call.
31
-
-**Rubric:** List one or more questions that an LLM will use to evaluate if the call was successful.
31
+
-**Type:** Set the type of the test. 'Chat' simulates a text conversation, which we recommend because it is faster. 'Voice' simulates a call so you can hear a voice recording of the two assistants talking to each other.
32
+
-**Rubric:** List one or more questions that an LLM will use to evaluate if the interaction was successful.
32
33
-**Attempts:** Choose the number of times (up to 5) the test case should be executed each time the **Test Suite** is run.
33
34
34
35
### Step 5: Run and Review Tests
@@ -42,13 +43,13 @@ Begin by creating a **Test Suite** that organizes and executes multiple test cas
42
43
43
44
When you run a **Test Suite**, the following steps occur:
44
45
45
-
-**Call Simulation:** An AI tester dials your voice agent, executing the pre-defined script.
46
+
-**Simulation:** An AI tester chats with or calls your voice agent, executing the pre-defined script.
46
47
-**Transcript Capture:** The entire conversation is transcribed, capturing both the caller's behavior and your voice agent's responses.
47
48
-**Automated Evaluation:** The transcript, along with your Success Criteria, is processed by an LLM to determine if the call was successful.
48
49
-**Results Display:** Each test case outcome is shown with details. Clicking on a test case reveals:
0 commit comments