Skip to content

Commit f6329b8

Browse files
authored
add chat to test suite docs (#279)
1 parent f3cad9c commit f6329b8

File tree

1 file changed

+5
-4
lines changed

1 file changed

+5
-4
lines changed

fern/test/test-suites.mdx

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ slug: /test/test-suites
66

77
## Overview
88

9-
**Test Suite** is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates a call from an AI tester that interacts with your voice agent by following a pre-defined call script. After the call, the transcript is sent to a language model (LLM) along with your evaluation rubric. The LLM then determines if the call met the defined objectives.
9+
**Test Suite** is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates an AI tester that interacts with your voice agent by following a pre-defined script. After the interaction, the transcript is sent to a language model (LLM) along with your evaluation rubric. The LLM then determines if the interaction met the defined objectives.
1010

1111
## Creating a Test Suite
1212

@@ -28,7 +28,8 @@ Begin by creating a **Test Suite** that organizes and executes multiple test cas
2828

2929
### Step 4: Configure Each Test Case
3030
- **Script:** Define how the testing agent should behave, including a detailed multi-step prompt to simulate how the customer should behave on the call.
31-
- **Rubric:** List one or more questions that an LLM will use to evaluate if the call was successful.
31+
- **Type:** Set the type of the test. 'Chat' simulates a text conversation, which we recommend because it is faster. 'Voice' simulates a call so you can hear a voice recording of the two assistants talking to each other.
32+
- **Rubric:** List one or more questions that an LLM will use to evaluate if the interaction was successful.
3233
- **Attempts:** Choose the number of times (up to 5) the test case should be executed each time the **Test Suite** is run.
3334

3435
### Step 5: Run and Review Tests
@@ -42,13 +43,13 @@ Begin by creating a **Test Suite** that organizes and executes multiple test cas
4243

4344
When you run a **Test Suite**, the following steps occur:
4445

45-
- **Call Simulation:** An AI tester dials your voice agent, executing the pre-defined script.
46+
- **Simulation:** An AI tester chats with or calls your voice agent, executing the pre-defined script.
4647
- **Transcript Capture:** The entire conversation is transcribed, capturing both the caller's behavior and your voice agent's responses.
4748
- **Automated Evaluation:** The transcript, along with your Success Criteria, is processed by an LLM to determine if the call was successful.
4849
- **Results Display:** Each test case outcome is shown with details. Clicking on a test case reveals:
4950
- The number of attempts made.
5051
- The LLM's reasoning for each attempt.
51-
- The complete call transcript.
52+
- The complete transcript.
5253
- The configured script and rubric.
5354

5455
## Example Test Cases

0 commit comments

Comments
 (0)