Skip to content

add chat to test suite docs #279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 28, 2025
Merged

add chat to test suite docs #279

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions fern/test/test-suites.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ slug: /test/test-suites

## Overview

**Test Suite** is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates a call from an AI tester that interacts with your voice agent by following a pre-defined call script. After the call, the transcript is sent to a language model (LLM) along with your evaluation rubric. The LLM then determines if the call met the defined objectives.
**Test Suite** is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates an AI tester that interacts with your voice agent by following a pre-defined script. After the interaction, the transcript is sent to a language model (LLM) along with your evaluation rubric. The LLM then determines if the interaction met the defined objectives.

## Creating a Test Suite

Expand All @@ -28,7 +28,8 @@ Begin by creating a **Test Suite** that organizes and executes multiple test cas

### Step 4: Configure Each Test Case
- **Script:** Define how the testing agent should behave, including a detailed multi-step prompt to simulate how the customer should behave on the call.
- **Rubric:** List one or more questions that an LLM will use to evaluate if the call was successful.
- **Type:** Set the type of the test. 'Chat' simulates a text conversation, which we recommend because it is faster. 'Voice' simulates a call so you can hear a voice recording of the two assistants talking to each other.
- **Rubric:** List one or more questions that an LLM will use to evaluate if the interaction was successful.
- **Attempts:** Choose the number of times (up to 5) the test case should be executed each time the **Test Suite** is run.

### Step 5: Run and Review Tests
Expand All @@ -42,13 +43,13 @@ Begin by creating a **Test Suite** that organizes and executes multiple test cas

When you run a **Test Suite**, the following steps occur:

- **Call Simulation:** An AI tester dials your voice agent, executing the pre-defined script.
- **Simulation:** An AI tester chats with or calls your voice agent, executing the pre-defined script.
- **Transcript Capture:** The entire conversation is transcribed, capturing both the caller's behavior and your voice agent's responses.
- **Automated Evaluation:** The transcript, along with your Success Criteria, is processed by an LLM to determine if the call was successful.
- **Results Display:** Each test case outcome is shown with details. Clicking on a test case reveals:
- The number of attempts made.
- The LLM's reasoning for each attempt.
- The complete call transcript.
- The complete transcript.
- The configured script and rubric.

## Example Test Cases
Expand Down
Loading