diff --git a/fern/test/test-suites.mdx b/fern/test/test-suites.mdx index 71a7703fb..e5e0a9811 100644 --- a/fern/test/test-suites.mdx +++ b/fern/test/test-suites.mdx @@ -6,7 +6,7 @@ slug: /test/test-suites ## Overview -**Test Suite** is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates a call from an AI tester that interacts with your voice agent by following a pre-defined call script. After the call, the transcript is sent to a language model (LLM) along with your evaluation rubric. The LLM then determines if the call met the defined objectives. +**Test Suite** is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates an AI tester that interacts with your voice agent by following a pre-defined script. After the interaction, the transcript is sent to a language model (LLM) along with your evaluation rubric. The LLM then determines if the interaction met the defined objectives. ## Creating a Test Suite @@ -28,7 +28,8 @@ Begin by creating a **Test Suite** that organizes and executes multiple test cas ### Step 4: Configure Each Test Case - **Script:** Define how the testing agent should behave, including a detailed multi-step prompt to simulate how the customer should behave on the call. - - **Rubric:** List one or more questions that an LLM will use to evaluate if the call was successful. + - **Type:** Set the type of the test. 'Chat' simulates a text conversation, which we recommend because it is faster. 'Voice' simulates a call so you can hear a voice recording of the two assistants talking to each other. + - **Rubric:** List one or more questions that an LLM will use to evaluate if the interaction was successful. - **Attempts:** Choose the number of times (up to 5) the test case should be executed each time the **Test Suite** is run. ### Step 5: Run and Review Tests @@ -42,13 +43,13 @@ Begin by creating a **Test Suite** that organizes and executes multiple test cas When you run a **Test Suite**, the following steps occur: -- **Call Simulation:** An AI tester dials your voice agent, executing the pre-defined script. +- **Simulation:** An AI tester chats with or calls your voice agent, executing the pre-defined script. - **Transcript Capture:** The entire conversation is transcribed, capturing both the caller's behavior and your voice agent's responses. - **Automated Evaluation:** The transcript, along with your Success Criteria, is processed by an LLM to determine if the call was successful. - **Results Display:** Each test case outcome is shown with details. Clicking on a test case reveals: - The number of attempts made. - The LLM's reasoning for each attempt. - - The complete call transcript. + - The complete transcript. - The configured script and rubric. ## Example Test Cases