|
| 1 | +--- |
| 2 | +title: Voice AI Testing |
| 3 | +subtitle: End-to-end test automation for AI voice agents |
| 4 | +slug: /test/voice-testing |
| 5 | +--- |
| 6 | + |
| 7 | +## Overview |
| 8 | + |
| 9 | +Voice Testing is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates a call from an AI tester that interacts with your voice agent by following a pre-defined call script. After the call, the transcript is sent to a language model (LLM) along with your success criteria. The LLM then determines if the call met the defined objectives. |
| 10 | + |
| 11 | +## Creating a Test Suite |
| 12 | + |
| 13 | +Begin by creating a Test Suite that organizes and executes multiple test cases. |
| 14 | + |
| 15 | +<Steps> |
| 16 | + ### Step 1: Create a New Test Suite |
| 17 | + - Navigate to the **Test** tab in your dashboard and select **Voice Testing**. |
| 18 | + - Click the **Create Test Suite** button. |
| 19 | + |
| 20 | + ### Step 2: Define Test Suite Details |
| 21 | + - Enter a title for your Test Suite. |
| 22 | + - Select a phone number from your organization using the dropdown. |
| 23 | + |
| 24 | + ### Step 3: Add Test Cases |
| 25 | + - Once your Test Suite is created, you will see a table where you can add test cases. |
| 26 | + - Click **Add Test Case** to add a new test case (up to 50 can be added). |
| 27 | + |
| 28 | + ### Step 4: Configure Each Test Case |
| 29 | + - **Caller Behavior:** Define how the testing agent should behave, including a detailed multi-step prompt that outlines the customer's intent, emotions, and interaction style. |
| 30 | + - **Success Criteria:** List one or more questions that an LLM will use to evaluate if the call was successful. |
| 31 | + - **Attempts:** Choose the number of times (up to 5) the test case should be executed each time the Test Suite is run. |
| 32 | + |
| 33 | + ### Step 5: Run and Review Tests |
| 34 | + - Click **Run Tests** to execute all test cases one by one. |
| 35 | + - While tests are running, you will see a loading state. |
| 36 | + - Upon completion, a table displays the outcomes with check marks (success) or x-marks (failure). |
| 37 | + - Click on a test row to view detailed results: a dropdown shows each attempt, the LLM's reasoning, the transcript of the call, the defined caller behavior, and the success criteria. |
| 38 | + |
| 39 | + ### Step 6: Export Results |
| 40 | + - You can export the test results as a CSV file for further analysis. |
| 41 | +</Steps> |
| 42 | + |
| 43 | +## Test Execution and Evaluation |
| 44 | + |
| 45 | +When you run a Test Suite, the following steps occur: |
| 46 | + |
| 47 | +- **Call Simulation:** An AI voice agent dials your voice agent, executing the pre-defined script. |
| 48 | +- **Transcript Capture:** The entire conversation is transcribed, capturing both the caller's behavior and your voice agent's responses. |
| 49 | +- **Automated Evaluation:** The transcript, along with your Success Criteria, is processed by an LLM to determine if the call was successful. |
| 50 | +- **Results Display:** Each test case outcome is shown with details. Clicking on a test case reveals: |
| 51 | + - The number of attempts made. |
| 52 | + - The LLM's reasoning for each attempt. |
| 53 | + - The complete call transcript. |
| 54 | + - The configured caller behavior and success criteria. |
| 55 | +- **CSV Export:** Results can be exported for additional review or compliance purposes. |
| 56 | + |
| 57 | +## Example Test Cases |
| 58 | + |
| 59 | +Below are three example test cases to illustrate how you can configure detailed caller behavior and success criteria. |
| 60 | + |
| 61 | +### Example 1: Account Inquiry |
| 62 | + |
| 63 | +**Caller Behavior:** |
| 64 | +Simulate a customer inquiring about their account status with growing concern as unexplained charges appear in their statement. |
| 65 | + |
| 66 | +**Example Prompt:** |
| 67 | +```md title="Example Prompt" wordWrap |
| 68 | +[Identity] |
| 69 | +You are a long-time bank customer with a keen eye for your financial details. |
| 70 | + |
| 71 | +[Personality] |
| 72 | +Normally calm and polite, you become increasingly anxious when you notice discrepancies on your account statement. Your tone shifts from supportive to urgent as the conversation progresses. |
| 73 | + |
| 74 | +[Goals] |
| 75 | +Your primary objective is to clarify several unexplained charges by requesting a detailed breakdown of your recent transactions and ensuring your account balance is accurate. |
| 76 | + |
| 77 | +[Interaction Style] |
| 78 | +Begin the call by stating your name and expressing concern over unexpected charges. Ask straightforward questions and press for more details if the explanation is not satisfactory. |
| 79 | +``` |
| 80 | + |
| 81 | +**Success Criteria:** |
| 82 | +```md title="Success Criteria" wordWrap |
| 83 | +1. The voice agent clearly presents the current account balance. |
| 84 | +2. The voice agent provides a detailed breakdown of recent transactions. |
| 85 | +3. The response addresses the customer's concerns in a calm and informative manner. |
| 86 | +``` |
| 87 | + |
| 88 | +### Example 2: Billing Support |
| 89 | + |
| 90 | +**Caller Behavior:** |
| 91 | +Simulate a customer who is frustrated and calling about a billing discrepancy. |
| 92 | + |
| 93 | +**Example Prompt:** |
| 94 | +```txt title="Example Prompt" wordWrap |
| 95 | +[Identity] |
| 96 | +You are a loyal customer who has always trusted the billing process, but the current bill appears unusually high. |
| 97 | +
|
| 98 | +[Personality] |
| 99 | +Frustrated and assertive, you express anger over an unexpected charge while remaining focused on obtaining clarification. |
| 100 | +
|
| 101 | +[Goals] |
| 102 | +Your goal is to understand the discrepancy in your bill by obtaining a detailed explanation, confirming whether an overcharge occurred, and understanding the steps for resolution. |
| 103 | +
|
| 104 | +[Interaction Style] |
| 105 | +Start the call by clearly stating your billing concern, describing the specific overcharge, and requesting a comprehensive explanation with resolution options. |
| 106 | +``` |
| 107 | + |
| 108 | +**Success Criteria:** |
| 109 | +```md title="Success Criteria" wordWrap |
| 110 | +1. The voice agent acknowledges the billing discrepancy respectfully without dismissing the concern. |
| 111 | +2. The agent provides a clear explanation of the charges, detailing possible reasons for the discrepancy. |
| 112 | +3. The conversation concludes with a proposed solution or a clear escalation plan to address the overcharge. |
| 113 | +``` |
| 114 | + |
| 115 | +### Example 3: Appointment Scheduling |
| 116 | + |
| 117 | +**Caller Behavior:** |
| 118 | +Simulate a customer trying to schedule an appointment with a hint of urgency due to previous delays. |
| 119 | + |
| 120 | +**Example Prompt:** |
| 121 | +```md title="Example Prompt" wordWrap |
| 122 | +[Identity] |
| 123 | +You are an organized customer who values efficiency and punctuality. |
| 124 | + |
| 125 | +[Personality] |
| 126 | +While generally courteous and friendly, you are anxious due to previous delays in scheduling appointments, and your tone conveys urgency. |
| 127 | + |
| 128 | +[Goals] |
| 129 | +Your goal is to secure an appointment at your preferred time, while remaining flexible enough to consider alternative timings if your desired slot is unavailable. |
| 130 | + |
| 131 | +[Interaction Style] |
| 132 | +Begin the call by stating your need for an appointment, specifying a preferred date and time (e.g., next Monday at 3 PM). Request clear confirmation of your slot, and if unavailable, ask for suitable alternatives. |
| 133 | +``` |
| 134 | + |
| 135 | +**Success Criteria:** |
| 136 | +```md title="Success Criteria" wordWrap |
| 137 | +1. The voice agent confirms the requested appointment time clearly and accurately. |
| 138 | +2. The agent reiterates the appointment details to ensure clarity. |
| 139 | +3. The scheduling process ends with a definitive confirmation message of the booked appointment. |
| 140 | +``` |
0 commit comments