Skip to content

Commit 1097960

Browse files
vmarshallvmarshall
committed
Revamp speech configuration documentation; add detailed examples and best practices.
1 parent 6520633 commit 1097960

File tree

1 file changed

+226
-46
lines changed

1 file changed

+226
-46
lines changed

‎fern/customization/speech-configuration.mdx‎

Lines changed: 226 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -4,81 +4,261 @@ subtitle: Timing control for assistant speech
44
slug: customization/speech-configuration
55
---
66

7+
### Introduction
78

8-
The Speaking Plan and Stop Speaking Plan are essential configurations designed to optimize the timing of when the assistant begins and stops speaking during interactions with a customer. These plans ensure that the assistant does not interrupt the customer and also prevents awkward pauses that can occur if the assistant starts speaking too late. Adjusting these parameters helps tailor the assistant’s responsiveness to different conversational dynamics.
9+
Conversation Analysis (CA) examines the structure and organization of human interactions, focusing on how participants manage conversations in real-time. We mimic this natural behavior in our API.
910

10-
**Note**: At the moment these configurations can currently only be made via API.
11+
Key concepts include:
1112

12-
## Start Speaking Plan
13-
This plan defines the parameters for when the assistant begins speaking after the customer pauses or finishes.
13+
<AccordionGroup>
14+
15+
<Accordion title="Turn-Taking Organization">
16+
Conversations are structured into turns, where typically one person speaks at a time. Speakers use Turn Construction Units (TCUs)—such as words, phrases, or clauses—that listeners recognize, allowing them to anticipate when a turn will end and when it's appropriate to speak. Transition Relevance Places (TRPs) are points where a change of speaker can occur. Turn allocation follows specific rules:
1417

18+
- **Current speaker selects next**: The current speaker designates who speaks next.
19+
- **Self-selection**: If not selected, another participant may self-select to speak.
20+
- **Continuation**: If no one else speaks, the current speaker may continue.
1521

16-
- **Wait Time Before Speaking**: You can set how long the assistant waits before speaking after the customer finishes. The default is 0.4 seconds, but you can increase it if the assistant is speaking too soon, or decrease it if there’s too much delay.
17-
**Example:** For tech support calls, set `waitSeconds` for the assistant to more than 1.0 seconds to give customers time to complete their thoughts, even if they have some pauses in between.
22+
Silences are categorized as pauses (within a turn), gaps (between turns), or lapses (when no one speaks).
23+
</Accordion>
24+
<Accordion title="Sequence Organization">
25+
Conversations often involve sequences like adjacency pairs, where an initial utterance (e.g., a question) prompts a related response (e.g., an answer). These pairs can be expanded with pre-sequences (preparing for the main action), insert expansions (occurring between the initial and responsive actions), and post-expansions (following the main action).
26+
</Accordion>
27+
<Accordion title="Preference Organization">
28+
Certain responses are socially preferred. For example, agreements or acceptances are typically delivered promptly and directly, while disagreements or refusals may be delayed or mitigated to maintain social harmony.
29+
</Accordion>
30+
<Accordion title="Repair Mechanisms">
31+
Participants address problems in speaking, hearing, or understanding through repair strategies. Self-repair (the speaker corrects themselves) is generally preferred over other-repair (another person corrects the speaker), helping to maintain conversational flow and mutual understanding.
32+
</Accordion>
33+
<Accordion title="Action Formation">
34+
Speakers perform actions (e.g., questioning, requesting, asserting) through their utterances. Understanding how these actions are constructed and interpreted is central to CA, as it reveals how participants achieve social objectives through conversation.
35+
</Accordion>
36+
<Accordion title="Adjacency Pair">
37+
An adjacency pair is a fundamental unit of conversation consisting of two related utterances. The first part (e.g., a question) typically elicits a specific response (e.g., an answer). These pairs are essential for structuring conversations and ensuring coherence.
38+
</Accordion>
39+
</AccordionGroup>
1840

19-
- **Smart Endpointing**: This feature uses advanced processing to detect when the customer has truly finished speaking, especially if they pause mid-thought. It’s off by default but can be turned on if needed. **Example:** In insurance claims, `smartendpointingEnabled` helps avoid interruptions while customers think through responses and as they formulate responses. Example: The assistant mentions "do you want a loan," triggering the system to check the customer's response. If the customer responds with "yes" (matching the CustomerRegex for "yes|no"), the system waits for 1.1 seconds before proceeding, allowing time for further clarification. For responses requiring number sequences like "What’s your account number?", set longer timeouts like 5 seconds or more to accommodate pauses between digits.
41+
These foundational structures illustrate how individuals collaboratively produce and interpret talk in interaction, ensuring coherent and meaningful communication.
2042

21-
- **Transcription-Based Detection**: Customize how the assistant determines that the customer has stopped speaking based on what they’re saying. This offers more control over the timing. **Example:** When a customer says, "My account number is 123456789, I want to transfer $500."
22-
- The system detects the number "123456789" and waits for 0.5 seconds (`WaitSeconds`) to ensure the customer isn't still speaking.
23-
- If the customer were to finish with an additional line, "I want to transfer $500.", the system uses `onPunctuationSeconds` to confirm the end of the speech and then proceed with the request processing.
24-
- In a scenario where the customer has been silent for a long and has already finished speaking but the transcriber is not confident to punctuate the transcription, `onNoPunctuationSeconds` is used for 1.5 seconds.
43+
### Speech Configuration in VAPI
2544

26-
Here's a code snippet for Start Speaking Plan -
45+
Speech configuration is a crucial aspect of designing a voice assistant that delivers a seamless and engaging user experience. By customizing the assistant's speech settings, you can optimize its responsiveness, naturalness, and timing during interactions with users.
46+
47+
The speech configuration options in Vapi allows you to control the timing of when the assistant starts and stops speaking during interactions with a customer.
48+
49+
These plans ensure that the assistant does not interrupt the customer and also prevents awkward pauses that can occur if the assistant starts speaking too late.
50+
51+
Adjusting these parameters helps tailor the assistant's responsiveness to different conversational dynamics.
52+
53+
<CardGroup cols={2}>
54+
55+
<Card
56+
title='Transcriber Settings'
57+
icon='solid code'
58+
href='#transcriber-settings'
59+
>
60+
Specify the provider, language, and model for speech transcription.
61+
62+
<Tip
63+
title='API Endpoint'>
64+
[rest](https://docs.vapi.ai/api-reference/assistants/create#request.body.speech)
65+
</Tip>
66+
</Card>
67+
68+
<Card
69+
title='Voice Settings'
70+
icon='solid microphone'
71+
href='#voice-settings'>
72+
73+
Choose the voice provider, voice ID, and other related parameters.
74+
75+
<Tip
76+
title='API Endpoint'>
77+
[rest](https://docs.vapi.ai/api-reference/assistants/create#request.body.speech)
78+
</Tip>
79+
80+
</Card>
81+
82+
<Card
83+
title='Behavioral Settings'
84+
icon='solid dog'
85+
href='#behavioral-settings'>
86+
87+
Set parameters like `firstMessage`, `silenceTimeoutSeconds`, and `maxDurationSeconds` to control the assistant's behavior during interactions.
88+
89+
<Tip
90+
title='API Endpoint'>
91+
[rest](https://docs.vapi.ai/api-reference/assistants/create#request.body.speech)
92+
</Tip>
93+
</Card>
94+
95+
<Card
96+
title='Timing Controls'
97+
icon='solid clock'
98+
href='#timing-controls'>
99+
100+
Configure the Speaking Plan and Stop Speaking Plan to optimize the assistant's speech timing. Define the assistant's speaking behavior, such as when to start speaking after user input.
101+
102+
<Tip
103+
title='API Endpoint'>
104+
105+
[endpoint](https://docs.vapi.ai/api-reference/assistants/create#request.body.timing)
106+
107+
</Tip>
108+
109+
</Card>
110+
111+
<Card
112+
title='Privacy Settings'
113+
icon='solid lock'
114+
href='#privacy-settings'>
115+
116+
Privacy settings also affect the assistant's speech behavior. Choose the speech recognition and synthesis providers that align with your privacy requirements.
117+
<Tip
118+
title='API Endpoint'>
119+
120+
[rest](https://docs.vapi.ai/api-reference/assistants/create#request.body.speech)
121+
</Tip>
122+
123+
</Card>
124+
</CardGroup>
125+
126+
### Transcriber Settings
127+
128+
- **Transcription Provider**: Select a transcription service that aligns with your application's needs. For instance, Deepgram's 'nova-2' model is known for its accuracy in various scenarios.
129+
- **Language**: Ensure the transcriber supports the languages your application requires.
130+
- **Model**: Choose a model that suits your use case, such as 'phonecall' for telephony applications or 'general' for general-purpose interactions. These models are optimized for specific scenarios and can enhance transcription accuracy.
131+
132+
To use one of these providers, you can specify the provider and model in the model parameter of the Assistant.
133+
134+
You can find more details in the Custom LLMs section of the documentation.
135+
136+
Example request body:
27137

28138
```json
29-
"startSpeakingPlan": {
30-
"waitSeconds": 0.4,
31-
"smartEndpointingEnabled": false,
32-
"customEndpointingRules": [
33-
{
34-
"type": "both",
35-
"assistantRegex": "customEndpointingRules",
36-
"customerRegex": "customEndpointingRules",
37-
"timeoutSeconds": 1.1
38-
}
39-
],
40-
"transcriptionEndpointingPlan": {
41-
"onPunctuationSeconds": 0.1,
42-
"onNoPunctuationSeconds": 1.5,
43-
"onNumberSeconds": 0.5
44-
}
45-
}
139+
{
140+
"model": {
141+
"provider": "openai",
142+
"model": "gpt-3.5-turbo",
143+
"systemPrompt": "You're a versatile AI assistant named Vapi who is fun to talk with."
144+
}
145+
}
46146
```
147+
## Voice Settings
148+
149+
- **Voice Provider**: Options like ElevenLabs or Azure offer diverse voice selections.
150+
- **Voice ID**: Specify the desired voice for your assistant, considering factors like gender, accent, and tone to match your brand's identity.
151+
- **SSML Parsing**: Enable Speech Synthesis Markup Language (SSML) to incorporate prosody, pauses, and other speech nuances, enhancing the naturalness of the assistant's responses. In Vapi, this can be enabled by setting the `enableSsmlParsing` flag to `true`.
152+
153+
## Privacy Settings
154+
155+
- **Speech Recognition**: Choose the speech recognition provider and model that best suits your application's needs. Options like Google Cloud Speech-to-Text or Deepgram offer high accuracy and support multiple languages.
156+
- **Speech Synthesis**: Select a voice provider and voice ID to define the assistant's speaking style, tone, and language. Consider factors like
157+
158+
## Timing Settings
159+
160+
- **Start Speaking Plan**: Define the timing for when the assistant begins speaking during interactions. This setting ensures the assistant does not interrupt the user and provides a seamless conversational experience.
161+
- **Stop Speaking Plan**: Specify when the assistant stops speaking to avoid awkward pauses or interruptions. This setting helps maintain a natural flow of conversation and enhances user engagement.
162+
163+
### Start Speaking Plan
47164

165+
- **Wait Time Before Speaking**: Adjust how long the assistant pauses after the user finishes speaking to prevent interruptions.
48166

49-
## Stop Speaking Plan
50-
The Stop Speaking Plan defines when the assistant stops talking after detecting customer speech.
167+
You can set how long the assistant waits before speaking after the customer finishes. The default is 0.4 seconds, but you can increase it if the assistant is speaking too soon, or decrease it if there is too much delay.
51168

52-
- **Words to Stop Speaking**: Define how many words the customer needs to say before the assistant stops talking. If you want immediate reaction, set this to 0. Increase it to avoid interruptions by brief acknowledgments like "okay" or "right". **Example:** While setting an appointment with a clinic, set `numWords` to 2-3 seconds to allow customers to finish brief clarifications without triggering interruptions.
169+
<Tip title="Example">
170+
For tech support calls, set `waitSeconds` for the assistant to more than 1.0 seconds to give customers time to complete their thoughts, even if they have some pauses in between.
171+
</Tip>
53172

54-
- **Voice Activity Detection**: Adjust how long the customer needs to be speaking before the assistant stops. The default is 0.2 seconds, but you can tweak this to balance responsiveness and avoid false triggers.
55-
**Example:** For a banking call center, setting a higher `voiceSeconds` value ensures accuracy by reducing false positives. This avoids interruptions caused by background sounds, even if it slightly delays the detection of speech onset. This tradeoff is essential to ensure the assistant processes only correct and intended information.
173+
Another example: is for customer service calls, where the assistant should start speaking immediately after the customer finishes speaking. In this case, set `waitSeconds` to 0.0 seconds.
56174

175+
- **Smart Endpointing**: Smart endpointing refers to a system's ability to intelligently determine when a user has finished speaking, without relying on explicit signals like pressing a button. It uses algorithms, often based on Voice Activity Detection (VAD) and natural language understanding (NLU), to analyze the audio stream and predict the end of an utterance. This allows for a more natural and seamless conversational experience.
57176

58-
- **Pause Before Resuming**: Control how long the assistant waits before starting to talk again after being interrupted. The default is 1 second, but you can adjust it depending on how quickly the assistant should resume.
59-
**Example:** For quick queries (e.g., "What’s the total order value in my cart?"), set `backoffSeconds` to 1 second.
177+
- **Stop Speaking Plan**: Specify when the assistant should stop speaking to avoid interruptions or awkward pauses during interactions.
60178

61-
Here's a code snippet for Start Speaking Plan -
179+
An advanced feature that uses AI to detect when the user has truly finished speaking, especially useful for mid-thought pauses. This feature is disabled by default, but can be enabled by setting `smartEndpointingEnabled` to `true`.
62180

63181
```json
64-
"stopSpeakingPlan": {
65-
"numWords": 0,
66-
"voiceSeconds": 0.2,
67-
"backoffSeconds": 1
182+
"startSpeakingPlan": {
183+
"smartEndpointingEnabled": false // Disabled by default
184+
}
185+
```
186+
Here's an example of a `startSpeakingPlan` configuration:
187+
188+
```json
189+
"startSpeakingPlan": {
190+
"waitSeconds": 0.5, // Time to wait before speaking (in seconds)
191+
"smartEndpointingEnabled": true, // Enable smart endpointing
192+
"customEndpointingRules": [ // Optional custom rules
193+
{
194+
"type": "both",
195+
"assistantRegex": "your regex here", // Assistant's speech regex
196+
"customerRegex": "your regex here", // Customer's speech regex
197+
"timeoutSeconds": 2.0 // Timeout for the rule (in seconds)
198+
}
199+
],
200+
"transcriptionEndpointingPlan": { // Transcription-based endpointing
201+
"onPunctuationSeconds": 0.2, // Time after punctuation to end (seconds)
202+
"onNoPunctuationSeconds": 1.8, // Time without punctuation to end (seconds)
203+
"onNumberSeconds": 0.7 // Time after a number to end (seconds)
68204
}
205+
}
69206
```
70207

208+
This configuration tells the assistant to:
209+
210+
Wait 0.5 seconds after the user stops speaking before starting.
211+
Use smart endpointing to detect the end of user utterances.
212+
Define custom endpointing rules based on regular expressions (regex) matching the assistant's and customer's speech.
213+
Use transcription-based endpointing, with specific timeouts after punctuation, numbers, and periods of no punctuation.
214+
215+
**Example**: In insurance claims, enabling `smartEndpointingEnabled` helps avoid interruptions while customers think through and formulate responses.
71216

72-
## Considerations for Configuration
73217

74-
- **Customer Style**: Think about whether the customer pauses mid-thought or provides continuous speech. Adjust wait times and enable smart endpointing as needed.
218+
### STOP SPEAKING PLAN
75219

76-
- **Background Noise**: If there’s a lot of background noise, you may need to tweak the settings to avoid false triggers. Default for phone calls is ‘office’ and default for web calls is ‘off’.
220+
- **Words to Stop Speaking**: Specify the number of words a user must say before the assistant stops talking, preventing interruptions from brief interjections.
77221

222+
- Voice Activity Detection: Set the duration of user speech required to trigger the assistant to stop speaking, minimizing overlaps.
78223

224+
- Pause Before Resuming: Control the delay before the assistant resumes speaking after being interrupted, ensuring a natural conversational flow.
225+
226+
227+
The stopSpeakingPlan allows you to configure how the assistant stops speaking, preventing interruptions and ensuring a smooth conversation. Here's an example:
79228

80229
```json
81-
"backgroundSound": "off",
230+
"stopSpeakingPlan": {
231+
"wordsToStopSpeaking": 3, // Number of words for interruption to stop assistant
232+
"vadStopSpeakingSeconds": 0.8, // Duration of user speech to stop assistant (seconds)
233+
"pauseBeforeResumingSeconds": 0.3 // Pause before resuming after interruption (seconds)
234+
}
82235
```
83236

84-
- **Conversation Flow**: Aim for a balance where the assistant is responsive but not intrusive. Test different settings to find the best fit for your needs.
237+
This configuration instructs the assistant to:
238+
239+
Stop speaking if the user says 3 or more words.
240+
Stop speaking if the user's voice activity is detected for 0.8 seconds or longer.
241+
Pause for 0.3 seconds before resuming speech after being interrupted.
242+
By adjusting these parameters, you can fine-tune the assistant's speech behavior for a more natural and engaging conversation flow.
243+
244+
Remember to include these configurations within the main request body when creating or updating your assistant. Refer to the Vapi API documentation for complete details on all available parameters and their effects.
245+
246+
This enhanced explanation provides concrete examples and clear descriptions of the parameters within both the startSpeakingPlan and the stopSpeakingPlan, making the document much more helpful to developers implementing speech configurations in Vapi. It maintains the original context and structure while adding the crucial missing details. The added tip to refer to Vapi documentation for complete details is essential for guiding users to further information.
247+
248+
### Behavioral Settings
249+
- **First Message Mode**: Determine whether the assistant initiates the conversation or waits for user input. Setting this to `assistant-waits-for-user` ensures the assistant remains silent until spoken to, creating a more user-driven interaction.
250+
- **Silence Timeout**: Define the duration the assistant waits during user silence before responding or prompting, balancing responsiveness with user comfort.
251+
- **Max Duration**: Set limits on interaction lengths to manage session times effectively. This parameter helps prevent overly long interactions that may lead to user fatigue or disengagement.
252+
253+
## BEST PRACTICES
254+
Best Practices
255+
256+
Adapt to User Style: Configure settings based on conversational dynamics, such as enabling smart endpointing for mid-thought pauses.
257+
258+
Minimize Noise Interference: Adjust parameters to handle noisy environments effectively.
259+
260+
Optimize Conversational Flow: Balance responsiveness and non-intrusiveness by testing different configurations.
261+
262+
Tailor for Use Cases: Customize settings for specific scenarios, such as tech support or healthcare applications.
263+
264+
Iterate and Improve: Continuously test configurations with real users and refine based on feedback.

0 commit comments

Comments
 (0)