Skip to content

feat: integrate Google Cloud and Azure Text-to-Speech services #6828

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

feat: integrate Google Cloud and Azure Text-to-Speech services #6828

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions packages/types/src/global-settings.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,12 @@ export const globalSettingsSchema = z.object({

ttsEnabled: z.boolean().optional(),
ttsSpeed: z.number().optional(),
ttsProvider: z.enum(["native", "google-cloud", "azure"]).optional(),
ttsVoice: z.string().optional(),
googleCloudTtsApiKey: z.string().optional(),
Copy link
Contributor Author

@roomote roomote bot Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security concern: These API keys should be stored using VSCode's SecretStorage API instead of global state. Storing sensitive credentials in global state could expose them through settings sync or exports.

Consider moving these to SecretStorage instead of adding them to the global settings schema.

googleCloudTtsProjectId: z.string().optional(),
azureTtsSubscriptionKey: z.string().optional(),
azureTtsRegion: z.string().optional(),
soundEnabled: z.boolean().optional(),
soundVolume: z.number().optional(),

Expand Down Expand Up @@ -255,6 +261,12 @@ export const EVALS_SETTINGS: RooCodeSettings = {

ttsEnabled: false,
ttsSpeed: 1,
ttsProvider: "native",
ttsVoice: undefined,
googleCloudTtsApiKey: undefined,
googleCloudTtsProjectId: undefined,
azureTtsSubscriptionKey: undefined,
azureTtsRegion: undefined,
soundEnabled: false,
soundVolume: 0.5,

Expand Down
392 changes: 377 additions & 15 deletions pnpm-lock.yaml
View file
Open in desktop

Large diffs are not rendered by default.

48 changes: 41 additions & 7 deletions src/core/webview/ClineProvider.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ import type { IndexProgressUpdate } from "../../services/code-index/interfaces/m
import { MdmService } from "../../services/mdm/MdmService"

import { fileExistsAtPath } from "../../utils/fs"
import { setTtsEnabled, setTtsSpeed } from "../../utils/tts"
import { setTtsEnabled, setTtsSpeed, initializeTts } from "../../utils/tts"
import { getWorkspaceGitInfo } from "../../utils/git"
import { getWorkspacePath } from "../../utils/path"

Expand Down Expand Up @@ -532,13 +532,29 @@ export class ClineProvider
},
)

// Initialize tts enabled state
this.getState().then(({ ttsEnabled }) => {
setTtsEnabled(ttsEnabled ?? false)
})
// Initialize TTS with configuration
this.getState().then(async (state) => {
const {
ttsEnabled,
ttsSpeed,
ttsProvider,
googleCloudTtsApiKey,
googleCloudTtsProjectId,
azureTtsSubscriptionKey,
azureTtsRegion,
} = state

// Initialize TTS manager with provider configuration
await initializeTts({
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the TTS initialization block, consider adding additional logging or error-handling around the initializeTts() call so that any configuration issues (e.g. missing API keys) are clearly logged.

provider: ttsProvider as "native" | "google-cloud" | "azure" | undefined,
googleCloudApiKey: googleCloudTtsApiKey,
googleCloudProjectId: googleCloudTtsProjectId,
azureSubscriptionKey: azureTtsSubscriptionKey,
azureRegion: azureTtsRegion,
})

// Initialize tts speed state
this.getState().then(({ ttsSpeed }) => {
// Set enabled state and speed
setTtsEnabled(ttsEnabled ?? false)
setTtsSpeed(ttsSpeed ?? 1)
})

Expand Down Expand Up @@ -1567,6 +1583,12 @@ export class ClineProvider
soundEnabled,
ttsEnabled,
ttsSpeed,
ttsProvider,
ttsVoice,
googleCloudTtsApiKey,
googleCloudTtsProjectId,
azureTtsSubscriptionKey,
azureTtsRegion,
diffEnabled,
enableCheckpoints,
taskHistory,
Expand Down Expand Up @@ -1671,6 +1693,12 @@ export class ClineProvider
soundEnabled: soundEnabled ?? false,
ttsEnabled: ttsEnabled ?? false,
ttsSpeed: ttsSpeed ?? 1.0,
ttsProvider: ttsProvider ?? "native",
ttsVoice: ttsVoice ?? undefined,
googleCloudTtsApiKey: googleCloudTtsApiKey ?? undefined,
googleCloudTtsProjectId: googleCloudTtsProjectId ?? undefined,
azureTtsSubscriptionKey: azureTtsSubscriptionKey ?? undefined,
azureTtsRegion: azureTtsRegion ?? undefined,
diffEnabled: diffEnabled ?? true,
enableCheckpoints: enableCheckpoints ?? true,
shouldShowAnnouncement:
Expand Down Expand Up @@ -1863,6 +1891,12 @@ export class ClineProvider
soundEnabled: stateValues.soundEnabled ?? false,
ttsEnabled: stateValues.ttsEnabled ?? false,
ttsSpeed: stateValues.ttsSpeed ?? 1.0,
ttsProvider: stateValues.ttsProvider ?? "native",
ttsVoice: stateValues.ttsVoice ?? undefined,
googleCloudTtsApiKey: stateValues.googleCloudTtsApiKey ?? undefined,
googleCloudTtsProjectId: stateValues.googleCloudTtsProjectId ?? undefined,
azureTtsSubscriptionKey: stateValues.azureTtsSubscriptionKey ?? undefined,
azureTtsRegion: stateValues.azureTtsRegion ?? undefined,
diffEnabled: stateValues.diffEnabled ?? true,
enableCheckpoints: stateValues.enableCheckpoints ?? true,
soundVolume: stateValues.soundVolume,
Expand Down
113 changes: 90 additions & 23 deletions src/core/webview/webviewMessageHandler.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ import { getTheme } from "../../integrations/theme/getTheme"
import { discoverChromeHostUrl, tryChromeHostUrl } from "../../services/browser/browserDiscovery"
import { searchWorkspaceFiles } from "../../services/search/file-search"
import { fileExistsAtPath } from "../../utils/fs"
import { playTts, setTtsEnabled, setTtsSpeed, stopTts } from "../../utils/tts"
import { playTts, setTtsEnabled, setTtsSpeed, stopTts, setTtsProvider, initializeTts } from "../../utils/tts"
import { searchCommits } from "../../utils/git"
import { exportSettings, importSettingsWithFeedback } from "../config/importExport"
import { getOpenAiModels } from "../../api/providers/openai"
Expand Down Expand Up @@ -330,11 +330,11 @@ export const webviewMessageHandler = async (
await provider.postStateToWebview()
break
case "allowedMaxRequests":
await updateGlobalState("allowedMaxRequests", message.value)
await updateGlobalState("allowedMaxRequests", Number(message.value))
await provider.postStateToWebview()
break
case "allowedMaxCost":
await updateGlobalState("allowedMaxCost", message.value)
await updateGlobalState("allowedMaxCost", Number(message.value))
await provider.postStateToWebview()
break
case "alwaysAllowSubtasks":
Expand All @@ -353,7 +353,7 @@ export const webviewMessageHandler = async (
await provider.postStateToWebview()
break
case "autoCondenseContextPercent":
await updateGlobalState("autoCondenseContextPercent", message.value)
await updateGlobalState("autoCondenseContextPercent", Number(message.value))
await provider.postStateToWebview()
break
case "terminalOperation":
Expand Down Expand Up @@ -936,21 +936,88 @@ export const webviewMessageHandler = async (
break
case "soundVolume":
const soundVolume = message.value ?? 0.5
await updateGlobalState("soundVolume", soundVolume)
await updateGlobalState("soundVolume", Number(soundVolume))
await provider.postStateToWebview()
break
case "ttsEnabled":
const ttsEnabled = message.bool ?? true
await updateGlobalState("ttsEnabled", ttsEnabled)
setTtsEnabled(ttsEnabled) // Add this line to update the tts utility
setTtsEnabled(ttsEnabled)
await provider.postStateToWebview()
break
case "ttsSpeed":
const ttsSpeed = message.value ?? 1.0
const ttsSpeed = Number(message.value ?? 1.0)
await updateGlobalState("ttsSpeed", ttsSpeed)
setTtsSpeed(ttsSpeed)
await provider.postStateToWebview()
break
case "ttsProvider":
const ttsProvider = String(message.value) as "native" | "google-cloud" | "azure"
await updateGlobalState("ttsProvider", ttsProvider)
await setTtsProvider(ttsProvider)
await provider.postStateToWebview()
break
case "ttsVoice":
const ttsVoice = String(message.value)
await updateGlobalState("ttsVoice", ttsVoice)
await provider.postStateToWebview()
break
case "googleCloudTtsApiKey":
const googleCloudApiKey = String(message.value)
await updateGlobalState("googleCloudTtsApiKey", googleCloudApiKey)
// Re-initialize TTS with new config
const gcState = await provider.getState()
await initializeTts({
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For new TTS configuration message handlers (e.g. for 'ttsProvider', 'googleCloudTtsApiKey', etc.), consider wrapping the re-initialization calls to initializeTts() in try/catch blocks. This will help prevent an unhandled error from crashing the handler and will allow a user‐friendly error message.

provider: gcState.ttsProvider as "native" | "google-cloud" | "azure" | undefined,
googleCloudApiKey: googleCloudApiKey,
googleCloudProjectId: gcState.googleCloudTtsProjectId,
azureSubscriptionKey: gcState.azureTtsSubscriptionKey,
azureRegion: gcState.azureTtsRegion,
})
await provider.postStateToWebview()
break
case "googleCloudTtsProjectId":
const googleCloudProjectId = String(message.value)
await updateGlobalState("googleCloudTtsProjectId", googleCloudProjectId)
// Re-initialize TTS with new config
const gcpState = await provider.getState()
await initializeTts({
provider: gcpState.ttsProvider as "native" | "google-cloud" | "azure" | undefined,
googleCloudApiKey: gcpState.googleCloudTtsApiKey,
googleCloudProjectId: googleCloudProjectId,
azureSubscriptionKey: gcpState.azureTtsSubscriptionKey,
azureRegion: gcpState.azureTtsRegion,
})
await provider.postStateToWebview()
break
case "azureTtsSubscriptionKey":
const azureSubscriptionKey = String(message.value)
await updateGlobalState("azureTtsSubscriptionKey", azureSubscriptionKey)
// Re-initialize TTS with new config
const azState = await provider.getState()
await initializeTts({
provider: azState.ttsProvider as "native" | "google-cloud" | "azure" | undefined,
googleCloudApiKey: azState.googleCloudTtsApiKey,
googleCloudProjectId: azState.googleCloudTtsProjectId,
azureSubscriptionKey: azureSubscriptionKey,
azureRegion: azState.azureTtsRegion,
})
await provider.postStateToWebview()
break
case "azureTtsRegion":
const azureRegion = String(message.value)
await updateGlobalState("azureTtsRegion", azureRegion)
// Re-initialize TTS with new config
const azrState = await provider.getState()
await initializeTts({
provider: azrState.ttsProvider as "native" | "google-cloud" | "azure" | undefined,
googleCloudApiKey: azrState.googleCloudTtsApiKey,
googleCloudProjectId: azrState.googleCloudTtsProjectId,
azureSubscriptionKey: azrState.azureTtsSubscriptionKey,
azureRegion: azureRegion,
})
await provider.postStateToWebview()
break
case "playTts":
if (message.text) {
playTts(message.text, {
Expand Down Expand Up @@ -1028,7 +1095,7 @@ export const webviewMessageHandler = async (
}
break
case "fuzzyMatchThreshold":
await updateGlobalState("fuzzyMatchThreshold", message.value)
await updateGlobalState("fuzzyMatchThreshold", Number(message.value))
await provider.postStateToWebview()
break
case "updateVSCodeSetting": {
Expand Down Expand Up @@ -1072,11 +1139,11 @@ export const webviewMessageHandler = async (
await provider.postStateToWebview()
break
case "requestDelaySeconds":
await updateGlobalState("requestDelaySeconds", message.value ?? 5)
await updateGlobalState("requestDelaySeconds", Number(message.value ?? 5))
await provider.postStateToWebview()
break
case "writeDelayMs":
await updateGlobalState("writeDelayMs", message.value)
await updateGlobalState("writeDelayMs", Number(message.value))
await provider.postStateToWebview()
break
case "diagnosticsEnabled":
Expand Down Expand Up @@ -1109,10 +1176,10 @@ export const webviewMessageHandler = async (
}
break
case "terminalShellIntegrationTimeout":
await updateGlobalState("terminalShellIntegrationTimeout", message.value)
await updateGlobalState("terminalShellIntegrationTimeout", Number(message.value))
await provider.postStateToWebview()
if (message.value !== undefined) {
Terminal.setShellIntegrationTimeout(message.value)
Terminal.setShellIntegrationTimeout(Number(message.value))
Copy link
Contributor Author

@roomote roomote bot Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Type coercion without validation could result in NaN values. Should we validate the input before converting to ensure we don't store invalid numbers?

}
break
case "terminalShellIntegrationDisabled":
Expand All @@ -1123,10 +1190,10 @@ export const webviewMessageHandler = async (
}
break
case "terminalCommandDelay":
await updateGlobalState("terminalCommandDelay", message.value)
await updateGlobalState("terminalCommandDelay", Number(message.value))
await provider.postStateToWebview()
if (message.value !== undefined) {
Terminal.setCommandDelay(message.value)
Terminal.setCommandDelay(Number(message.value))
}
break
case "terminalPowershellCounter":
Expand Down Expand Up @@ -1242,16 +1309,16 @@ export const webviewMessageHandler = async (
break
}
case "screenshotQuality":
await updateGlobalState("screenshotQuality", message.value)
await updateGlobalState("screenshotQuality", Number(message.value))
await provider.postStateToWebview()
break
case "maxOpenTabsContext":
const tabCount = Math.min(Math.max(0, message.value ?? 20), 500)
const tabCount = Math.min(Math.max(0, Number(message.value ?? 20)), 500)
await updateGlobalState("maxOpenTabsContext", tabCount)
await provider.postStateToWebview()
break
case "maxWorkspaceFiles":
const fileCount = Math.min(Math.max(0, message.value ?? 200), 500)
const fileCount = Math.min(Math.max(0, Number(message.value ?? 200)), 500)
await updateGlobalState("maxWorkspaceFiles", fileCount)
await provider.postStateToWebview()
break
Expand All @@ -1260,7 +1327,7 @@ export const webviewMessageHandler = async (
await provider.postStateToWebview()
break
case "followupAutoApproveTimeoutMs":
await updateGlobalState("followupAutoApproveTimeoutMs", message.value)
await updateGlobalState("followupAutoApproveTimeoutMs", Number(message.value))
await provider.postStateToWebview()
break
case "browserToolEnabled":
Expand All @@ -1281,20 +1348,20 @@ export const webviewMessageHandler = async (
await provider.postStateToWebview()
break
case "maxReadFileLine":
await updateGlobalState("maxReadFileLine", message.value)
await updateGlobalState("maxReadFileLine", Number(message.value))
await provider.postStateToWebview()
break
case "maxImageFileSize":
await updateGlobalState("maxImageFileSize", message.value)
await updateGlobalState("maxImageFileSize", Number(message.value))
await provider.postStateToWebview()
break
case "maxTotalImageSize":
await updateGlobalState("maxTotalImageSize", message.value)
await updateGlobalState("maxTotalImageSize", Number(message.value))
await provider.postStateToWebview()
break
case "maxConcurrentFileReads":
const valueToSave = message.value // Capture the value intended for saving
await updateGlobalState("maxConcurrentFileReads", valueToSave)
await updateGlobalState("maxConcurrentFileReads", Number(valueToSave))
await provider.postStateToWebview()
break
case "includeDiagnosticMessages":
Expand All @@ -1304,7 +1371,7 @@ export const webviewMessageHandler = async (
await provider.postStateToWebview()
break
case "maxDiagnosticMessages":
await updateGlobalState("maxDiagnosticMessages", message.value ?? 50)
await updateGlobalState("maxDiagnosticMessages", Number(message.value ?? 50))
await provider.postStateToWebview()
break
case "setHistoryPreviewCollapsed": // Add the new case handler
Expand Down
2 changes: 2 additions & 0 deletions src/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -415,6 +415,7 @@
"@anthropic-ai/vertex-sdk": "^0.7.0",
"@aws-sdk/client-bedrock-runtime": "^3.848.0",
"@aws-sdk/credential-providers": "^3.848.0",
"@google-cloud/text-to-speech": "^6.2.0",
"@google/genai": "^1.0.0",
"@lmstudio/sdk": "^1.1.1",
"@mistralai/mistralai": "^1.3.6",
Expand Down Expand Up @@ -448,6 +449,7 @@
"isbinaryfile": "^5.0.2",
"lodash.debounce": "^4.0.8",
"mammoth": "^1.9.1",
"microsoft-cognitiveservices-speech-sdk": "^1.45.0",
"monaco-vscode-textmate-theme-converter": "^0.1.7",
"node-cache": "^5.1.2",
"node-ipc": "^12.0.0",
Expand Down
Loading
Loading