Skip to content

Commit 5a45a19

Browse files
author
Xiting Zhang
committed
[VoiceLive] Add sample of function calling
1 parent 4aeaa80 commit 5a45a19

File tree

2 files changed

+689
-2
lines changed

2 files changed

+689
-2
lines changed

sdk/ai/azure-ai-voicelive/README.md

Lines changed: 61 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,7 @@ The following sections provide code snippets for common scenarios:
125125
* [Send audio input](#send-audio-input)
126126
* [Handle event types](#handle-event-types)
127127
* [Voice configuration](#voice-configuration)
128+
* [Function calling](#function-calling)
128129
* [Complete voice assistant with microphone](#complete-voice-assistant-with-microphone)
129130

130131
### Focused Sample Files
@@ -158,9 +159,16 @@ For easier learning, explore these focused samples in order:
158159
- Noise reduction and echo cancellation
159160
- Multi-threaded audio processing
160161

161-
> **Note:** To run audio samples (AudioPlaybackSample, MicrophoneInputSample, VoiceAssistantSample):
162+
6. **FunctionCallingSample.java** - Voice assistant with custom function tools
163+
- Define function tools with parameters
164+
- Register functions with the VoiceLive session
165+
- Handle function call requests from the AI model
166+
- Execute functions locally and return results
167+
- Continue conversation with function results
168+
169+
> **Note:** To run audio samples (AudioPlaybackSample, MicrophoneInputSample, VoiceAssistantSample, FunctionCallingSample):
162170
> ```bash
163-
> mvn exec:java -Dexec.mainClass=com.azure.ai.voicelive.AudioPlaybackSample -Dexec.classpathScope=test
171+
> mvn exec:java -Dexec.mainClass=com.azure.ai.voicelive.FunctionCallingSample -Dexec.classpathScope=test
164172
> ```
165173
> These samples use `javax.sound.sampled` for audio I/O.
166174
@@ -328,6 +336,57 @@ VoiceLiveSessionOptions options3 = new VoiceLiveSessionOptions()
328336
new AzurePersonalVoice("speakerProfileId", PersonalVoiceModels.PHOENIX_LATEST_NEURAL)));
329337
```
330338
339+
### Function calling
340+
341+
Enable your voice assistant to call custom functions during conversations. This allows the AI to request information or perform actions by executing your code:
342+
343+
```java com.azure.ai.voicelive.functioncalling
344+
// 1. Define function tool with parameters
345+
VoiceLiveFunctionDefinition getWeatherFunction = new VoiceLiveFunctionDefinition("get_current_weather")
346+
.setDescription("Get the current weather in a given location")
347+
.setParameters(BinaryData.fromObject(parametersSchema)); // JSON schema
348+
349+
// 2. Configure session with tools
350+
VoiceLiveSessionOptions options = new VoiceLiveSessionOptions()
351+
.setTools(Arrays.asList(getWeatherFunction))
352+
.setInstructions("You have access to weather information. Use get_current_weather when asked about weather.");
353+
354+
// 3. Handle function call events
355+
session.receiveEvents()
356+
.subscribe(event -> {
357+
if (event instanceof SessionUpdateConversationItemCreated) {
358+
SessionUpdateConversationItemCreated itemCreated = (SessionUpdateConversationItemCreated) event;
359+
if (itemCreated.getItem().getType() == ItemType.FUNCTION_CALL) {
360+
ResponseFunctionCallItem functionCall = (ResponseFunctionCallItem) itemCreated.getItem();
361+
362+
// Wait for arguments
363+
String callId = functionCall.getCallId();
364+
String arguments = waitForArguments(session, callId); // Helper method
365+
366+
// Execute function
367+
Map<String, Object> result = getCurrentWeather(arguments);
368+
String resultJson = new ObjectMapper().writeValueAsString(result);
369+
370+
// Return result
371+
FunctionCallOutputItem output = new FunctionCallOutputItem(callId, resultJson);
372+
ClientEventConversationItemCreate createItem = new ClientEventConversationItemCreate()
373+
.setItem(output)
374+
.setPreviousItemId(functionCall.getId());
375+
376+
session.sendEvent(createItem).subscribe();
377+
session.sendEvent(new ClientEventResponseCreate()).subscribe();
378+
}
379+
}
380+
});
381+
```
382+
383+
**Key points:**
384+
* Define function tools with JSON schemas describing parameters
385+
* The AI decides when to call functions based on conversation context
386+
* Your code executes the function and returns results
387+
* Results are sent back to continue the conversation
388+
* See `FunctionCallingSample.java` for a complete working example
389+
331390
### Complete voice assistant with microphone
332391
333392
A full example demonstrating real-time microphone input and audio playback:

0 commit comments

Comments
 (0)