|
| 1 | +--- |
| 2 | +title: Integrate Microsoft Copilot Studio to Azure Communication Services |
| 3 | +titleSuffix: An Azure Communication Services sample showing how to integrate with agents built using Microsoft Copilot Studio |
| 4 | +description: Overview of Call Automation sample using Azure Communication Services to enable developers to learn how to incorporate voice into their Microsoft Copilot Agent. |
| 5 | +author: kpunjabi |
| 6 | +services: azure-communication-services |
| 7 | +ms.author: kpunjabi |
| 8 | +ms.date: 04/01/2024 |
| 9 | +ms.topic: overview |
| 10 | +ms.service: azure-communication-services |
| 11 | +ms.subservice: call-automation |
| 12 | +ms.custom: devx-track-extended-csharp |
| 13 | +zone_pivot_groups: acs-csharp |
| 14 | +--- |
| 15 | + |
| 16 | +# Supercharge Your Voice Interactions: Integrating Azure Communication Services with Microsoft Copilot Studio Agents |
| 17 | + |
| 18 | +This document provides step-by-step instructions on how to create and integrate a Copilot Studio agent with Azure Communication Services. This guide will allow you to create voice-enabled agents that your users can call into. |
| 19 | + |
| 20 | +## Download the sample |
| 21 | +Find the project for this sample on [GitHub](https://github.com/Azure-Samples/communication-services-dotnet-quickstarts/tree/main/CallAutomation_MCS_Sample). You can download this code and run it locally to try it for yourself. |
| 22 | + |
| 23 | +## Prerequisites |
| 24 | +Before you begin, ensure you have: |
| 25 | +- Azure account with an active subscription, for details see [Create an account for free](https://azure.microsoft.com/free/). |
| 26 | +- Azure Communication Services resource, see [create a new Azure Communication Services resource](../quickstarts/create-communication-resource.md?tabs=windows&pivots=platform-azp). You need to record your resource **connection string** for this sample. |
| 27 | +- Create a new web service application using Call automation SDK. |
| 28 | +- An Azure AI Multiservice resource and a custom domain. |
| 29 | +- [Connect Azure Communication Services and Azure AI](/azure/communication-services/concepts/call-automation/azure-communication-services-azure-cognitive-services-integration). |
| 30 | +- A Copilot Studio License so that you can create and publish an agent. |
| 31 | + |
| 32 | +## 1. Create your Agent in Copilot Studio |
| 33 | +After logging in or signing up for Copilot Studio, you land on the Home page. Select **Create** in the left navigation. |
| 34 | + |
| 35 | + |
| 36 | + |
| 37 | +On the Create page, select **New agent**. |
| 38 | +Use the chat to describe your agent, using the provided questions for guidance. |
| 39 | +Once you provided all the requested information, click **Create**. |
| 40 | + |
| 41 | + |
| 42 | + |
| 43 | +For more details on creating and customizing your agent, you can see the [Copilot Studio quickstart](/copilot-studio/quickstart). |
| 44 | + |
| 45 | +## 2. Disable Authentication |
| 46 | +Once you’ve created your agent, you need to make some updates so that you can integrate it with Azure Communication Service. |
| 47 | + |
| 48 | +- Navigate to the **Settings** tab. |
| 49 | + |
| 50 | + |
| 51 | + |
| 52 | +- Click on **Security** on the left pane. |
| 53 | + |
| 54 | + |
| 55 | + |
| 56 | +- Select **Authentication**, select **No Authentication**, and click **Save**. |
| 57 | + |
| 58 | + |
| 59 | + |
| 60 | +## 3. Get the Webchannel Security Key |
| 61 | +Navigating back to the **Security** section select **Web Channel Security**. Copy and save this key somewhere. You need this when you’re deploying your application. |
| 62 | + |
| 63 | +## 4. Publish Agent |
| 64 | +Now that you’ve got your agents settings updated and saved your agent key, you can publish your agent. |
| 65 | + |
| 66 | +## 5. Setting up Code |
| 67 | +Now that you created your agent, make sure to download the [sample](https://github.com/Azure-Samples/communication-services-dotnet-quickstarts/tree/main/CallAutomation_MCS_Sample). After downloading the sample, you will need to update some of the properties. |
| 68 | + |
| 69 | +- Your connection string: You can get your Connection string from your Azure Communication Services resource. |
| 70 | +- Microsoft Copilot Studio Direct Line key: Which you saved in step 3, your webchannel security key. |
| 71 | +- Azure AI Services custom endpoint: You can get this endpoint from your Azure AI Services resource. |
| 72 | +- You need to have a port running to receive event notifications from Azure Communication Services. You can use tools like [DevTunnels](/azure/developer/dev-tunnels/overview) to help set one up. |
| 73 | + |
| 74 | +## 6. Overview of the Code |
| 75 | +There a few basic concepts you must be familiar with that the sample uses to build out this workflow. |
| 76 | + |
| 77 | +### Incoming call |
| 78 | +Register an [incoming call event](../concepts/call-automation/incoming-call-notification.md), so that your application knows when a call is coming in and needs to answer. |
| 79 | + |
| 80 | +### Answer call with real-time transcription |
| 81 | +When answering the call you also enable streaming of real-time transcription, which sends the Speech-to-Text converted content the caller is saying in near real-time. |
| 82 | + |
| 83 | +``` csharp |
| 84 | +app.MapPost("/api/incomingCall", async ( |
| 85 | + [FromBody] EventGridEvent[] eventGridEvents, |
| 86 | + ILogger<Program> logger) => |
| 87 | +{ |
| 88 | + foreach (var eventGridEvent in eventGridEvents) |
| 89 | + { |
| 90 | + logger.LogInformation($"Incoming Call event received : {JsonConvert.SerializeObject(eventGridEvent)}"); |
| 91 | + // Handle system events |
| 92 | + if (eventGridEvent.TryGetSystemEventData(out object eventData)) |
| 93 | + { |
| 94 | + // Handle the subscription validation event. |
| 95 | + if (eventData is SubscriptionValidationEventData subscriptionValidationEventData) |
| 96 | + { |
| 97 | + var responseData = new SubscriptionValidationResponse |
| 98 | + { |
| 99 | + ValidationResponse = subscriptionValidationEventData.ValidationCode |
| 100 | + }; |
| 101 | + return Results.Ok(responseData); |
| 102 | + } |
| 103 | + } |
| 104 | + var jsonObject = JsonNode.Parse(eventGridEvent.Data).AsObject(); |
| 105 | + var incomingCallContext = (string)jsonObject["incomingCallContext"]; |
| 106 | + var callbackUri = new Uri(baseUri + $"/api/calls/{Guid.NewGuid()}"); |
| 107 | + |
| 108 | + var answerCallOptions = new AnswerCallOptions(incomingCallContext, callbackUri) |
| 109 | + { |
| 110 | + CallIntelligenceOptions = new CallIntelligenceOptions() |
| 111 | + { |
| 112 | + CognitiveServicesEndpoint = new Uri(cognitiveServicesEndpoint) |
| 113 | + }, |
| 114 | + TranscriptionOptions = new TranscriptionOptions(new Uri($"wss://{baseWssUri}/ws"), "en-US", true, TranscriptionTransport.Websocket) |
| 115 | + { |
| 116 | + EnableIntermediateResults = true |
| 117 | + } |
| 118 | + }; |
| 119 | + |
| 120 | + try |
| 121 | + { |
| 122 | + AnswerCallResult answerCallResult = await client.AnswerCallAsync(answerCallOptions); |
| 123 | + |
| 124 | + var correlationId = answerCallResult?.CallConnectionProperties.CorrelationId; |
| 125 | + logger.LogInformation($"Correlation Id: {correlationId}"); |
| 126 | + |
| 127 | + if (correlationId != null) |
| 128 | + { |
| 129 | + CallStore[correlationId] = new CallContext() |
| 130 | + { |
| 131 | + CorrelationId = correlationId |
| 132 | + }; |
| 133 | + } |
| 134 | + } |
| 135 | + catch (Exception ex) |
| 136 | + { |
| 137 | + logger.LogError($"Answer call exeception : {ex.StackTrace}"); |
| 138 | + } |
| 139 | + } |
| 140 | + return Results.Ok(); |
| 141 | +}); |
| 142 | +``` |
| 143 | +### Establish a copilot connection |
| 144 | +Once call is connected, the application needs to establish a connection to the AI agent you built using Direct Line APIs with websockets. |
| 145 | + |
| 146 | +### Start conversation |
| 147 | +``` csharp |
| 148 | +var response = await httpClient.PostAsync("https://directline.botframework.com/v3/directline/conversations", null); |
| 149 | +response.EnsureSuccessStatusCode(); |
| 150 | +var content = await response.Content.ReadAsStringAsync(); |
| 151 | +return JsonConvert.DeserializeObject(content); |
| 152 | +``` |
| 153 | +### Listen to webscket |
| 154 | +```csharp |
| 155 | +await webSocket.ConnectAsync(new Uri(streamUrl), cancellationToken); |
| 156 | + |
| 157 | +var buffer = new byte[4096]; // Set the buffer size to 4096 bytes |
| 158 | +var messageBuilder = new StringBuilder(); |
| 159 | + |
| 160 | +while (webSocket.State == WebSocketState.Open && !cancellationToken.IsCancellationRequested) |
| 161 | +{ |
| 162 | + messageBuilder.Clear(); // Reset buffer for each new message |
| 163 | + WebSocketReceiveResult result; |
| 164 | + do |
| 165 | + { |
| 166 | + result = await webSocket.ReceiveAsync(new ArraySegment<byte>(buffer), cancellationToken); |
| 167 | + messageBuilder.Append(Encoding.UTF8.GetString(buffer, 0, result.Count)); |
| 168 | + } |
| 169 | + while (!result.EndOfMessage); // Continue until we've received the full message |
| 170 | +} |
| 171 | +``` |
| 172 | + |
| 173 | +### Built in barge-in logic: |
| 174 | +The application uses intermediate results received from real-time transcription to detect barge-in from the caller and [cancels the play operation](../how-tos/call-automation/play-action.md). |
| 175 | + |
| 176 | +``` csharp |
| 177 | +if (data.Contains("Intermediate")) |
| 178 | +{ |
| 179 | + Console.WriteLine("\nCanceling prompt"); |
| 180 | + if (callMedia != null) |
| 181 | + { |
| 182 | + await callMedia.CancelAllMediaOperationsAsync(); |
| 183 | + } |
| 184 | +} |
| 185 | +``` |
| 186 | +- When the AI agent provides responses, the application uses [Play API](../how-tos/call-automation/play-action.md) to convert that text to audio the Text-to-Speech service. |
| 187 | + |
| 188 | +```csharp |
| 189 | +var ssmlPlaySource = new SsmlSource($"{message}"); |
| 190 | + |
| 191 | +var playOptions = new PlayToAllOptions(ssmlPlaySource) |
| 192 | +{ |
| 193 | + OperationContext = "Testing" |
| 194 | +}; |
| 195 | + |
| 196 | +await callConnectionMedia.PlayToAllAsync(playOptions); |
| 197 | +``` |
| 198 | +- Escalate call when caller asks for a representative: When the user asks to speak to a representative, the AI agent transfers the call to a human agent. |
| 199 | + |
| 200 | +```csharp |
| 201 | +if (botActivity.Type == "handoff") |
| 202 | +{ |
| 203 | + var transferOptions = new TransferToParticipantOptions(agentPhoneIdentity) |
| 204 | + { |
| 205 | + SourceCallerIdNumber = acsPhoneIdentity |
| 206 | + }; |
| 207 | + |
| 208 | + await Task.Delay(6000); |
| 209 | + await callConnection.TransferCallToParticipantAsync(transferOptions); |
| 210 | +} |
| 211 | +``` |
| 212 | + |
| 213 | +## 7. Run |
| 214 | +You should now be able to make a call and talk to your agent. |
| 215 | + |
| 216 | +## Tips |
| 217 | +### Topics |
| 218 | +To optimize for voice, we would recommend you update topics where you’re using the "Message" type of Text to Speech, as it optimizes the agents responses for Speech scenarios. |
| 219 | + |
| 220 | +### How to Handle System Topics |
| 221 | +Your agent has System Topics built in by default. You can choose to disable these topics, but if you wish to continue using them, your application should build logic to handle these topics. Such as: |
| 222 | +- **Escalate**: You need to build agent transfer into your application to escalate the call from this copilot agent to a human representative. |
| 223 | + |
0 commit comments