Add custom orchestration with Semantic Kernel

Author: jelledruyts

README.md

@@ -2,7 +2,7 @@
 
 ## About
 
-This project provides a very easy to use learning and experimentation playground to try out various AI-enabled search scenarios in Azure. It provides a web application front-end which uses [Azure Cognitive Search](https://learn.microsoft.com/azure/search/search-what-is-azure-search) and [Azure OpenAI](https://learn.microsoft.com/azure/ai-services/openai/overview) to execute searches with a variety of options. This allows you to quickly understand what each option does, how it affects the search results, and how various approaches compare against each other.
+This project provides a very easy to use learning and experimentation playground to try out various AI-enabled search scenarios in Azure. It provides a web application front-end which uses [Azure Cognitive Search](https://learn.microsoft.com/azure/search/search-what-is-azure-search) and [Azure OpenAI](https://learn.microsoft.com/azure/ai-services/openai/overview) to execute searches with a variety of options - ranging from simple text search, to semantic search, vector and hybrid search, and using generative AI to answer search queries in various ways. This allows you to quickly understand what each option does, how it affects the search results, and how various approaches compare against each other.
 
 By default, a few documents are added automatically to allow you to use the application directly. You're encouraged to upload your own documents (which you can also easily do via the web app) so you can experiment with searching over your own content.
 
@@ -44,7 +44,7 @@ When you deploy the solution, it creates an [Azure Cognitive Search](https://lea
 
 The documents in the index are also chunked into smaller pieces, and vector embeddings are created for these chunks using a Function App based on the [Azure OpenAI Embeddings Generator power skill](https://github.com/Azure-Samples/azure-search-power-skills/tree/main/Vector/EmbeddingGenerator). This allows you to easily try out [vector and hybrid search](https://learn.microsoft.com/azure/search/vector-search-overview). With Azure Cognitive Search on its own, the responses *always* come directly from the source data, rather than being generated by an AI model. You can optionally use [semantic search](https://learn.microsoft.com/azure/search/semantic-search-overview) which *does* use AI, not to generate content but to increase the relevancy of the results and provide semantic answers and captions.
 
-The solution also deploys an [Azure OpenAI](https://learn.microsoft.com/azure/ai-services/openai/overview) service. It provides an embeddings model to generate the vector representations of the document chunks and search queries, and a GPT model to generate answers to your search queries. If you choose the option to use [Azure OpenAI "on your data"](https://learn.microsoft.com/azure/ai-services/openai/concepts/use-your-data), these AI-generated responses can be grounded in (and even limited to) the information in your Azure Cognitive Search indexes. This option allows you to let Azure OpenAI orchestrate the [Retrieval Augmented Generation (RAG)](https://aka.ms/what-is-rag) pattern. This means your search query will first be used to retrieve the most relevant documents (or preferably *smaller chunks of those documents*) from your private data source. Those search results are then used as context in the prompt that gets sent to the AI model, along with the original search query. This allows the AI model to generate a response based on the most relevant source data, rather than the public data that was used to train the model.
+The solution also deploys an [Azure OpenAI](https://learn.microsoft.com/azure/ai-services/openai/overview) service. It provides an embeddings model to generate the vector representations of the document chunks and search queries, and a GPT model to generate answers to your search queries. If you choose the option to use [Azure OpenAI "on your data"](https://learn.microsoft.com/azure/ai-services/openai/concepts/use-your-data), these AI-generated responses can be grounded in (and even limited to) the information in your Azure Cognitive Search indexes. This option allows you to let Azure OpenAI orchestrate the [Retrieval Augmented Generation (RAG)](https://aka.ms/what-is-rag) pattern. This means your search query will first be used to retrieve the most relevant documents (or preferably *smaller chunks of those documents*) from your private data source. Those search results are then used as context in the prompt that gets sent to the AI model, along with the original search query. This allows the AI model to generate a response based on the most relevant source data, rather than the public data that was used to train the model. Next to letting Azure OpenAI orchestrate the RAG pattern, the web application can also use [Semantic Kernel](https://learn.microsoft.com/semantic-kernel/overview/) to perform that orchestration, using a prompt and other parameters you can control yourself.
 
 ## Deployment
 
@@ -117,7 +117,8 @@ The ARM template deploys the services and sets the configuration settings for th
 | `SearchIndexNameBlobChunks`* | The name of the search index that contains the document chunks | `blob-chunks` |
 | `SearchIndexerScheduleMinutes`* | The number of minutes between indexer executions in Azure Cognitive Search | `5` |
 | `InitialDocumentUrls` | A space-separated list of URLs for the documents to include by default | A [resiliency](https://azure.microsoft.com/mediahandler/files/resourcefiles/resilience-in-azure-whitepaper/Resiliency-whitepaper.pdf) and [compliance](https://azure.microsoft.com/mediahandler/files/resourcefiles/data-residency-data-sovereignty-and-compliance-in-the-microsoft-cloud/Data_Residency_Data_Sovereignty_Compliance_Microsoft_Cloud.pdf) document |
-| `DefaultSystemRoleInformation` | The default instructions for the AI model | You are an AI assistant that helps people find information. |
+| `DefaultSystemRoleInformation` | The default instructions for the AI model | "You are an AI assistant that helps people find information." |
+| `DefaultCustomOrchestrationPrompt` | The default prompt for the AI model when using custom orchestration | A prompt that instructs the AI model to respond from provided data sources (with citations). |
 | `DisableUploadDocuments` | Set this to `true` to disable the functionality to upload documents, preventing uploads by users of the Web App (you can still upload documents directly to the Azure storage container if you have permissions there) | `false` |
 | `DisableResetSearchConfiguration` | Set this to `true` to disable the functionality to reset the search configuration by users of the Web App | `false` |
 

azuredeploy-webapp.json

@@ -66,7 +66,8 @@
         "searchIndexNameBlobDocuments": "blob-documents",
         "searchIndexNameBlobChunks": "blob-chunks",
         "searchIndexerScheduleMinutes": 5,
-        "defaultSystemRoleInformation": "You are an AI assistant that helps people find information.",
+        "defaultSystemRoleInformation": "",
+        "defaultCustomOrchestrationPrompt": "",
         "disableUploadDocuments": false,
         "disableResetSearchConfiguration": false
     },
@@ -380,6 +381,10 @@
                             "name": "DefaultSystemRoleInformation",
                             "value": "[variables('defaultSystemRoleInformation')]"
                         },
+                        {
+                            "name": "DefaultCustomOrchestrationPrompt",
+                            "value": "[variables('defaultCustomOrchestrationPrompt')]"
+                        },
                         {
                             "name": "DisableUploadDocuments",
                             "value": "[variables('disableUploadDocuments')]"
@@ -478,6 +483,10 @@
             "type": "string",
             "value": "[variables('defaultSystemRoleInformation')]"
         },
+        "defaultCustomOrchestrationPrompt": {
+            "type": "string",
+            "value": "[variables('defaultCustomOrchestrationPrompt')]"
+        },
         "disableUploadDocuments": {
             "type": "bool",
             "value": "[variables('disableUploadDocuments')]"

media/compare-results.png

@@ -22,7 +22,18 @@ public class AppSettings
     public string? SearchIndexNameBlobChunks { get; set; }
     public int? SearchIndexerScheduleMinutes { get; set; } // If unspecified, will be set to 5 minutes.
     public string? InitialDocumentUrls { get; set; }
-    public string? DefaultSystemRoleInformation { get; set; } = "You are an AI assistant that helps people find information.";
+    public string? DefaultSystemRoleInformation { get; set; }
+    public string? DefaultCustomOrchestrationPrompt { get; set; }
     public bool DisableUploadDocuments { get; set; } // If true, the Upload Documents functionality will be disabled.
     public bool DisableResetSearchConfiguration { get; set; } // If true, the Reset Search Configuration functionality will be disabled.
+
+    public string GetDefaultSystemRoleInformation()
+    {
+        return string.IsNullOrWhiteSpace(this.DefaultSystemRoleInformation) ? Constants.Defaults.SystemRoleInformation : this.DefaultSystemRoleInformation;
+    }
+
+    public string GetDefaultCustomOrchestrationPrompt()
+    {
+        return string.IsNullOrWhiteSpace(this.DefaultCustomOrchestrationPrompt) ? Constants.Defaults.CustomOrchestrationPrompt : this.DefaultCustomOrchestrationPrompt;
+    }
 }

media/search-options.png

@@ -13,6 +13,7 @@
     <PackageReference Include="Azure.Storage.Blobs" Version="12.17.0" />
     <PackageReference Include="Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation" Version="6.0.20" />
     <PackageReference Include="Microsoft.Extensions.Http" Version="6.0.0" />
+    <PackageReference Include="Microsoft.SemanticKernel" Version="0.23.230906.2-preview" />
   </ItemGroup>
 
 </Project>

src/Azure.AISearch.WebApp/AppSettings.cs

@@ -1,18 +1,57 @@
+using Azure.Search.Documents.Indexes.Models;
+
 namespace Azure.AISearch.WebApp;
 
 public static class Constants
 {
-    public static class ChatRoles
-    {
-        public const string System = "system";
-        public const string Assistant = "assistant";
-        public const string Tool = "tool";
-        public const string User = "user";
-    }
-
     public static class ConfigurationNames
     {
         public const string SemanticConfigurationNameDefault = "default";
         public const string VectorSearchConfigurationNameDefault = "default";
     }
+
+    public static class Defaults
+    {
+        public const string SystemRoleInformation = "You are an AI assistant that helps people find information.";
+
+        // Adapted from https://github.com/Azure-Samples/azure-search-openai-demo-csharp/blob/feature/embeddingSearch/app/backend/Services/RetrieveThenReadApproachService.cs
+        public const string CustomOrchestrationPrompt = @"You are an intelligent assistant.
+Use 'you' to refer to the individual asking the questions even if they ask with 'I'.
+Answer the following question using only the data provided in the sources below.
+For tabular information return it as an HTML table. Do not return markdown format.
+Each source has a name followed by a colon and the actual information, always include the source name for each fact you use in the response.
+If you cannot answer using the sources below, say you don't know.
+
+###
+Question: 'What is the deductible for the employee plan for a visit to Overlake in Bellevue?'
+
+Sources:
+info1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employees and $1000 for families. Out-of-network deductibles are $1000 for employees and $2000 for families.
+info2.pdf: Overlake is in-network for the employee plan.
+info3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue.
+info4.pdf: In-network institutions include Overlake, Swedish, and others in the region
+
+Answer:
+In-network deductibles are $500 for employees and $1000 for families <cite>info1.txt</cite> and Overlake is in-network for the employee plan <cite>info2.pdf</cite><cite>info4.pdf</cite>.
+
+###
+Question: {{$query}}?
+
+Sources:
+{{$sources}}
+
+Answer:
+";
+        public const int VectorNearestNeighborsCount = 50;
+        public const int MaxTokens = 800;
+        public const double Temperature = 0.7;
+        public const double TopP = 0.95;
+        public const double FrequencyPenalty = 0.0;
+        public const double PresencePenalty = 0.0;
+        public const string StopSequences = "";
+        public const int HnswParametersM = 4;
+        public const int HnswParametersEfConstruction = 400;
+        public const int HnswParametersEfSearch = 500;
+        public static readonly VectorSearchAlgorithmMetric HnswParametersMetric = VectorSearchAlgorithmMetric.Cosine;
+    }
 }

src/Azure.AISearch.WebApp/Azure.AISearch.WebApp.csproj

@@ -3,5 +3,6 @@ namespace Azure.AISearch.WebApp.Models;
 public enum EngineType
 {
     AzureOpenAI,
-    AzureCognitiveSearch
+    AzureCognitiveSearch,
+    CustomOrchestration
 }

src/Azure.AISearch.WebApp/Constants.cs

@@ -8,8 +8,16 @@ public class SearchRequest
     public SearchIndexType SearchIndex { get; set; } = SearchIndexType.Documents;
     public QueryType QueryType { get; set; } = QueryType.TextStandard;
     public DataSourceType DataSource { get; set; } = DataSourceType.None;
+    public int? VectorNearestNeighborsCount { get; set; } = Constants.Defaults.VectorNearestNeighborsCount;
     public bool LimitToDataSource { get; set; } = true; // "Limit responses to your data content"
     public string? SystemRoleInformation { get; set; } // Give the model instructions about how it should behave and any context it should reference when generating a response. You can describe the assistant’s personality, tell it what it should and shouldn’t answer, and tell it how to format responses. There’s no token limit for this section, but it will be included with every API call, so it counts against the overall token limit.
+    public string? CustomOrchestrationPrompt { get; set; }
+    public int? MaxTokens { get; set; } = Constants.Defaults.MaxTokens;
+    public double? Temperature { get; set; } = Constants.Defaults.Temperature;
+    public double? TopP { get; set; } = Constants.Defaults.TopP;
+    public double? FrequencyPenalty { get; set; } = Constants.Defaults.FrequencyPenalty;
+    public double? PresencePenalty { get; set; } = Constants.Defaults.PresencePenalty;
+    public string? StopSequences { get; set; } = Constants.Defaults.StopSequences;
 
     public bool IsVectorSearch => QueryType == Models.QueryType.Vector || QueryType == Models.QueryType.HybridStandard || QueryType == Models.QueryType.HybridSemantic;
     public bool IsSemanticSearch => QueryType == Models.QueryType.TextSemantic || QueryType == Models.QueryType.HybridSemantic;