Merge branch 'main' into workshop-page

blessing-sanusi · web-flow · commit 75e228b9feb5 · 2025-06-02T17:13:22.000-05:00
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
@@ -3,7 +3,8 @@
     "image": "mcr.microsoft.com/devcontainers/python:3.11-bullseye",
     "forwardPorts": [50505],
     "features": {
-        "ghcr.io/azure/azure-dev/azd:latest": {}
+        "ghcr.io/azure/azure-dev/azd:latest": {},
+        "ghcr.io/devcontainers/features/azure-cli:1": {}
     },
     "customizations": {
         "vscode": {
diff --git a/.devcontainer/setup_env.sh b/.devcontainer/setup_env.sh
@@ -5,4 +5,5 @@ git pull
 
 # provide execute permission to quotacheck script
 sudo chmod +x ./infra/scripts/checkquota_km.sh
-sudo chmod +x ./infra/scripts/quota_check_params.sh
+sudo chmod +x ./infra/scripts/quota_check_params.sh
+sudo chmod +x ./infra/scripts/run_process_data_scripts.sh
diff --git a/README.md b/README.md
@@ -15,19 +15,19 @@ Analysts working with large volumes of conversational data can use this solution
 </div>
 <br/>
 
-<h2><img src="./docs/Images/ReadMe/solution-overview.png" width="48" />
+<h2><img src="./documents/Images/ReadMe/solution-overview.png" width="48" />
 Solution overview
 </h2>
 
 Leverages Azure AI Content Understanding, Azure AI Search, Azure OpenAI Service, Semantic Kernel, Azure SQL Database, and Cosmos DB to process large volumes of conversational data. Audio and text inputs are analyzed through event-driven pipelines to extract and vectorize key information, orchestrate intelligent responses, and power an interactive web front-end for exploring insights using natural language.
 
 ### Solution architecture
-|![image](./docs/Images/ReadMe/solution-architecture.png)|
+|![image](./documents/Images/ReadMe/solution-architecture.png)|
 |---|
 
 ### Additional resources
 
-[Technical Architecture](./docs/TechnicalArchitecture.md)
+[Technical Architecture](./documents/TechnicalArchitecture.md)
 
 <br/>
 
@@ -55,14 +55,14 @@ Summarized conversations, topic generation, and key phrase extraction support fa
 
 
 <br /><br />
-<h2><img src="./docs/Images/ReadMe/quick-deploy.png" width="48" />
+<h2><img src="./documents/Images/ReadMe/quick-deploy.png" width="48" />
 Quick deploy
 </h2>
 
 ### How to install or deploy
 Follow the quick deploy steps on the deployment guide to deploy this solution to your own Azure subscription.
 
-[Click here to launch the deployment guide](./docs/DeploymentGuide.md)
+[Click here to launch the deployment guide](./documents/DeploymentGuide.md)
 <br/><br/>
 
 
@@ -72,12 +72,12 @@ Follow the quick deploy steps on the deployment guide to deploy this solution
 <br/>
 
 > ⚠️ **Important: Check Azure OpenAI Quota Availability**
- <br/>To ensure sufficient quota is available in your subscription, please follow [quota check instructions guide](./docs/QuotaCheck.md) before you deploy the solution.
+ <br/>To ensure sufficient quota is available in your subscription, please follow [quota check instructions guide](./documents/QuotaCheck.md) before you deploy the solution.
 
 <br/>
 
 ### Prerequisites and costs
-To deploy this solution accelerator, ensure you have access to an [Azure subscription](https://azure.microsoft.com/free/) with the necessary permissions to create **resource groups, resources, app registrations, and assign roles at the resource group level**. This should include Contributor role at the subscription level and  Role Based Access Control role on the subscription and/or resource group level. Follow the steps in [Azure Account Set Up](./docs/AzureAccountSetUp.md).
+To deploy this solution accelerator, ensure you have access to an [Azure subscription](https://azure.microsoft.com/free/) with the necessary permissions to create **resource groups, resources, app registrations, and assign roles at the resource group level**. This should include Contributor role at the subscription level and  Role Based Access Control role on the subscription and/or resource group level. Follow the steps in [Azure Account Set Up](./documents/AzureAccountSetUp.md).
 
 Here are some example regions where the services are available: East US, East US2, Australia East, UK South, France Central.
 
@@ -114,12 +114,12 @@ _Note: This is not meant to outline all costs as selected SKUs, scaled use, cust
 either by deleting the resource group in the Portal or running `azd down`.
 
 <br /><br />
-<h2><img src="./docs/Images/ReadMe/business-scenario.png" width="48" />
+<h2><img src="./documents/Images/ReadMe/business-scenario.png" width="48" />
 Business scenario
 </h2>
 
 
-|![image](./docs/Images/ReadMe/ui.png)|
+|![image](./documents/Images/ReadMe/ui.png)|
 |---|
 
 <br/>
@@ -152,7 +152,7 @@ Clear, contextual insights empower employees to take meaningful action based on
 
 <br /><br />
 
-<h2><img src="./docs/Images/ReadMe/supporting-documentation.png" width="48" />
+<h2><img src="./documents/Images/ReadMe/supporting-documentation.png" width="48" />
 Supporting documentation
 </h2>
 
@@ -204,4 +204,4 @@ You acknowledge that the Software and Microsoft Products and Services (1) are no
 
 You acknowledge the Software is not subject to SOC 1 and SOC 2 compliance audits. No Microsoft technology, nor any of its component technologies, including the Software, is intended or made available as a substitute for the professional advice, opinion, or judgement of a certified financial services professional. Do not use the Software to replace, substitute, or provide professional financial advice or judgment.  
 
-BY ACCESSING OR USING THE SOFTWARE, YOU ACKNOWLEDGE THAT THE SOFTWARE IS NOT DESIGNED OR INTENDED TO SUPPORT ANY USE IN WHICH A SERVICE INTERRUPTION, DEFECT, ERROR, OR OTHER FAILURE OF THE SOFTWARE COULD RESULT IN THE DEATH OR SERIOUS BODILY INJURY OF ANY PERSON OR IN PHYSICAL OR ENVIRONMENTAL DAMAGE (COLLECTIVELY, “HIGH-RISK USE”), AND THAT YOU WILL ENSURE THAT, IN THE EVENT OF ANY INTERRUPTION, DEFECT, ERROR, OR OTHER FAILURE OF THE SOFTWARE, THE SAFETY OF PEOPLE, PROPERTY, AND THE ENVIRONMENT ARE NOT REDUCED BELOW A LEVEL THAT IS REASONABLY, APPROPRIATE, AND LEGAL, WHETHER IN GENERAL OR IN A SPECIFIC INDUSTRY. BY ACCESSING THE SOFTWARE, YOU FURTHER ACKNOWLEDGE THAT YOUR HIGH-RISK USE OF THE SOFTWARE IS AT YOUR OWN RISK.
+BY ACCESSING OR USING THE SOFTWARE, YOU ACKNOWLEDGE THAT THE SOFTWARE IS NOT DESIGNED OR INTENDED TO SUPPORT ANY USE IN WHICH A SERVICE INTERRUPTION, DEFECT, ERROR, OR OTHER FAILURE OF THE SOFTWARE COULD RESULT IN THE DEATH OR SERIOUS BODILY INJURY OF ANY PERSON OR IN PHYSICAL OR ENVIRONMENTAL DAMAGE (COLLECTIVELY, “HIGH-RISK USE”), AND THAT YOU WILL ENSURE THAT, IN THE EVENT OF ANY INTERRUPTION, DEFECT, ERROR, OR OTHER FAILURE OF THE SOFTWARE, THE SAFETY OF PEOPLE, PROPERTY, AND THE ENVIRONMENT ARE NOT REDUCED BELOW A LEVEL THAT IS REASONABLY, APPROPRIATE, AND LEGAL, WHETHER IN GENERAL OR IN A SPECIFIC INDUSTRY. BY ACCESSING THE SOFTWARE, YOU FURTHER ACKNOWLEDGE THAT YOUR HIGH-RISK USE OF THE SOFTWARE IS AT YOUR OWN RISK.
diff --git a/docs/workshop/docs/workshop/Challenge-1/Code_Walkthrough/02_Frontend.md b/docs/workshop/docs/workshop/Challenge-1/Code_Walkthrough/02_Frontend.md
@@ -5,8 +5,10 @@
 
 The frontend is a **React-based web interface** that allows users to explore insights from conversations, interact with an AI-powered chatbot, and view dynamic visualizations.
 
+
 ![image](../../img/ReadMe/ckm-ui.png)
 
+
 ### Features
 
 1. **Dynamic Chart Rendering**
diff --git a/docs/workshop/docs/workshop/Challenge-1/Solution_Overview.md b/docs/workshop/docs/workshop/Challenge-1/Solution_Overview.md
@@ -2,8 +2,10 @@
 <!-- ## Overview -->
 The Conversation Knowledge Mining Solution Accelerator is a robust application designed to extract actionable insights from conversational data. It leverages Azure AI services and provides an interactive user interface for querying and visualizing data. The solution is built with a modular architecture, combining a React-based frontend, a FastAPI backend, and Azure services for data processing and storage.
 
+
 ![image](../img/ReadMe/ckm-sol-arch.png)
 
+
 The solution extracts insights from call audio files or transcripts and enables users to interact with the data via a chatbot and dynamic charts:
 
 1. **Ingest**: Audio/transcripts are stored.
diff --git a/documents/CustomizeData.md b/documents/CustomizeData.md
@@ -0,0 +1,23 @@
+## Customize the solution with your own data 
+
+If you would like to update the solution to leverage your own data please follow the steps below. 
+> Note: you will need to complete the deployment steps [here](./DeploymentGuide.md) before proceeding. 
+
+## Prerequisites: 
+1. Your data will need to be in JSON or wav format with the file name formated prefixed with "convo" then a GUID followed by a timestamp. For more examples of the data format, please review the sample transcripts and audio data included [here](/infra/data/)
+    * Example: convo_32e38683-bbf7-407e-a541-09b37b77921d_2024-12-07 04%3A00%3A00 
+
+
+1. Navigate to the storage account in the resource group you are using for this solution. 
+2. Open the `data` container
+3. If you have audio files, upload them to `custom_audiodata` folder. If you have call transcript files, upload them to `custom_transcripts` folder.
+4. Navigate to the terminal and run the `run_process_data_scripts.sh` to process the new data into the solution with the following commands. 
+    ```shell
+    cd infra/scripts
+
+    az login
+
+    bash run_process_data_scripts.sh resourcegroupname_param
+    ```
+    a. resourcegroupname_param - the name of the resource group.
+
diff --git a/documents/DeploymentGuide.md b/documents/DeploymentGuide.md
@@ -210,4 +210,8 @@ To help you get started, here are some **Sample Questions** you can ask in the a
 - Give a summary of billing issues
 - When customers call in about unexpected charges, what types of charges are they seeing?
 
-These questions serve as a great starting point to explore insights from the data.
+These questions serve as a great starting point to explore insights from the data.
+
+## Next Steps: 
+Now that you've completed your deployment, you can start using the solution. Try out these things to start getting familiar with the capabilities:
+* [Customize the solution](./CustomizeData.md) with your own data
diff --git a/documents/TechnicalArchitecture.md b/documents/TechnicalArchitecture.md
@@ -2,7 +2,7 @@
 
 This section outlines the components and interactions that power the conversational insights platform. The architecture ingests call transcripts and audio files, applies AI services for enrichment and structuring, and surfaces insights via an interactive web experience.
 
-![image](./images/readme/solution-architecture.png)
+![image](./Images/ReadMe/solution-architecture.png)
 
 ### Call Audio Files / Call Transcripts  
 Raw audio and text-based transcripts are the primary input into the system. These files are uploaded and stored for downstream processing.
diff --git a/infra/process_data_scripts.bicep b/infra/process_data_scripts.bicep
@@ -0,0 +1,25 @@
+param solutionLocation string
+param keyVaultName string
+param identity string
+
+var baseUrl = 'https://raw.githubusercontent.com/microsoft/Conversation-Knowledge-Mining-Solution-Accelerator/main/'
+
+resource process_data_scripts 'Microsoft.Resources/deploymentScripts@2020-10-01' = {
+  kind:'AzureCLI'
+  name: 'process_data_scripts'
+  location: solutionLocation // Replace with your desired location
+  identity: {
+    type: 'UserAssigned'
+    userAssignedIdentities: {
+      '${identity}' : {}
+    }
+  }
+  properties: {
+    azCliVersion: '2.52.0'
+    primaryScriptUri: '${baseUrl}infra/scripts/process_data_scripts.sh' 
+    arguments: '${baseUrl} ${keyVaultName}' // Specify any arguments for the script
+    timeout: 'PT1H' // Specify the desired timeout duration
+    retentionInterval: 'PT1H' // Specify the desired retention interval
+    cleanupPreference:'OnSuccess'
+  }
+}
diff --git a/infra/scripts/copy_kb_files.sh b/infra/scripts/copy_kb_files.sh
@@ -32,4 +32,6 @@ echo "Script Started"
 az login --identity --client-id ${managedIdentityClientId}
 # Using az storage blob upload-batch to upload files with managed identity authentication, as the az storage fs directory upload command is not working with managed identity authentication.
 az storage blob upload-batch --account-name "$storageAccount" --destination data/"$extractedFolder1" --source /mnt/azscripts/azscriptinput/"$extractedFolder1" --auth-mode login --pattern '*' --overwrite
-az storage blob upload-batch --account-name "$storageAccount" --destination data/"$extractedFolder2" --source /mnt/azscripts/azscriptinput/"$extractedFolder2" --auth-mode login --pattern '*' --overwrite
+az storage blob upload-batch --account-name "$storageAccount" --destination data/"$extractedFolder2" --source /mnt/azscripts/azscriptinput/"$extractedFolder2" --auth-mode login --pattern '*' --overwrite
+az storage fs directory create --account-name "$storageAccount" --file-system data --name custom_audiodata --auth-mode login
+az storage fs directory create --account-name "$storageAccount" --file-system data --name custom_transcripts --auth-mode login
diff --git a/infra/scripts/index_scripts/04_cu_process_data_new_data.py b/infra/scripts/index_scripts/04_cu_process_data_new_data.py
diff --git a/infra/scripts/process_data_scripts.sh b/infra/scripts/process_data_scripts.sh
diff --git a/infra/scripts/run_process_data_scripts.sh b/infra/scripts/run_process_data_scripts.sh
