refactor: scripts for improved logging and error handling; remove redundant code and streamline role assignments in Azure CLI scripts

Author: Avijit-Microsoft

documents/DeploymentGuide.md

@@ -147,7 +147,7 @@ If you're not using one of the above options for opening the project, then you'l
 1. Make sure the following tools are installed:
     - [PowerShell](https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7.5) <small>(v7.0+)</small> - available for Windows, macOS, and Linux.
     - [Azure Developer CLI (azd)](https://aka.ms/install-azd) <small>(v1.18.0+)</small> - version
-    - [Python 3.9 to 3.11](https://www.python.org/downloads/)
+    - [Python 3.9+](https://www.python.org/downloads/)
     - [Docker Desktop](https://www.docker.com/products/docker-desktop/)
     - [Git](https://git-scm.com/downloads)
     - [Microsoft ODBC Driver 18](https://learn.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16) for SQL Server.
@@ -248,7 +248,32 @@ Once you've opened the project in [Codespaces](#github-codespaces), [Dev Contain
     - This deployment generally takes **7-10 minutes** to provision the resources in your account and set up the solution.
     - If you encounter an error or timeout during deployment, changing the location may help, as there could be availability constraints for the resources.
 
-5. Once the deployment has completed successfully and you would like to use the **sample data**, please open a **Git Bash terminal** and run the bash command printed below. The bash command will look like the following:
+5. Once the deployment has completed successfully, copy the bash command from terminal: (ex: `bash ./infra/scripts/process_sample_data.sh`) for later use.
+
+> **Note**: If you are running this deployment in GitHub Codespaces or VS Code Dev Container or Visual Studio Code (WEB) skip to step 7.
+
+6. Create and activate a virtual environment in bash terminal:
+  
+    ```shell
+    python -m venv .venv
+    ```
+
+    ```shell
+    source .venv/Scripts/activate
+    ```
+
+7. Login to Azure:
+
+    ```shell
+    az login
+    ```
+
+    Alternatively, login to Azure using a device code (recommended when using VS Code Web):
+
+    ```shell
+    az login --use-device-code
+    ```
+8. Run the bash script from the output of the azd deployment. The script will look like the following:
 
     ```bash
     bash ./infra/scripts/process_sample_data.sh
@@ -267,9 +292,9 @@ Once you've opened the project in [Codespaces](#github-codespaces), [Dev Contain
       <CU-Endpoint> <AI-Agent-Endpoint> <CU-API-Version>
     ```
 
-6. Once the deployment has completed successfully, open the [Azure Portal](https://portal.azure.com/), go to the deployed resource group, find the App Service, and get the app URL from `Default domain`.
+9. Once the deployment has completed successfully, open the [Azure Portal](https://portal.azure.com/), go to the deployed resource group, find the App Service, and get the app URL from `Default domain`.
 
-7. You can now delete the resources by running `azd down`, if you are done trying out the application.
+10. You can now delete the resources by running `azd down`, if you are done trying out the application.
    > **Note:** If you deployed with `enableRedundancy=true` and Log Analytics workspace replication is enabled, you must first disable replication before running `azd down` else resource group delete will fail. Follow the steps in [Handling Log Analytics Workspace Deletion with Replication Enabled](./LogAnalyticsReplicationDisable.md), wait until replication returns `false`, then run `azd down`.
 
 ### 🛠️ Troubleshooting

documents/LocalDebuggingSetup.md

@@ -11,7 +11,7 @@ Install these tools before you start:
   - [Azure Tools](https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-node-azure-pack)
   - [Bicep](https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-bicep)
   - [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
-- [Python 3.9 to 3.11](https://www.python.org/downloads/). **Important:** Check "Add Python to PATH" during installation.
+- [Python 3.9+](https://www.python.org/downloads/). **Important:** Check "Add Python to PATH" during installation.
 - [PowerShell 7.0+](https://github.com/PowerShell/PowerShell#get-powershell).
 - [Node.js (LTS)](https://nodejs.org/en).
 - [Git](https://git-scm.com/downloads).

infra/scripts/add_user_scripts/assign_sql_roles.py

@@ -55,24 +55,18 @@ def assign_sql_roles(server, database, roles_json):
         credential = AzureCliCredential()
 
         # Connect to SQL Server
-        print(f"Connecting to {server}/{database}...")
         conn = connect_with_token(server, database, credential)
         cursor = conn.cursor()
 
-        print("Connected successfully.")
-        
         # Process each role assignment
         for role_assignment in roles:
             client_id = role_assignment.get("clientId")
             display_name = role_assignment.get("displayName")
             role = role_assignment.get("role")
 
             if not client_id or not display_name or not role:
-                print(f"Skipping invalid role assignment: {role_assignment}")
                 continue
 
-            print(f"\nProcessing: {display_name} -> {role}")
-            
             # Check if user already exists
             check_user_sql = f"SELECT COUNT(*) FROM sys.database_principals WHERE name = '{display_name}'"
             cursor.execute(check_user_sql)
@@ -81,16 +75,13 @@ def assign_sql_roles(server, database, roles_json):
             if not user_exists:
                 # Create user from external provider
                 create_user_sql = f"CREATE USER [{display_name}] FROM EXTERNAL PROVIDER"
-                print(f"  Creating user: {display_name}")
                 try:
                     cursor.execute(create_user_sql)
                     conn.commit()
-                    print("  ✓ User created successfully")
+                    print(f"✓ Created user: {display_name}")
                 except Exception as e:
-                    print(f"  ✗ Failed to create user: {e}")
+                    print(f"✗ Failed to create user: {e}")
                     continue
-            else:
-                print(f"  User already exists: {display_name}")
 
             # Check if user already has the role
             check_role_sql = f"""
@@ -106,21 +97,17 @@ def assign_sql_roles(server, database, roles_json):
             if not has_role:
                 # Add user to role
                 add_role_sql = f"ALTER ROLE [{role}] ADD MEMBER [{display_name}]"
-                print(f"  Assigning role: {role}")
                 try:
                     cursor.execute(add_role_sql)
                     conn.commit()
-                    print("  ✓ Role assigned successfully")
+                    print(f"✓ Assigned {role} to {display_name}")
                 except Exception as e:
-                    print(f"  ✗ Failed to assign role: {e}")
+                    print(f"✗ Failed to assign {role}: {e}")
                     continue
-            else:
-                print(f"  User already has role: {role}")
 
         # Close connection
         cursor.close()
         conn.close()
-        print("\n✓ All role assignments completed successfully")
         return 0
 
     except Exception as e:

infra/scripts/copy_kb_files.sh

@@ -11,11 +11,6 @@ extractedFolder1="call_transcripts"
 zipFileName2="infra/data/audio_data.zip"
 extractedFolder2="audio_data"
 
-echo "Script Started"
-echo "Storage Account: $storageAccountName"
-echo "Container Name: $containerName"
-echo "Resource Group: $resourceGroupName"
-
 # Validate required parameters
 if [ -z "$storageAccountName" ] || [ -z "$containerName" ] || [ -z "$resourceGroupName" ]; then
     echo "Error: Missing required parameters."
@@ -25,102 +20,76 @@ fi
 
 # Extract zip files if they exist
 if [ -f "$zipFileName1" ]; then
-	echo "Extracting $zipFileName1..."
-	unzip -o "$zipFileName1" -d "$extractedFolder1"
-else
-	echo "Warning: $zipFileName1 not found. Skipping extraction."
+	unzip -q -o "$zipFileName1" -d "$extractedFolder1"
 fi
 
 if [ -f "$zipFileName2" ]; then
-	echo "Extracting $zipFileName2..."
-	unzip -o "$zipFileName2" -d "$extractedFolder2"
-else
-	echo "Warning: $zipFileName2 not found. Skipping extraction."
+	unzip -q -o "$zipFileName2" -d "$extractedFolder2"
 fi
 
 # Authenticate with Azure
-if az account show &> /dev/null; then
-	echo "Already authenticated with Azure."
-else
+if ! az account show &> /dev/null; then
     echo "Authenticating with Azure CLI..."
-	az login
+	az login --use-device-code
 fi
 
 # Check and assign Storage Blob Data Contributor role to current user
-echo "Checking Storage Blob Data Contributor role assignment..."
 signed_user_id=$(az ad signed-in-user show --query id --output tsv)
 storage_resource_id=$(az storage account show --name "$storageAccountName" --resource-group "$resourceGroupName" --query id --output tsv)
 
 role_assignment=$(MSYS_NO_PATHCONV=1 az role assignment list --assignee $signed_user_id --role "Storage Blob Data Contributor" --scope $storage_resource_id --query "[].roleDefinitionId" -o tsv)
 if [ -z "$role_assignment" ]; then
-    echo "Assigning Storage Blob Data Contributor role to current user..."
+    echo "✓ Assigning Storage Blob Data Contributor role"
     MSYS_NO_PATHCONV=1 az role assignment create --assignee $signed_user_id --role "Storage Blob Data Contributor" --scope $storage_resource_id --output none
-    if [ $? -eq 0 ]; then
-        echo "Storage Blob Data Contributor role assigned successfully."
-        # Wait a bit for role assignment to propagate
-        echo "Waiting for role assignment to propagate..."
-        sleep 10
-    else
-        echo "Failed to assign Storage Blob Data Contributor role."
+    if [ $? -ne 0 ]; then
+        echo "✗ Failed to assign Storage Blob Data Contributor role"
         exit 1
     fi
-else
-    echo "User already has Storage Blob Data Contributor role."
+    sleep 10
 fi
 
 # Upload files to storage account
-# Using az storage blob upload-batch to upload files with Azure CLI authentication
-echo "Uploading call transcripts to storage account..."
 if [ -d "$extractedFolder1" ]; then
+	echo "✓ Uploading call transcripts"
 	az storage blob upload-batch \
 		--account-name "$storageAccountName" \
 		--destination "$containerName/$extractedFolder1" \
 		--source "$extractedFolder1" \
 		--auth-mode login \
 		--pattern '*' \
-		--overwrite
-	if [ $? -eq 0 ]; then
-		echo "✓ Call transcripts uploaded successfully"
-	else
+		--overwrite \
+		--output none
+	if [ $? -ne 0 ]; then
 		echo "✗ Failed to upload call transcripts"
 		exit 1
 	fi
-else
-	echo "Warning: $extractedFolder1 directory not found. Skipping upload."
 fi
 
-echo "Uploading audio data to storage account..."
 if [ -d "$extractedFolder2" ]; then
+	echo "✓ Uploading audio data"
 	az storage blob upload-batch \
 		--account-name "$storageAccountName" \
 		--destination "$containerName/$extractedFolder2" \
 		--source "$extractedFolder2" \
 		--auth-mode login \
 		--pattern '*' \
-		--overwrite
-	if [ $? -eq 0 ]; then
-		echo "✓ Audio data uploaded successfully"
-	else
+		--overwrite \
+		--output none
+	if [ $? -ne 0 ]; then
 		echo "✗ Failed to upload audio data"
 		exit 1
 	fi
-else
-	echo "Warning: $extractedFolder2 directory not found. Skipping upload."
 fi
 
 # Create custom data directories for user uploads
-echo "Creating custom data directories..."
 az storage fs directory create \
 	--account-name "$storageAccountName" \
 	--file-system "$containerName" \
 	--name custom_audiodata \
-	--auth-mode login 2>/dev/null || echo "custom_audiodata directory may already exist"
+	--auth-mode login --output none 2>/dev/null
 
 az storage fs directory create \
 	--account-name "$storageAccountName" \
 	--file-system "$containerName" \
 	--name custom_transcripts \
-	--auth-mode login 2>/dev/null || echo "custom_transcripts directory may already exist"
-
-echo "✓ Custom data directories created successfully"
-echo "Script completed successfully"
+	--auth-mode login --output none 2>/dev/null

infra/scripts/index_scripts/01_create_search_index.py

@@ -30,9 +30,6 @@
 
 INDEX_NAME = "call_transcripts_index"
 
-print("calling create_search_index()....")
-
-
 def create_search_index():
     """
     Creates or updates an Azure Cognitive Search index configured for:
@@ -105,7 +102,7 @@ def create_search_index():
     )
 
     result = index_client.create_or_update_index(index)
-    print(f"Search index '{result.name}' created or updated successfully.")
+    print(f"✓ Search index '{result.name}' created")
 
 
 create_search_index()

infra/scripts/index_scripts/02_create_cu_template_audio.py

@@ -36,9 +36,9 @@
     analyzer = client.get_analyzer_detail_by_id(ANALYZER_ID)
     if analyzer is not None:
         client.delete_analyzer(ANALYZER_ID)
-except Exception as e:
-    print(f"Analyzer with ID {ANALYZER_ID} was not found. Proceeding to create a new one.")
+except Exception:
+    pass
 
 response = client.begin_create_analyzer(ANALYZER_ID, analyzer_template_path=ANALYZER_TEMPLATE_FILE)
 result = client.poll_result(response)
-print(f"Analyzer with ID {ANALYZER_ID} created successfully.")
+print(f"✓ Analyzer '{ANALYZER_ID}' created")

infra/scripts/index_scripts/02_create_cu_template_text.py

@@ -31,9 +31,9 @@
     analyzer = client.get_analyzer_detail_by_id(ANALYZER_ID)
     if analyzer is not None:
         client.delete_analyzer(ANALYZER_ID)
-except Exception as e:
-    print(f"Analyzer with ID {ANALYZER_ID} was not found. Proceeding to create a new one.")
+except Exception:
+    pass
 
 response = client.begin_create_analyzer(ANALYZER_ID, analyzer_template_path=ANALYZER_TEMPLATE_FILE)
 result = client.poll_result(response)
-print(f"Analyzer with ID {ANALYZER_ID} created successfully.")
+print(f"✓ Analyzer '{ANALYZER_ID}' created")