fix: merging dev changes to main branch

[Roopan-Microsoft]

Purpose

• This pull request introduces significant improvements to the Azure deployment workflow, enhances the Azure Cognitive Search indexing pipeline, and increases security by removing hardcoded database credentials. The changes include a new Azure Function for combining page and chunk information, updates to the search skillset to use this function, and various workflow and documentation updates for better maintainability and compatibility.

Azure Cognitive Search Pipeline Enhancements:

• Added a new Azure Function combine_pages_and_chunknos to merge page texts with their chunk numbers, exposed via an HTTP endpoint for use in the search skillset. (code/backend/batch/combine_pages_chunknos.py, code/backend/batch/function_app.py) [1] [2] [3]
• Modified the search skillset to include a WebApiSkill that calls the new function, and updated downstream skills and index projections to use the combined page/chunk data structure. Also added a ShaperSkill for structuring metadata. (code/backend/batch/utilities/integrated_vectorization/azure_search_skillset.py) [1] [2] [3]
• Updated indexer field mappings to use a base64 encoding function for the id field. (code/backend/batch/utilities/integrated_vectorization/azure_search_indexer.py) [1] [2]

Azure Deployment Workflow Improvements:

• Increased GPT_MIN_CAPACITY from 30 to 150 and added new environment variables and outputs for principal and resource group information in the CI workflow. (.github/workflows/ci.yml) [1] [2] [3] [4] [5]
• Enhanced resource group creation to include a CreatedBy="Pipeline" tag. (.github/workflows/ci.yml)
• Updated the database population step to use Azure AD authentication (service principal and token) instead of hardcoded credentials, and removed the output of sensitive information from logs. (.github/workflows/ci.yml, Makefile) [1] [2] [3] [4]
• Refined environment variable propagation and output handling for downstream jobs. (.github/workflows/ci.yml) [1] [2]

Security and Maintenance:

• Removed hardcoded PostgreSQL usernames and passwords from workflow logs and the Makefile. (.github/workflows/ci.yml, Makefile) [1] [2]
• Removed a redundant azd-login Makefile target and improved the destroy target to select the correct environment. (Makefile) [1] [2]

Documentation and Metadata Updates:

• Added a section to the README.md listing supported Azure regions and clarifying region/location usage in deployments.
• Updated azure.yaml to require azd version 1.18.0 or higher.
• Changed the release workflow trigger to run on completion of the "Validate Deployment" workflow instead of "CI". (.github/workflows/create-release.yml)

These changes collectively improve the security, maintainability, and scalability of the solution, particularly in the way document content is indexed and how deployments are managed in Azure.

Does this introduce a breaking change?

• Yes
• No

How to Test

• Get the code

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install

• Test the code

What to Check

Verify that the following are valid

• ...

Other Information