Bug fixes for deployment

Author: BenConstable9

.vscode/tasks.json

@@ -5,7 +5,7 @@
       "isBackground": true,
       "label": "func: host start",
       "options": {
-        "cwd": "${workspaceFolder}/ai_search_with_adi_function_app"
+        "cwd": "${workspaceFolder}/ai_search_with_image_processing"
       },
       "problemMatcher": "$func-python-watch",
       "type": "func"

README.md

@@ -11,7 +11,7 @@ It is intended that the plugins and skills provided in this repository, are adap
 ## Components
 
 - `./text_2_sql` contains an three Multi-Shot implementations for Text2SQL generation and querying which can be used to answer questions backed by a database as a knowledge base. A **prompt based** and **vector based** approach are shown, both of which exhibit great performance in answering sql queries. Additionally, a further iteration on the vector based approach is shown which uses a **query cache** to further speed up generation.  With these plugins, your RAG application can now access and pull data from any SQL table exposed to it to answer questions.
-- `./adi_function_app` contains code for linking **Azure Document Intelligence** with AI Search to process complex documents with charts and images, and uses **multi-modal models (gpt4o)** to interpret and understand these. With this custom skill, the RAG application can **draw insights from complex charts** and images during the vector search. This function app also contains a **Semantic Text Chunking** method that aims to intelligently group similar sentences, retaining figures and tables together, whilst separating out distinct sentences.
+- `./image_processing` contains code for linking **Azure Document Intelligence** with AI Search to process complex documents with charts and images, and uses **multi-modal models (gpt4o)** to interpret and understand these. With this custom skill, the RAG application can **draw insights from complex charts** and images during the vector search. This function app also contains a **Semantic Text Chunking** method that aims to intelligently group similar sentences, retaining figures and tables together, whilst separating out distinct sentences.
 - `./deploy_ai_search` provides an easy Python based utility for deploying an index, indexer and corresponding skillset for AI Search and for Text2SQL.
 
 The above components have been successfully used on production RAG projects to increase the quality of responses.

adi_function_app/.env

@@ -20,6 +20,7 @@
 )
 import os
 from text_2_sql_core.utils.database import DatabaseEngine
+from text_2_sql_core.connectors.factory import ConnectorFactory
 
 
 class Text2SqlColumnValueStoreAISearch(AISearch):
@@ -43,25 +44,9 @@ def __init__(
             os.environ["Text2Sql__DatabaseEngine"].upper()
         ]
 
-        self.parsing_mode = BlobIndexerParsingMode.JSON_LINES
-
-    @property
-    def excluded_fields_for_database_engine(self):
-        """A method to get the excluded fields for the database engine."""
-
-        all_engine_specific_fields = ["Warehouse", "Database", "Catalog"]
-        if self.database_engine == DatabaseEngine.SNOWFLAKE:
-            engine_specific_fields = ["Warehouse", "Database"]
-        elif self.database_engine == DatabaseEngine.TSQL:
-            engine_specific_fields = ["Database"]
-        elif self.database_engine == DatabaseEngine.DATABRICKS:
-            engine_specific_fields = ["Catalog"]
+        self.database_connector = ConnectorFactory.get_database_connector()
 
-        return [
-            field
-            for field in all_engine_specific_fields
-            if field not in engine_specific_fields
-        ]
+        self.parsing_mode = BlobIndexerParsingMode.JSON_LINES
 
     def get_index_fields(self) -> list[SearchableField]:
         """This function returns the index fields for sql index.
@@ -123,7 +108,7 @@ def get_index_fields(self) -> list[SearchableField]:
         fields = [
             field
             for field in fields
-            if field.name not in self.excluded_fields_for_database_engine
+            if field.name not in self.database_connector.excluded_engine_specific_fields
         ]
 
         return fields
@@ -238,7 +223,7 @@ def get_indexer(self) -> SearchIndexer:
             field_mapping
             for field_mapping in indexer.output_field_mappings
             if field_mapping.target_field_name
-            not in self.excluded_fields_for_database_engine
+            not in self.database_connector.excluded_engine_specific_fields
         ]
 
         return indexer

deploy_ai_search/src/deploy_ai_search/text_2_sql_column_value_store.py

@@ -346,7 +346,7 @@ def get_indexer(self) -> SearchIndexer:
             field_mapping
             for field_mapping in indexer.output_field_mappings
             if field_mapping.target_field_name
-            not in self.excluded_fields_for_database_engine
+            not in self.database_connector.excluded_engine_specific_fields
         ]
 
         return indexer

deploy_ai_search/src/deploy_ai_search/text_2_sql_schema_store.py

@@ -4,7 +4,7 @@ To get started, perform the following steps:
 
 1. Setup Azure OpenAI in your subscription with **gpt-4o-mini** & an embedding model, an Python Function App, AI Search and a storage account.
 2. Clone this repository and deploy the AI Search rag documents indexes from `deploy_ai_search`.
-3. Run `uv sync` within the adi_function_app directory to install dependencies.
+3. Run `uv sync` within the image_processing directory to install dependencies.
 4. Configure the environment variables of the function app based on the provided sample
 5. Package your Azure Function and upload to your Function App
 6. Upload a document for indexing or send a direct HTTP request to the Azure Function.