Normalise environment variables (#144)

Author: BenConstable9

deploy_ai_search/README.md

@@ -1,8 +1,8 @@
 # AI Search Indexing Pre-built Index Setup
 
-The associated scripts in this portion of the repository contains pre-built scripts to deploy the skillset with Azure Document Intelligence.
+The associated scripts in this portion of the repository contains pre-built scripts to deploy the skillsets needed for both Text2SQL and Image Processing.
 
-## Steps for Rag Documents Index Deployment (For Unstructured RAG)
+## Steps for Rag Documents Index Deployment (For Image Processing)
 
 1. Update `.env` file with the associated values. Not all values are required dependent on whether you are using System / User Assigned Identities or a Key based authentication.
 2. Adjust `rag_documents.py` with any changes to the index / indexer. The `get_skills()` method implements the skills pipeline. Make any adjustments here in the skills needed to enrich the data source.
@@ -13,7 +13,7 @@ The associated scripts in this portion of the repository contains pre-built scri
     - `rebuild`. Whether to delete and rebuild the index.
     - `suffix`. Optional parameter that will apply a suffix onto the deployed index and indexer. This is useful if you want deploy a test version, before overwriting the main version.
 
-## Steps for Text2SQL Index Deployment (For Structured RAG)
+## Steps for Text2SQL Index Deployment (For Text2SQL)
 
 ### Schema Store Index
 

text_2_sql/.env.example

@@ -1,6 +1,11 @@
 # Environment variables for Text2SQL
 IdentityType=<identityType> # system_assigned or user_assigned or key
 
+Text2Sql__DatabaseEngine=<DatabaseEngine> # TSQL or PostgreSQL or Snowflake or Databricks
+Text2Sql__UseQueryCache=<Determines if the Query Cache will be used to speed up query generation. Defaults to True.> # True or False
+Text2Sql__PreRunQueryCache=<Determines if the results from the Query Cache will be pre-run to speed up answer generation. Defaults to True.> # True or False
+Text2Sql__UseColumnValueStore=<Determines if the Column Value Store will be used for schema selection Defaults to True.> # True or False
+
 # Open AI Connection Details
 OpenAI__CompletionDeployment=<openAICompletionDeploymentId. Used for data dictionary creator>
 OpenAI__MiniCompletionDeployment=<OpenAI__MiniCompletionDeploymentId. Used for agentic text2sql>
@@ -17,17 +22,20 @@ AIService__AzureSearchOptions__Text2SqlQueryCache__Index=<Query cache index name
 AIService__AzureSearchOptions__Text2SqlQueryCache__SemanticConfig=<Query cache semantic config. Default is created as "text-2-sql-query-cache-semantic-config">
 AIService__AzureSearchOptions__Text2SqlColumnValueStore__Index=<Column value store index name. Default is created as "text-2-sql-column-value-store-index">
 
-# All SQL Engine specific connection details
-Text2Sql__DatabaseName=<databaseName>
+# TSQL
+Text2Sql__Tsql__ConnectionString=<Tsql databaseConnectionString if using Tsql Data Source>
+Text2Sql__Tsql__Database=<Tsql database if using Tsql Data Source>
 
-# TSQL or PostgreSQL Specific Connection Details
-Text2Sql__DatabaseConnectionString=<databaseConnectionString>
+# PostgreSQL Specific Connection Details
+Text2Sql__Postgresql__ConnectionString=<Postgresql databaseConnectionString if using Postgresql Data Source>
+Text2Sql__Postgresql__Database=<Postgresql database if using Postgresql Data Source>
 
 # Snowflake Specific Connection Details
 Text2Sql__Snowflake__User=<snowflakeUser if using Snowflake Data Source>
 Text2Sql__Snowflake__Password=<snowflakePassword if using Snowflake Data Source>
 Text2Sql__Snowflake__Account=<snowflakeAccount if using Snowflake Data Source>
 Text2Sql__Snowflake__Warehouse=<snowflakeWarehouse if using Snowflake Data Source>
+Text2Sql__Snowflake__Database=<snowflakeDatabase if using Snowflake Data Source>
 
 # Databricks Specific Connection Details
 Text2Sql__Databricks__Catalog=<databricksCatalog if using Databricks Data Source with Unity Catalog>

text_2_sql/GETTING_STARTED.md

@@ -2,12 +2,25 @@
 
 To get started, perform the following steps:
 
+**Execute the following commands in the `deploy_ai_search` directory:**
+
 1. Setup Azure OpenAI in your subscription with **gpt-4o-mini** & an embedding model, alongside a SQL Server sample database, AI Search and a storage account.
-2. Clone this repository and deploy the AI Search text2sql indexes from `deploy_ai_search`.
-3. Run `uv sync` within the text_2_sql directory to install dependencies.
+2. Create your `.env` file based on the provided sample `deploy_ai_search/.env.example`. Place this file in the same place in `deploy_ai_search/.env`.
+3. Clone this repository and deploy the AI Search text2sql indexes from `deploy_ai_search`. See the instructions in the **Steps for Text2SQL Index Deployment (For Structured RAG)** section of the `deploy_ai_search/README.md`.
+
+**Execute the following commands in the `text_2_sql_core` directory:**
+
+4. Create your `.env` file based on the provided sample `text_2_sql/.env.example`. Place this file in the same place in `text_2_sql/.env`.
+5. Run `uv sync` within the text_2_sql directory to install dependencies.
+    - Install the optional dependencies if you need a database connector other than TSQL. `uv sync --extra <DATABASE ENGINE>`
+    - See the supported connectors in `text_2_sql_core/src/text_2_sql_core/connectors`.
+6. Create your `.env` file based on the provided sample `text_2_sql/.env.example`. Place this file in the same place in `text_2_sql/.env`.
+7. Generate a data dictionary for your target server using the instructions in the **Running** section of the `data_dictionary/README.md`.
+8. Upload these generated data dictionaries files to the relevant containers in your storage account. Wait for them to be automatically indexed with the included skillsets.
+
+**Execute the following commands in the `autogen` directory:**
+
+9. Run `uv sync` within the text_2_sql directory to install dependencies.
     - Install the optional dependencies if you need a database connector other than TSQL. `uv sync --extra <DATABASE ENGINE>`
     - See the supported connectors in `text_2_sql_core/src/text_2_sql_core/connectors`.
-4. Create your `.env` file based on the provided sample `.env.example`. Place this file in the same place as the `.env.example`.
-5. Generate a data dictionary for your target server using the instructions in the **Running** section of the `data_dictionary/README.md`.
-6. Upload these generated data dictionaries files to the relevant containers in your storage account. Wait for them to be automatically indexed with the included skillsets.
-7. Navigate to `autogen` directory to view the AutoGen implementation. Follow the steps in `Iteration 5 - Agentic Vector Based Text2SQL.ipynb` to get started.
+10. Navigate to `autogen` directory to view the AutoGen implementation. Follow the steps in `Iteration 5 - Agentic Vector Based Text2SQL.ipynb` to get started.

text_2_sql/__init__.py

@@ -215,8 +215,8 @@
         "        \n",
         "        # Update database connection string for current database\n",
         "        db_path = DATABASE_DIR / db_id / f\"{db_id}.sqlite\"\n",
-        "        os.environ[\"Text2Sql__DatabaseConnectionString\"] = str(db_path)\n",
-        "        os.environ[\"Text2Sql__DatabaseName\"] = db_id\n",
+        "        os.environ[\"Text2Sql__Tsql__ConnectionString\"] = str(db_path)\n",
+        "        os.environ[\"Text2Sql__Database\"] = db_id\n",
         "        \n",
         "        sql = await generate_sql(question)\n",
         "        predictions.append(f\"{sql}\\t{db_id}\")\n",

text_2_sql/autogen/evaluate_autogen_text2sql.ipynb

@@ -219,12 +219,12 @@ async def consume_inner_messages_from_agentic_flow(
 
             # Add database connection info to injected parameters
             query_params = injected_parameters.copy() if injected_parameters else {}
-            if "Text2Sql__DatabaseConnectionString" in os.environ:
+            if "Text2Sql__Tsql__ConnectionString" in os.environ:
                 query_params["database_connection_string"] = os.environ[
-                    "Text2Sql__DatabaseConnectionString"
+                    "Text2Sql__Tsql__ConnectionString"
                 ]
-            if "Text2Sql__DatabaseName" in os.environ:
-                query_params["database_name"] = os.environ["Text2Sql__DatabaseName"]
+            if "Text2Sql__Tsql__Database" in os.environ:
+                query_params["database_name"] = os.environ["Text2Sql__Tsql__Database"]
 
             # Launch tasks for each sub-query
             inner_solving_generators.append(

text_2_sql/autogen/src/autogen_text_2_sql/custom_agents/parallel_query_solving_agent.py

@@ -45,27 +45,27 @@ def __init__(self, **kwargs: dict):
         self.set_mode()
 
         # Store original environment variables
-        self.original_db_conn = os.environ.get("Text2Sql__DatabaseConnectionString")
-        self.original_db_name = os.environ.get("Text2Sql__DatabaseName")
+        self.original_db_conn = os.environ.get("Text2Sql__Tsql__ConnectionString")
+        self.original_db_name = os.environ.get("Text2Sql__Tsql__Database")
 
     def _update_environment(self, injected_parameters: dict = None):
         """Update environment variables with injected parameters."""
         if injected_parameters:
             if "database_connection_string" in injected_parameters:
-                os.environ["Text2Sql__DatabaseConnectionString"] = injected_parameters[
+                os.environ["Text2Sql__Tsql__ConnectionString"] = injected_parameters[
                     "database_connection_string"
                 ]
             if "database_name" in injected_parameters:
-                os.environ["Text2Sql__DatabaseName"] = injected_parameters[
+                os.environ["Text2Sql__Tsql__Database"] = injected_parameters[
                     "database_name"
                 ]
 
     def _restore_environment(self):
         """Restore original environment variables."""
         if self.original_db_conn:
-            os.environ["Text2Sql__DatabaseConnectionString"] = self.original_db_conn
+            os.environ["Text2Sql__Tsql__ConnectionString"] = self.original_db_conn
         if self.original_db_name:
-            os.environ["Text2Sql__DatabaseName"] = self.original_db_name
+            os.environ["Text2Sql__Tsql__Database"] = self.original_db_name
 
     def set_mode(self):
         """Set the mode of the plugin based on the environment variables."""

text_2_sql/autogen/src/autogen_text_2_sql/inner_autogen_text_2_sql.py

@@ -224,7 +224,11 @@ If there is no pre-built script for your database engine, take one of the above
 
 ## Running
 
-1. Create your `.env` file based on the provided sample `.env.example`. Place this file in the same place as the `.env.example`.
+To generate a data dictionary, perform the following steps:
+
+1. Create your `.env` file based on the provided sample `text_2_sql/.env.example`. Place this file in the same place in `text_2_sql/.env`.
+
+**Execute the following commands in the `text_2_sql_core` directory:**
 2. Package and install the `text_2_sql_core` library. See [build](https://docs.astral.sh/uv/concepts/projects/build/) if you want to build as a wheel and install on an agent. Or you can run from within a `uv` environment and skip packaging.
     - Install the optional dependencies if you need a database connector other than TSQL. `uv sync --extra <DATABASE ENGINE>`
 3. Run `data_dictionary <DATABASE ENGINE>`

text_2_sql/data_dictionary/README.md

@@ -16,6 +16,11 @@ def __init__(self):
 
         self.database_engine = DatabaseEngine.POSTGRESQL
 
+    @property
+    def engine_specific_rules(self) -> str:
+        """Get the engine specific rules."""
+        return ""
+
     @property
     def engine_specific_fields(self) -> list[str]:
         """Get the engine specific fields."""
@@ -61,10 +66,10 @@ async def query_execution(
         """
         logging.info(f"Running query: {sql_query}")
         results = []
-        connection_string = os.environ["Text2Sql__DatabaseConnectionString"]
+        connection_string = os.environ["Text2Sql__Postgresql__ConnectionString"]
 
         # Establish an asynchronous connection to the PostgreSQL database
-        async with psycopg.AsyncConnection.connect(connection_string) as conn:
+        async with await psycopg.AsyncConnection.connect(connection_string) as conn:
             # Create an asynchronous cursor
             async with conn.cursor() as cursor:
                 await cursor.execute(sql_query)

text_2_sql/text_2_sql_core/src/text_2_sql_core/connectors/postgresql_sql.py

@@ -100,7 +100,7 @@ async def query_execution(
             password=os.environ["Text2Sql__Snowflake__Password"],
             account=os.environ["Text2Sql__Snowflake__Account"],
             warehouse=os.environ["Text2Sql__Snowflake__Warehouse"],
-            database=os.environ["Text2Sql__DatabaseName"],
+            database=os.environ["Text2Sql__Snowflake__Database"],
         )
 
         try: