Complete DDL Compatibility Enhancements (Feature 036) - Implementation, Tests, and Documentation

Author: tom-dyar

AGENTS.md

@@ -1,6 +1,6 @@
 # iris-pgwire-gh Development Guidelines
 
-Auto-generated from all feature plans. Last updated: 2026-01-02
+Auto-generated from all feature plans. Last updated: 2026-01-17
 
 ## Active Technologies
 - Python 3.11 + python>=3.11, psycopg[binary], iris-devtester, intersystems-irispython (026-address-gaps-in)
@@ -10,6 +10,7 @@ Auto-generated from all feature plans. Last updated: 2026-01-02
 - Python 3.11 + intersystems-irispython, psycopg[binary], iris-devtester (035-number-1-short)
 - Python 3.11 + psycopg[binary], intersystems-irispython, iris-devtester (036-address-all-6)
 - InterSystems IRIS (via pgwire) (036-address-all-6)
+- Python 3.11 + psycopg[binary], intersystems-irispython, iris-devtester (036-address-all-6)
 
 - Python 3.11+ + `iris-devtester`, `intersystems-irispython`, `psycopg[binary]` (033-devtester-skills)
 
@@ -30,8 +31,8 @@ Python 3.11+: Follow standard conventions
 
 ## Recent Changes
 - 036-address-all-6: Added Python 3.11 + psycopg[binary], intersystems-irispython, iris-devtester
+- 036-address-all-6: Added Python 3.11 + psycopg[binary], intersystems-irispython, iris-devtester
 - 035-number-1-short: Added Python 3.11 + intersystems-irispython, psycopg[binary], iris-devtester
-- 034-issues-that-likely: Added Python 3.11 + intersystems-irispython, psycopg[binary], iris-devtester
 
 
 <!-- MANUAL ADDITIONS START -->

CHANGELOG.md

@@ -5,6 +5,16 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.0.7] - 2026-01-17
+
+### Added
+- **PostgreSQL DDL Compatibility Enhancement**: Implemented automatic interception and transformation of PostgreSQL-specific DDL constructs to enable seamless migrations.
+- **Generated Column Stripping**: Added support for automatically removing `GENERATED ALWAYS AS ... STORED` column definitions from `CREATE TABLE` statements.
+- **Enum Type Registration**: Added logic to skip `CREATE TYPE ... AS ENUM`, register the type name, and automatically map subsequent columns using that type to `VARCHAR(64)`.
+- **Index Dependency Tracking**: Implemented `SkippedTableSet` to track tables whose creation was skipped, ensuring dependent `CREATE INDEX` statements are also skipped.
+- **Strict DDL Mode**: Added a configurable `strict_ddl` flag (default `false`) to control whether unsupported constructs should be skipped with a warning or raise an error.
+- **Construct Stripping**: Added automatic stripping of `USING btree`, PostgreSQL type casts (`::type`), and `WITH (fillfactor)` from DDL statements.
+
 ## [1.0.6] - 2026-01-16
 
 ### Added

KNOWN_LIMITATIONS.md

@@ -49,6 +49,30 @@ Based on comprehensive research of 9 major implementations (November 2025):
 
 ---
 
+## 🟢 PostgreSQL DDL Compatibility
+
+**Status**: Enhanced Compatibility ✅
+**Feature**: Automated transformations for PostgreSQL-specific DDL
+
+### Capabilities
+
+IRIS PGWire automatically handles several common PostgreSQL DDL constructs that are natively unsupported by IRIS:
+
+- ✅ **Generated Columns**: `GENERATED ALWAYS AS ... STORED` are automatically stripped.
+- ✅ **Enum Types**: `CREATE TYPE ... ENUM` are registered and columns mapped to `VARCHAR(64)`.
+- ✅ **Storage Parameters**: `WITH (fillfactor = ...)` and `SET (fillfactor)` are stripped/skipped.
+- ✅ **Index Methods**: `USING btree` is automatically removed.
+- ✅ **Casts**: `'value'::text` syntax in defaults is stripped.
+- ✅ **CHECK Constraints**: `ADD CONSTRAINT ... CHECK` are skipped with a warning.
+
+### Configuration
+
+Use the `strict_ddl` flag to control this behavior:
+- `strict_ddl = false` (default): Skip/transform with a warning.
+- `strict_ddl = true`: Raise an error for unsupported DDL.
+
+---
+
 ## 🟡 INFORMATION_SCHEMA Compatibility
 
 **Severity**: Medium

README.md

@@ -108,7 +108,7 @@ See [Client Compatibility Guide](https://github.com/intersystems-community/iris-
 
 - **pgvector Syntax**: Use familiar `<=>` and `<#>` operators - auto-translated to IRIS VECTOR_COSINE/DOT_PRODUCT. HNSW indexes provide 5× speedup on 100K+ vectors. See [Vector Operations Guide](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/VECTOR_PARAMETER_BINDING.md)
 
-- **ORM Compatibility**: Automatic `public` ↔ `SQLUser` schema mapping for Prisma, SQLAlchemy, Drizzle, Hibernate introspection. Catalog emulation with 6 pg_catalog tables + 5 functions. See [pg_catalog Documentation](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/PG_CATALOG.md)
+- **ORM & DDL Compatibility**: Automatic `public` ↔ `SQLUser` schema mapping and PostgreSQL DDL transformations (stripping `fillfactor`, `GENERATED` columns, `USING btree`, etc.) for seamless migrations. See [DDL Compatibility Guide](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/DDL_COMPATIBILITY.md)
 
 - **Enterprise Security**: SCRAM-SHA-256, OAuth 2.0, IRIS Wallet authentication. Industry-standard security matching PgBouncer, YugabyteDB. See [Deployment Guide](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/DEPLOYMENT.md)
 
@@ -201,7 +201,8 @@ async def get_user(user_id: int, db: AsyncSession = Depends(get_db)):
 - **[BI Tools Setup](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/BI_TOOLS.md)** - Superset, Metabase, Grafana integration
 
 ### Features & Capabilities
-- **[Features Overview](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/FEATURES_OVERVIEW.md)** - pgvector, ORM compatibility, authentication
+- **[Features Overview](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/FEATURES_OVERVIEW.md)** - pgvector, ORM compatibility, DDL transformations, authentication
+- **[DDL Compatibility](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/DDL_COMPATIBILITY.md)** - Automatic handling of PostgreSQL-specific DDL (fillfactor, generated columns, enums)
 - **[pg_catalog Support](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/PG_CATALOG.md)** - 6 catalog tables + 5 functions for ORM introspection
 - **[Vector Operations](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/VECTOR_PARAMETER_BINDING.md)** - High-dimensional vectors, parameter binding
 - **[Client Compatibility](https://github.com/intersystems-community/iris-pgwire/blob/main/docs/CLIENT_RECOMMENDATIONS.md)** - 171 tests across 8 languages

benchmarks/3way_comparison.py

@@ -11,32 +11,31 @@
 Per FR-006: Aborts on connection failure.
 """
 
-import sys
 import argparse
+import sys
 from pathlib import Path
-from typing import Dict, List
 
 # Add benchmarks to path
 sys.path.insert(0, str(Path(__file__).parent.parent))
 
 from benchmarks.config import BenchmarkConfiguration, ConnectionConfig
-from benchmarks.runner import BenchmarkRunner
+from benchmarks.executors.dbapi_executor import DbapiExecutor
 from benchmarks.executors.pgwire_executor import PGWireExecutor
 from benchmarks.executors.postgres_executor import PostgresExecutor
-from benchmarks.executors.dbapi_executor import DbapiExecutor
+from benchmarks.output.json_exporter import export_json
+from benchmarks.output.table_exporter import export_table
+from benchmarks.runner import BenchmarkRunner
 from benchmarks.test_data.query_templates import (
+    COMPLEX_QUERIES,
     SIMPLE_QUERIES,
     VECTOR_QUERIES,
-    COMPLEX_QUERIES,
-    format_query_for_method
+    format_query_for_method,
 )
 from benchmarks.test_data.vector_generator import generate_query_vector, vector_to_text
-from benchmarks.output.json_exporter import export_json
-from benchmarks.output.table_exporter import export_table
 from benchmarks.validate_connections import validate_all_connections
 
 
-def create_test_queries(method: str, dimensions: int = 1024) -> Dict[str, List[str]]:
+def create_test_queries(method: str, dimensions: int = 1024) -> dict[str, list[str]]:
     """
     Create test queries for a specific database method.
 

benchmarks/async_sqlalchemy_stress_test.py

@@ -16,12 +16,12 @@
     python3 benchmarks/async_sqlalchemy_stress_test.py
 """
 
+import asyncio
 from random import randint, random
 from time import time_ns
-import asyncio
-import psycopg
 
-from sqlalchemy import MetaData, Table, Column, Integer, String, text, bindparam
+import psycopg
+from sqlalchemy import Column, Integer, MetaData, String, Table, bindparam, text
 from sqlalchemy.ext.asyncio import create_async_engine
 
 # Test configuration

benchmarks/config.py

@@ -6,7 +6,6 @@
 """
 
 from dataclasses import dataclass
-from typing import Dict, List, Optional
 from datetime import datetime
 from enum import Enum
 
@@ -34,11 +33,11 @@ class ConnectionConfig:
     host: str
     port: int
     database: str
-    username: Optional[str] = None
-    password: Optional[str] = None
+    username: str | None = None
+    password: str | None = None
     connection_timeout: float = 10.0
 
-    def validate(self) -> List[str]:
+    def validate(self) -> list[str]:
         """
         Validate configuration parameters.
 
@@ -69,9 +68,9 @@ class BenchmarkConfiguration:
     concurrent_connections: int = 1
     warmup_queries: int = 100
     random_seed: int = 42
-    connection_configs: Optional[Dict[str, ConnectionConfig]] = None
+    connection_configs: dict[str, ConnectionConfig] | None = None
 
-    def validate(self) -> List[str]:
+    def validate(self) -> list[str]:
         """
         Validate configuration per functional requirements.
 
@@ -120,11 +119,11 @@ class PerformanceResult:
     timestamp: datetime
     elapsed_ms: float
     success: bool
-    error_message: Optional[str] = None
+    error_message: str | None = None
     row_count: int = 0
-    resource_usage: Optional[Dict] = None
+    resource_usage: dict | None = None
 
-    def validate(self) -> List[str]:
+    def validate(self) -> list[str]:
         """
         Validate result integrity.
 
@@ -165,9 +164,9 @@ class MethodResults:
     latency_p50_ms: float
     latency_p95_ms: float
     latency_p99_ms: float
-    by_category: Dict[str, CategoryMetrics]
+    by_category: dict[str, CategoryMetrics]
 
-    def validate(self) -> List[str]:
+    def validate(self) -> list[str]:
         """
         Validate aggregated results.
 
@@ -205,12 +204,12 @@ class BenchmarkReport:
     start_time: datetime
     end_time: datetime
     total_duration_seconds: float
-    method_results: Dict[str, MethodResults]
-    raw_results: List[PerformanceResult]
-    validation_errors: List[str]
+    method_results: dict[str, MethodResults]
+    raw_results: list[PerformanceResult]
+    validation_errors: list[str]
     state: BenchmarkState = BenchmarkState.INITIALIZING
 
-    def to_json(self) -> Dict:
+    def to_json(self) -> dict:
         """
         Export report as JSON (FR-010).
 
@@ -239,7 +238,7 @@ def to_json(self) -> Dict:
             }
         }
 
-    def to_table_rows(self) -> List[List]:
+    def to_table_rows(self) -> list[list]:
         """
         Export report as table rows for console display (FR-010).
 

benchmarks/executors/dbapi_executor.py

@@ -5,7 +5,7 @@
 Pattern from: /Users/tdyar/ws/rag-templates/common/iris_connection_manager.py
 """
 
-from typing import Any, Optional
+from typing import Any
 
 
 class DbapiExecutor:
@@ -34,7 +34,7 @@ def __init__(
         self.namespace = namespace
         self.username = username
         self.password = password
-        self.connection: Optional[Any] = None
+        self.connection: Any | None = None
 
     def connect(self):
         """Establish DBAPI connection to IRIS."""
@@ -73,7 +73,7 @@ def execute(self, query: str) -> Any:
             results = cursor.fetchall()
             cursor.close()
             return results
-        except Exception as e:
+        except Exception:
             cursor.close()
             # Re-raise without printing (error logging handled by runner if needed)
             raise

benchmarks/executors/pgwire_executor.py

@@ -4,8 +4,9 @@
 Executes queries via psycopg3 to PGWire server.
 """
 
+from typing import Any
+
 import psycopg
-from typing import Any, Optional
 
 
 class PGWireExecutor:
@@ -31,7 +32,7 @@ def __init__(
         self.port = port
         self.database = database
         self.timeout_seconds = timeout_seconds
-        self.connection: Optional[psycopg.Connection] = None
+        self.connection: psycopg.Connection | None = None
 
     def connect(self):
         """Establish connection to PGWire server."""
@@ -68,7 +69,7 @@ def execute(self, query: str) -> Any:
             results = cursor.fetchall()
             cursor.close()
             return results
-        except Exception as e:
+        except Exception:
             cursor.close()
             raise
 

benchmarks/executors/postgres_executor.py

@@ -4,8 +4,9 @@
 Executes queries via psycopg3 against native PostgreSQL with pgvector.
 """
 
+from typing import Any
+
 import psycopg
-from typing import Any, Optional
 
 
 class PostgresExecutor:
@@ -34,7 +35,7 @@ def __init__(
         self.database = database
         self.username = username
         self.password = password
-        self.connection: Optional[psycopg.Connection] = None
+        self.connection: psycopg.Connection | None = None
 
     def connect(self):
         """Establish connection to PostgreSQL."""
@@ -76,7 +77,7 @@ def execute(self, query: str) -> Any:
             results = cursor.fetchall()
             cursor.close()
             return results
-        except Exception as e:
+        except Exception:
             cursor.close()
             raise