feat: topological sorting and foreign key retrieval enhancements (#260)

Introduce methods for topological sorting of tables and retrieval of foreign key metadata across multiple database adapters. Update metadata class patterns and improve test isolation in examples. Add integration tests to validate new functionalities.

Author: cofin

AGENTS.md

@@ -451,6 +451,58 @@ if supports_where(obj):
     result = obj.where("condition")
 ```
 
+### Mypyc-Compatible Metadata Class Pattern
+
+When defining data-holding classes intended for core modules (`sqlspec/core/`, `sqlspec/driver/`) that will be compiled with MyPyC, use regular classes with `__slots__` and explicitly implement `__init__`, `__repr__`, `__eq__`, and `__hash__`. This approach ensures optimal performance and MyPyC compatibility, as `dataclasses` are not directly supported by MyPyC for compilation.
+
+**Key Principles:**
+
+- **`__slots__`**: Reduces memory footprint and speeds up attribute access.
+- **Explicit `__init__`**: Defines the constructor for the class.
+- **Explicit `__repr__`**: Provides a clear string representation for debugging.
+- **Explicit `__eq__`**: Enables correct equality comparisons.
+- **Explicit `__hash__`**: Makes instances hashable, allowing them to be used in sets or as dictionary keys. The hash implementation should be based on all fields that define the object's identity.
+
+**Example Implementation:**
+
+```python
+class MyMetadata:
+    __slots__ = ("field1", "field2", "optional_field")
+
+    def __init__(self, field1: str, field2: int, optional_field: str | None = None) -> None:
+        self.field1 = field1
+        self.field2 = field2
+        self.optional_field = optional_field
+
+    def __repr__(self) -> str:
+        return f"MyMetadata(field1={self.field1!r}, field2={self.field2!r}, optional_field={self.optional_field!r})"
+
+    def __eq__(self, other: object) -> bool:
+        if not isinstance(other, MyMetadata):
+            return NotImplemented
+        return (
+            self.field1 == other.field1
+            and self.field2 == other.field2
+            and self.optional_field == other.optional_field
+        )
+
+    def __hash__(self) -> int:
+        return hash((self.field1, self.field2, self.optional_field))
+```
+
+**When to Use:**
+
+- For all new data-holding classes in performance-critical paths (e.g., `sqlspec/driver/_common.py`).
+- When MyPyC compilation is enabled for the module containing the class.
+
+**Anti-Patterns to Avoid:**
+
+- Using `@dataclass` decorators for classes intended for MyPyC compilation.
+- Omitting `__slots__` when defining performance-critical data structures.
+- Relying on default `__eq__` or `__hash__` behavior for complex objects, especially for equality comparisons in collections.
+
+---
+
 ### Performance Patterns (MANDATORY)
 
 **PERF401 - List Operations**:

docs/examples/usage/usage_drivers_and_querying_10.py

@@ -12,30 +12,57 @@
 
 def test_example_10_duckdb_config(tmp_path: Path) -> None:
     # start-example
+    import tempfile
+
     from sqlspec import SQLSpec
     from sqlspec.adapters.duckdb import DuckDBConfig
 
-    spec = SQLSpec()
-    # In-memory
-    config = DuckDBConfig()
-
-    # Persistent
-    database_file = tmp_path / "analytics.duckdb"
-    config = DuckDBConfig(pool_config={"database": database_file.name, "read_only": False})
-
-    with spec.provide_session(config) as session:
-        # Create table from Parquet
-        session.execute(f"""
-           CREATE TABLE if not exists users AS
-           SELECT * FROM read_parquet('{Path(__file__).parent.parent / "queries/users.parquet"}')
-       """)
-
-        # Analytical query
-        session.execute("""
-           SELECT date_trunc('day', created_at) as day,
-                  count(*) as user_count
-           FROM users
-           GROUP BY day
-           ORDER BY day
-       """)
+    # Use a temporary directory for the DuckDB database for test isolation
+    with tempfile.TemporaryDirectory() as tmpdir:
+        db_path = Path(tmpdir) / "analytics.duckdb"
+
+        spec = SQLSpec()
+        # In-memory
+        in_memory_db = spec.add_config(DuckDBConfig())
+        persistent_db = spec.add_config(DuckDBConfig(pool_config={"database": str(db_path)}))
+
+        try:
+            # Test with in-memory config
+            with spec.provide_session(in_memory_db) as session:
+                # Create table from Parquet
+                session.execute(f"""
+                   CREATE TABLE if not exists users AS
+                   SELECT * FROM read_parquet('{Path(__file__).parent.parent / "queries/users.parquet"}')
+               """)
+
+                # Analytical query
+                session.execute("""
+                   SELECT date_trunc('day', created_at) as day,
+                          count(*) as user_count
+                   FROM users
+                   GROUP BY day
+                   ORDER BY day
+               """)
+
+            # Test with persistent config
+            with spec.provide_session(persistent_db) as session:
+                # Create table from Parquet
+                session.execute(f"""
+                   CREATE TABLE if not exists users AS
+                   SELECT * FROM read_parquet('{Path(__file__).parent.parent / "queries/users.parquet"}')
+               """)
+
+                # Analytical query
+                session.execute("""
+                   SELECT date_trunc('day', created_at) as day,
+                          count(*) as user_count
+                   FROM users
+                   GROUP BY day
+                   ORDER BY day
+               """)
+        finally:
+            # Close the pool for the persistent config
+            spec.get_config(in_memory_db).close_pool()
+            spec.get_config(persistent_db).close_pool()
+            # The TemporaryDirectory context manager handles directory cleanup automatically
     # end-example

docs/examples/usage/usage_drivers_and_querying_6.py

@@ -1,6 +1,7 @@
 # Test module converted from docs example - code-block 6
 """Minimal smoke test for drivers_and_querying example 6."""
 
+import tempfile
 from pathlib import Path
 
 from sqlspec import SQLSpec
@@ -12,24 +13,34 @@ def test_example_6_sqlite_config(tmp_path: Path) -> None:
     # start-example
     from sqlspec.adapters.sqlite import SqliteConfig
 
-    spec = SQLSpec()
-
-    database_file = tmp_path / "myapp.db"
-    config = SqliteConfig(pool_config={"database": database_file.name, "timeout": 5.0, "check_same_thread": False})
-
-    with spec.provide_session(config) as session:
-        # Create table
-        session.execute("""
-           CREATE TABLE IF NOT EXISTS usage6_users (
-               id INTEGER PRIMARY KEY,
-               name TEXT NOT NULL
-           )
-       """)
-
-        # Insert with parameters
-        session.execute("INSERT INTO usage6_users (name) VALUES (?)", "Alice")
-
-        # Query
-        result = session.execute("SELECT * FROM usage6_users")
-        result.all()
-    # end-example
+    # Use a temporary file for the SQLite database for test isolation
+    with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as tmp_db_file:
+        db_path = tmp_db_file.name
+
+        spec = SQLSpec()
+
+        db = spec.add_config(
+            SqliteConfig(pool_config={"database": db_path, "timeout": 5.0, "check_same_thread": False})
+        )
+
+        try:
+            with spec.provide_session(db) as session:
+                # Create table
+                session.execute("""
+                CREATE TABLE IF NOT EXISTS usage6_users (
+                    id INTEGER PRIMARY KEY,
+                    name TEXT NOT NULL
+                )
+            """)
+
+                # Insert with parameters
+                session.execute("INSERT INTO usage6_users (name) VALUES (?)", "Alice")
+
+                # Query
+                result = session.execute("SELECT * FROM usage6_users")
+                result.all()
+        finally:
+            # Clean up the temporary database file
+            spec.get_config(db).close_pool()
+            Path(db_path).unlink()
+        # end-example

docs/extensions/aiosql/api.rst

@@ -24,6 +24,7 @@ AiosqlAsyncAdapter
    :members:
    :undoc-members:
    :show-inheritance:
+   :no-index:
 
 AiosqlSyncAdapter
 -----------------
@@ -32,6 +33,7 @@ AiosqlSyncAdapter
    :members:
    :undoc-members:
    :show-inheritance:
+   :no-index:
 
 Query Operators
 ===============

docs/extensions/litestar/api.rst

@@ -11,6 +11,7 @@ SQLSpecPlugin
    :members:
    :undoc-members:
    :show-inheritance:
+   :no-index:
 
 Configuration
 =============

docs/guides/architecture/data-dictionary.md

@@ -0,0 +1,83 @@
+# Data Dictionary & Introspection
+
+SQLSpec provides a unified Data Dictionary API to introspect database schemas across all supported adapters. This allows you to retrieve table metadata, columns, indexes, and foreign keys in a consistent format, regardless of the underlying database engine.
+
+## Core Concepts
+
+The `DataDictionary` is accessed via the `driver.data_dictionary` property. It provides methods to query the database catalog.
+
+### Introspection Capabilities
+
+- **Tables**: List tables in a schema.
+- **Columns**: Get column details (name, type, nullable, default).
+- **Indexes**: Get index definitions (columns, uniqueness).
+- **Foreign Keys**: Get foreign key constraints and relationships.
+- **Topological Sorting**: Get tables sorted by dependency order (useful for cleanups or migrations).
+
+## Usage
+
+### Basic Introspection
+
+```python
+async with config.provide_session() as session:
+    # Get all tables in the default schema
+    tables = await session.data_dictionary.get_tables(session)
+    print(f"Tables: {tables}")
+
+    # Get columns for a specific table
+    columns = await session.data_dictionary.get_columns(session, "users")
+    for col in columns:
+        print(f"{col['column_name']}: {col['data_type']}")
+```
+
+### Topological Sort (Dependency Ordering)
+
+`get_tables` now returns table names sorted such that parent tables appear before child tables (tables with foreign keys to parents).
+
+This is essential for:
+
+- **Data Loading**: Insert into parents first.
+- **Cleanup**: Delete in reverse order to avoid foreign key violations.
+
+```python
+async with config.provide_session() as session:
+    # Get tables sorted parent -> child
+    sorted_tables = await session.data_dictionary.get_tables(session)
+
+    print("Insertion Order:", sorted_tables)
+    print("Deletion Order:", list(reversed(sorted_tables)))
+```
+
+**Implementation Details**:
+
+- **Postgres / SQLite / MySQL 8+**: Uses efficient Recursive CTEs in SQL.
+- **Oracle**: Uses `CONNECT BY` queries.
+- **Others (BigQuery, MySQL 5.7)**: Falls back to a Python-based topological sort using `graphlib`.
+
+### Metadata Types
+
+SQLSpec uses regular classes with __slots__ for metadata results to ensure mypyc compatibility and memory efficiency.
+
+```python
+from sqlspec.driver import ForeignKeyMetadata
+
+async with config.provide_session() as session:
+    fks: list[ForeignKeyMetadata] = await session.data_dictionary.get_foreign_keys(session, "orders")
+
+    for fk in fks:
+        print(f"FK: {fk.column_name} -> {fk.referenced_table}.{fk.referenced_column}")
+```
+
+## Adapter Support Matrix
+
+| Feature | Postgres | SQLite | Oracle | MySQL | DuckDB | BigQuery |
+|---------|----------|--------|--------|-------|--------|----------|
+| Tables | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| Columns | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| Indexes | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ |
+| Foreign Keys | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| Topological Sort | ✅ (CTE) | ✅ (CTE) | ✅ (Connect By) | ✅ (CTE/Python) | ✅ (CTE) | ✅ (Python) |
+
+## API Reference
+
+For a complete API reference of the Data Dictionary components, including `DataDictionaryMixin`, `AsyncDataDictionaryBase`, `SyncDataDictionaryBase`, and the metadata classes (`ForeignKeyMetadata`, `ColumnMetadata`, `IndexMetadata`), please refer to the :doc:`/reference/driver`.

docs/reference/driver.rst

@@ -103,6 +103,43 @@ Connection Pooling
    :undoc-members:
    :show-inheritance:
 
+Data Dictionary
+===============
+
+The Data Dictionary API provides standardized introspection capabilities across all supported databases.
+
+.. currentmodule:: sqlspec.driver
+
+.. autoclass:: DataDictionaryMixin
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: AsyncDataDictionaryBase
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: SyncDataDictionaryBase
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: ForeignKeyMetadata
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: ColumnMetadata
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+.. autoclass:: IndexMetadata
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 Driver Protocols
 ================
 

docs/usage/drivers_and_querying.rst

@@ -471,7 +471,7 @@ Performance Tips
    :start-after: # start-example
    :end-before: # end-example
    :caption: ``asyncpg connection pooling``
-   :dedent: 4
+   :dedent: 2
 
 **2. Batch Operations**