Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 72 additions & 7 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ make docs # Build Sphinx documentation
### Adapters Structure

Each adapter in `sqlspec/adapters/` follows this structure:

- `config.py` - Configuration classes with pool settings
- `driver.py` - Query execution implementation (sync/async)
- `_types.py` - Adapter-specific type definitions
Expand All @@ -88,15 +89,14 @@ Supported adapters: adbc, aiosqlite, asyncmy, asyncpg, bigquery, duckdb, oracled
### Database Connection Flow

```python
# 1. Create configuration
config = AsyncpgConfig(pool_config={"dsn": "postgresql://..."})
# 1. Create SQLSpec manager
manager = SQLSpec()

# 2. Register with SQLSpec
sql = SQLSpec()
sql.add_config(config)
# 2. Create and register configuration (returns same instance)
config = manager.add_config(AsyncpgConfig(pool_config={"dsn": "postgresql://..."}))

# 3. Get session via context manager
async with sql.provide_session(config) as session:
# 3. Get session via context manager (using config instance)
async with manager.provide_session(config) as session:
result = await session.execute("SELECT * FROM users")
```

Expand Down Expand Up @@ -133,6 +133,7 @@ async with sql.provide_session(config) as session:
### Mypyc Compatibility

For classes in `sqlspec/core/` and `sqlspec/driver/`:

- Use `__slots__` for data-holding classes
- Implement explicit `__init__`, `__repr__`, `__eq__`, `__hash__`
- Avoid dataclasses in performance-critical paths
Expand Down Expand Up @@ -257,18 +258,72 @@ def database_to_bytes(value: Any) -> bytes | None:
```

**Use this pattern when**:

- Database client library requires specific encoding for binary data
- Need transparent conversion between Python bytes and database format
- Want to centralize encoding/decoding logic for reuse

**Key principles**:

- Centralize conversion functions in `_type_handlers.py`
- Handle None/NULL values explicitly
- Support both raw and encoded formats on read (graceful handling)
- Use in `coerce_params_for_database()` and type converter

**Example**: Spanner requires base64-encoded bytes for `param_types.BYTES` parameters, but you work with raw Python bytes in application code.

### Instance-Based Config Registry Pattern

SQLSpec uses config instances as handles for database connections. The registry keys by `id(config)` instead of `type(config)` to support multiple databases of the same adapter type.

```python
from sqlspec import SQLSpec
from sqlspec.adapters.asyncpg import AsyncpgConfig

manager = SQLSpec()

# Config instance IS the handle - add_config returns same instance
main_db = manager.add_config(AsyncpgConfig(pool_config={"dsn": "postgresql://main/..."}))
analytics_db = manager.add_config(AsyncpgConfig(pool_config={"dsn": "postgresql://analytics/..."}))

# Type checker knows: AsyncpgConfig → AsyncContextManager[AsyncpgDriver]
async with manager.provide_session(main_db) as driver:
await driver.execute("SELECT 1")

# Different connection pool! Works correctly now.
async with manager.provide_session(analytics_db) as driver:
await driver.execute("SELECT 1")
```

**Use this pattern when**:

- Managing multiple databases of the same adapter type (e.g., main + analytics PostgreSQL)
- Integrating with DI frameworks (config instance is the dependency)
- Need type-safe session handles without `# type: ignore`

**Key principles**:

- Registry uses `id(config)` as key (not `type(config)`)
- Multiple configs of same adapter type are stored separately
- `add_config` returns the same instance passed in
- All methods require registered config instances
- Unregistered configs raise `ValueError`

**DI framework integration**:

```python
# Just pass the config instance - it's already correctly typed
def get_main_db() -> AsyncpgConfig:
return main_db

# DI provider knows this is async from the config type
async def provide_db_session(db: AsyncpgConfig, manager: SQLSpec):
async with manager.provide_session(db) as driver:
yield driver
```

**Reference implementation**: `sqlspec/base.py` (lines 58, 128-151, 435-581)

### Framework Extension Pattern

All extensions use `extension_config` in database config:
Expand Down Expand Up @@ -323,17 +378,20 @@ _register_with_sqlglot()
```

**Use this pattern when**:

- Database syntax varies significantly across dialects
- Standard SQLGlot expressions don't match any database's native syntax
- Need operator syntax (e.g., `<->`) vs function calls (e.g., `DISTANCE()`)

**Key principles**:

- Override `.sql()` method for dialect detection
- Register with SQLGlot's TRANSFORMS for nested expression support
- Store metadata (like metric) as `exp.Identifier` in `arg_types` for runtime access
- Provide generic fallback for unsupported dialects

**Example**: `VectorDistance` in `sqlspec/builder/_vector_expressions.py` generates:

- PostgreSQL: `embedding <-> '[0.1,0.2]'` (operator)
- MySQL: `DISTANCE(embedding, STRING_TO_VECTOR('[0.1,0.2]'), 'EUCLIDEAN')` (function)
- Oracle: `VECTOR_DISTANCE(embedding, TO_VECTOR('[0.1,0.2]'), EUCLIDEAN)` (function)
Expand Down Expand Up @@ -388,17 +446,20 @@ Dialect.classes["custom"] = CustomDialect
```

**Create custom dialect when**:

- Database has unique DDL/DML syntax not in existing dialects
- Need to parse and validate database-specific keywords
- Need to generate database-specific SQL from AST
- An existing dialect provides 80%+ compatibility to inherit from

**Do NOT create custom dialect if**:

- Only parameter style differences (use parameter profiles)
- Only type conversion differences (use type converters)
- Only connection management differences (use config/driver)

**Key principles**:

- **Inherit from closest dialect**: Spanner inherits BigQuery (both GoogleSQL)
- **Minimal overrides**: Only override methods that need customization
- **Store metadata in AST**: Use `expression.set(key, value)` for custom data
Expand Down Expand Up @@ -443,17 +504,20 @@ def command(config: str | None):
```

**Benefits:**

- Click automatically handles precedence (CLI flag always overrides env var)
- Help text automatically shows env var name
- Support for multiple fallback env vars via `envvar=["VAR1", "VAR2"]`
- Less code, fewer bugs

**For project file discovery (pyproject.toml, etc.):**

- Use custom logic as fallback after Click env var handling
- Walk filesystem from cwd to find config files
- Return `None` if not found to trigger helpful error message

**Multi-config support:**

- Split comma-separated values from CLI flag, env var, or pyproject.toml
- Resolve each config path independently
- Flatten results if callables return lists
Expand Down Expand Up @@ -531,6 +595,7 @@ if async_configs:
```

**Key principles:**

- Sync configs must execute outside event loop (direct function calls)
- Async configs must execute inside event loop (via `run_()`)
- Multi-config operations should batch sync and async separately for efficiency
Expand Down
7 changes: 3 additions & 4 deletions docs/examples/adapters/adbc_postgres_ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -126,13 +126,12 @@ def main(
destination.parent.mkdir(parents=True, exist_ok=True)

db_manager = SQLSpec()
adbc_config = AdbcConfig(connection_config={"uri": uri})
adbc_key = db_manager.add_config(adbc_config)
adbc_config = db_manager.add_config(AdbcConfig(connection_config={"uri": uri}))

_render_capabilities(console, db_manager.get_config(adbc_key))
_render_capabilities(console, adbc_config)
telemetry_records: list[dict[str, Any]] = []

with db_manager.provide_session(adbc_key) as session:
with db_manager.provide_session(adbc_config) as session:
export_job = session.select_to_storage(
source_sql, str(destination), format_hint=file_format, partitioner=partitioner
)
Expand Down
5 changes: 2 additions & 3 deletions docs/examples/frameworks/fastapi/aiosqlite_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,13 @@

from docs.examples.shared.configs import aiosqlite_registry
from docs.examples.shared.data import ARTICLES, CREATE_ARTICLES
from sqlspec.adapters.aiosqlite import AiosqliteConfig, AiosqliteDriver
from sqlspec.adapters.aiosqlite import AiosqliteDriver
from sqlspec.core import SQL

__all__ = ("get_session", "list_articles", "main", "on_startup", "seed_database")


registry = aiosqlite_registry()
config = registry.get_config(AiosqliteConfig)
registry, config = aiosqlite_registry()
app = FastAPI()


Expand Down
5 changes: 2 additions & 3 deletions docs/examples/frameworks/fastapi/sqlite_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,13 @@

from docs.examples.shared.configs import sqlite_registry
from docs.examples.shared.data import ARTICLES, CREATE_ARTICLES
from sqlspec.adapters.sqlite import SqliteConfig, SqliteDriver
from sqlspec.adapters.sqlite import SqliteDriver
from sqlspec.core import SQL

__all__ = ("get_session", "list_articles", "main", "on_startup", "seed_database")


registry = sqlite_registry()
config = registry.get_config(SqliteConfig)
registry, config = sqlite_registry()
app = FastAPI()


Expand Down
4 changes: 1 addition & 3 deletions docs/examples/frameworks/flask/sqlite_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,12 @@

from docs.examples.shared.configs import sqlite_registry
from docs.examples.shared.data import ARTICLES, CREATE_ARTICLES
from sqlspec.adapters.sqlite import SqliteConfig
from sqlspec.core import SQL

__all__ = ("list_articles", "main", "seed_database")


registry = sqlite_registry()
config = registry.get_config(SqliteConfig)
registry, config = sqlite_registry()
app = Flask(__name__)


Expand Down
5 changes: 2 additions & 3 deletions docs/examples/frameworks/litestar/aiosqlite_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,11 @@

from docs.examples.shared.configs import aiosqlite_registry
from docs.examples.shared.data import ARTICLES, CREATE_ARTICLES
from sqlspec.adapters.aiosqlite import AiosqliteConfig, AiosqliteDriver
from sqlspec.adapters.aiosqlite import AiosqliteDriver
from sqlspec.core import SQL
from sqlspec.extensions.litestar import SQLSpecPlugin

registry = aiosqlite_registry()
config = registry.get_config(AiosqliteConfig)
registry, config = aiosqlite_registry()
plugin = SQLSpecPlugin(sqlspec=registry)


Expand Down
5 changes: 2 additions & 3 deletions docs/examples/frameworks/litestar/duckdb_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,11 @@

from docs.examples.shared.configs import duckdb_registry
from docs.examples.shared.data import ARTICLES, CREATE_ARTICLES
from sqlspec.adapters.duckdb import DuckDBConfig, DuckDBDriver
from sqlspec.adapters.duckdb import DuckDBDriver
from sqlspec.core import SQL
from sqlspec.extensions.litestar import SQLSpecPlugin

registry = duckdb_registry()
config = registry.get_config(DuckDBConfig)
registry, config = duckdb_registry()
plugin = SQLSpecPlugin(sqlspec=registry)


Expand Down
5 changes: 2 additions & 3 deletions docs/examples/frameworks/litestar/sqlite_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,11 @@

from docs.examples.shared.configs import sqlite_registry
from docs.examples.shared.data import ARTICLES, CREATE_ARTICLES
from sqlspec.adapters.sqlite import SqliteConfig, SqliteDriver
from sqlspec.adapters.sqlite import SqliteDriver
from sqlspec.core import SQL
from sqlspec.extensions.litestar import SQLSpecPlugin

registry = sqlite_registry()
config = registry.get_config(SqliteConfig)
registry, config = sqlite_registry()
plugin = SQLSpecPlugin(sqlspec=registry)


Expand Down
4 changes: 1 addition & 3 deletions docs/examples/frameworks/starlette/aiosqlite_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,12 @@

from docs.examples.shared.configs import aiosqlite_registry
from docs.examples.shared.data import ARTICLES, CREATE_ARTICLES
from sqlspec.adapters.aiosqlite import AiosqliteConfig
from sqlspec.core import SQL

__all__ = ("list_articles", "main", "seed_database")


registry = aiosqlite_registry()
config = registry.get_config(AiosqliteConfig)
registry, config = aiosqlite_registry()


async def seed_database() -> None:
Expand Down
18 changes: 9 additions & 9 deletions docs/examples/shared/configs.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,34 +8,34 @@
__all__ = ("aiosqlite_registry", "duckdb_registry", "sqlite_registry")


def sqlite_registry(bind_key: str = "docs_sqlite") -> "SQLSpec":
def sqlite_registry(bind_key: str = "docs_sqlite") -> "tuple[SQLSpec, SqliteConfig]":
"""Return a registry with a single SQLite configuration."""
registry = SQLSpec()
registry.add_config(SqliteConfig(bind_key=bind_key, pool_config={"database": ":memory:"}))
return registry
config = registry.add_config(SqliteConfig(bind_key=bind_key, pool_config={"database": ":memory:"}))
return registry, config


def aiosqlite_registry(bind_key: str = "docs_aiosqlite") -> "SQLSpec":
def aiosqlite_registry(bind_key: str = "docs_aiosqlite") -> "tuple[SQLSpec, AiosqliteConfig]":
"""Return a registry backed by an AioSQLite pool."""
registry = SQLSpec()
registry.add_config(
config = registry.add_config(
AiosqliteConfig(
bind_key=bind_key,
pool_config={"database": ":memory:"},
extension_config={"litestar": {"commit_mode": "autocommit"}},
)
)
return registry
return registry, config


def duckdb_registry(bind_key: str = "docs_duckdb") -> "SQLSpec":
def duckdb_registry(bind_key: str = "docs_duckdb") -> "tuple[SQLSpec, DuckDBConfig]":
"""Return a registry with a DuckDB in-memory database."""
registry = SQLSpec()
registry.add_config(
config = registry.add_config(
DuckDBConfig(
bind_key=bind_key,
pool_config={"database": ":memory:shared_docs"},
extension_config={"litestar": {"commit_mode": "autocommit"}},
)
)
return registry
return registry, config
13 changes: 5 additions & 8 deletions docs/examples/usage/usage_configuration_19.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,18 +20,15 @@ def test_binding_multiple_configs() -> None:
with tempfile.NamedTemporaryFile(suffix=".db", delete=True) as tmp:
db_manager = SQLSpec()

# Add multiple configurations
sqlite_key = db_manager.add_config(SqliteConfig(pool_config={"database": tmp.name}))
# Add multiple configurations - add_config returns the config instance
sqlite_config = db_manager.add_config(SqliteConfig(pool_config={"database": tmp.name}))
dsn = os.getenv("SQLSPEC_USAGE_PG_DSN", "postgresql://localhost/db")
asyncpg_key = db_manager.add_config(AsyncpgConfig(pool_config={"dsn": dsn}))
pg_config = db_manager.add_config(AsyncpgConfig(pool_config={"dsn": dsn}))

# Use specific configuration
with db_manager.provide_session(sqlite_key) as session:
# Use specific configuration - pass the config instance directly
with db_manager.provide_session(sqlite_config) as session:
session.execute("SELECT 1")

sqlite_config = db_manager.get_config(sqlite_key)
pg_config = db_manager.get_config(asyncpg_key)

# end-example
assert sqlite_config.pool_config["database"] == tmp.name
assert pg_config.pool_config["dsn"] == os.getenv("SQLSPEC_USAGE_PG_DSN", "postgresql://localhost/db")
Loading