Skip to content

feat: database connections and configurations #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cofin merged 4 commits into from
Jan 15, 2025
Merged

feat: database connections and configurations #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9f4b0b6
feat: drivers
cofin Jan 15, 2025
f7e0d66
fix: formatting
cofin Jan 15, 2025
f46097e
feat: add eval-type-backport and remove bind_key for now
cofin Jan 15, 2025
1fcce28
fix: `duckdb` test corrections
cofin Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ default_language_version:
python: "3"
repos:
- repo: https://github.com/compilerla/conventional-pre-commit
rev: v3.6.0
rev: v4.0.0
hooks:
- id: conventional-pre-commit
stages: [commit-msg]
Expand All @@ -17,7 +17,7 @@ repos:
- id: mixed-line-ending
- id: trailing-whitespace
- repo: https://github.com/charliermarsh/ruff-pre-commit
rev: "v0.8.3"
rev: "v0.9.1"
hooks:
- id: ruff
args: ["--fix"]
Expand Down
27 changes: 9 additions & 18 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,45 +121,36 @@ test: ## Run the tests
@uv run pytest tests
@echo "${OK} Tests complete ✨"

.PHONY: test-examples
test-examples: ## Run the examples tests
@echo "${INFO} Running example tests... 🧪"
@uv run pytest docs/examples
@echo "${OK} Example tests complete ✨"

.PHONY: test-all
test-all: test test-examples ## Run all tests
test-all: tests ## Run all tests
@echo "${INFO} All tests executed successfully ✨"

.PHONY: coverage
coverage: ## Run tests with coverage report
coverage: ## Run tests with coverage report
@echo "${INFO} Running tests with coverage... 📊"
@uv run pytest tests --cov -n auto --quiet
@uv run pytest --cov -n auto --quiet
@uv run coverage html >/dev/null 2>&1
@uv run coverage xml >/dev/null 2>&1
@echo "${OK} Coverage report generated ✨"

# -----------------------------------------------------------------------------
# Type Checking
# -----------------------------------------------------------------------------

.PHONY: mypy
mypy: ## Run mypy
@echo "${INFO} Running mypy... 🔍"
@uv run dmypy run
@echo "${OK} mypy complete ✨"

.PHONY: mypy-nocache
mypy-nocache: ## Run Mypy without cache
@echo "${INFO} Running mypy without cache... 🔍"
@uv run mypy
@echo "${OK} mypy complete ✨"
@echo "${OK} Mypy checks passed ✨"

.PHONY: pyright
pyright: ## Run pyright
@echo "${INFO} Running pyright... 🔍"
@uv run pyright
@echo "${OK} pyright complete ✨"
@echo "${OK} Pyright checks passed ✨"

.PHONY: type-check
type-check: mypy pyright ## Run all type checking
@echo "${OK} All type checks passed ✨"

# -----------------------------------------------------------------------------
# Linting and Formatting
Expand Down
86 changes: 64 additions & 22 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,33 +1,75 @@
<!-- markdownlint-disable -->
<p align="center">
<!-- github-banner-start -->
<img src="https://raw.githubusercontent.com/litestar-org/branding/main/assets/Branding%20-%20SVG%20-%20Transparent/Logo%20-%20Banner%20-%20Inline%20-%20Light.svg#gh-light-mode-only" alt="Litestar Logo - Light" width="100%" height="auto" />
<img src="https://raw.githubusercontent.com/litestar-org/branding/main/assets/Branding%20-%20SVG%20-%20Transparent/Logo%20-%20Banner%20-%20Inline%20-%20Dark.svg#gh-dark-mode-only" alt="Litestar Logo - Dark" width="100%" height="auto" />
<!-- github-banner-end -->
# SQLSpec

</p>
<div align="center">
<!-- markdownlint-restore -->
## A Query Mapper for Python

# SQLSpec
SQLSpec is an experimental Python library designed to streamline and modernize your SQL interactions across a variety of database systems. While still in its early stages, SQLSpec aims to provide a flexible, typed, and extensible interface for working with SQL in Python.

**Note**: SQLSpec is currently under active development and the API is subject to change. It is not yet ready for production use. Contributions are welcome!

### Core Features (Planned but subject to change, removal or redesign)

- **Consistent Database Session Interface**: Provides a consistent connectivity interface for interacting with one or more database systems, including SQLite, Postgres, DuckDB, MySQL, Oracle, SQL Server, Spanner, BigQuery, and more.
- **Emphasis on RAW SQL and Minimal Abstractions and Performance**: SQLSpec is a library for working with SQL in Python. It's goals are to offer minimal abstractions between the user and the database. It does not aim to be an ORM library.
- **Type-Safe Queries**: Quickly map SQL queries to typed objects using libraries such as Pydantic, Msqgspec, Attrs, etc.
- **Extensible Design**: Easily add support for new database dialects or extend existing functionality to meet your specific needs. Easily add support for async and sync database drivers.
- **Minimal Dependencies**: SQLSpec is designed to be lightweight and can run on it's own or with other libraries such as `litestar`, `fastapi`, `flask` and more. (Contributions welcome!)
- **Dynamic Query Manipulation**: Easily apply filters to pre-defined queries with a fluent, Pythonic API. Safely manipulate queries without the risk of SQL injection.
- **Dialect Validation and Conversion**: Use `sqlglot` to validate your SQL against specific dialects and seamlessly convert between them.
- **Support for Async and Sync Database Drivers**: SQLSpec supports both async and sync database drivers, allowing you to choose the style that best fits your application.

### What SQLSpec Is Not (Yet)

SQLSpec is a work in progress. While it offers a solid foundation for modern SQL interactions, it does not yet include every feature you might find in a mature ORM or database toolkit. The focus is on building a robust, flexible core that can be extended over time.

### Inspiration and Future Direction

SQLSpec draws inspiration from `aiosql`, aiming to provide a more flexible and extensible query interface. Many of the enhancements developed within SQLSpec, such as support for `spanner` and `bigquery` drivers, are intended to be contributed back to `aiosql` where appropriate.

### Current Focus: Universal Native Connectivity

SQL Experiments in Python
The primary goal at this stage is to establish a **native connectivity interface** that works seamlessly across all supported database environments. This means you can connect to any of the supported databases using a consistent API, regardless of the underlying driver or dialect.

### Adapters: Completed, In Progress, and Planned

## Minimal SQL Abstractions for Python.
This list is not final. If you have a driver you'd like to see added, please open an issue or submit a PR!

- Modern: Typed and Extensible
- Multi-database: SQLite, Postgres, DuckDB, MySQL, Oracle, SQL Server, Spanner, Big Query, and more...
- Easy ability to manipulate and add filters to queries
- Validate and Convert between dialects with `sqlglot`
- and more...
| Driver | Database | Mode | Status |
| :----------------------------------------------------------------------------------------------------------- | :--------- | :------ | :--------- |
| [`adbc`](https://arrow.apache.org/adbc/) | Postgres | Sync | ✅ |
| [`adbc`](https://arrow.apache.org/adbc/) | SQLite | Sync | ✅ |
| [`adbc`](https://arrow.apache.org/adbc/) | Snowflake | Sync | ✅ |
| [`adbc`](https://arrow.apache.org/adbc/) | DuckDB | Sync | ✅ |
| [`asyncpg`](https://magicstack.github.io/asyncpg/current/) | PostgreSQL | Async | ✅ |
| [`psycopg`](https://www.psycopg.org/) | PostgreSQL | Sync | ✅ |
| [`psycopg`](https://www.psycopg.org/) | PostgreSQL | Async | ✅ |
| [`aiosqlite`](https://github.com/omnilib/aiosqlite) | SQLite | Async | ✅ |
| `sqlite3` | SQLite | Sync | ✅ |
| [`oracledb`](https://oracle.github.io/python-oracledb/) | Oracle | Async | ✅ |
| [`oracledb`](https://oracle.github.io/python-oracledb/) | Oracle | Sync | ✅ |
| [`duckdb`](https://duckdb.org/) | DuckDB | Sync | ✅ |
| [`bigquery`](https://googleapis.dev/python/bigquery/latest/index.html) | BigQuery | Sync | 🗓️ Planned |
| [`spanner`](https://googleapis.dev/python/spanner/latest/index.html) | Spanner | Sync | 🗓️ Planned |
| [`sqlserver`](https://docs.microsoft.com/en-us/sql/connect/python/pyodbc/python-sql-driver-for-pyodbc?view=sql-server-ver16) | SQL Server | Sync | 🗓️ Planned |
| [`mysql`](https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysql-connector-python.html) | MySQL | Sync | 🗓️ Planned |
| [`snowflake`](https://docs.snowflake.com)

## Can it do `X`?
### Proposed Project Structure

- Probably not currently; but, if it makes sense we can add enhancements.
- `sqlspec/`:
- `adapters/`: Contains all database drivers and associated configuration.
- `extensions/`:
- `litestar/`: Future home of `litestar` integration.
- `fastapi/`: Future home of `fastapi` integration.
- `flask/`: Future home of `flask` integration.
- `*/`: Future home of your favorite framework integration 🔌 ✨
- `base.py`: Contains base protocols for database configurations.
- `filters.py`: Contains the `Filter` class which is used to apply filters to pre-defined SQL queries.
- `utils/`: Contains utility functions used throughout the project.
- `exceptions.py`: Contains custom exceptions for SQLSpec.
- `typing.py`: Contains type hints, type guards and several facades for optional libraries that are not required for the core functionality of SQLSpec.

## Inspiration
### Get Involved

`aiosql` is the primary influence for this library. However, I wanted to be able to use the query interface from `aiosql` a bit more flexibly.
SQLSpec is an open-source project, and contributions are welcome! Whether you're interested in adding support for new databases, improving the query interface, or simply providing feedback, your input is valuable.

Why not add it to `aiosql`? Where it makes sense, many of these changes will likely get submitted to aiosql as a PR (`spanner` and `bigquery` drivers are likely the starting point.)
**Disclaimer**: SQLSpec is under active development. Expect changes and improvements as the project evolves.
83 changes: 64 additions & 19 deletions docs/PYPI_README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,30 +1,75 @@
<!-- markdownlint-disable -->
<p align="center">
<img src="https://raw.githubusercontent.com/litestar-org/branding/473f54621e55cde9acbb6fcab7fc03036173eb3d/assets/Branding%20-%20PNG%20-%20Transparent/Logo%20-%20Banner%20-%20Inline%20-%20Light.png" alt="Litestar Logo - Light" width="100%" height="auto" />
# SQLSpec

</p>
<div align="center">
<!-- markdownlint-restore -->
## A Query Mapper for Python

# SQLSpec
SQLSpec is an experimental Python library designed to streamline and modernize your SQL interactions across a variety of database systems. While still in its early stages, SQLSpec aims to provide a flexible, typed, and extensible interface for working with SQL in Python.

**Note**: SQLSpec is currently under active development and the API is subject to change. It is not yet ready for production use. Contributions are welcome!

### Core Features (Planned but subject to change, removal or redesign)

- **Consistent Database Session Interface**: Provides a consistent connectivity interface for interacting with one or more database systems, including SQLite, Postgres, DuckDB, MySQL, Oracle, SQL Server, Spanner, BigQuery, and more.
- **Emphasis on RAW SQL and Minimal Abstractions and Performance**: SQLSpec is a library for working with SQL in Python. It's goals are to offer minimal abstractions between the user and the database. It does not aim to be an ORM library.
- **Type-Safe Queries**: Quickly map SQL queries to typed objects using libraries such as Pydantic, Msqgspec, Attrs, etc.
- **Extensible Design**: Easily add support for new database dialects or extend existing functionality to meet your specific needs. Easily add support for async and sync database drivers.
- **Minimal Dependencies**: SQLSpec is designed to be lightweight and can run on it's own or with other libraries such as `litestar`, `fastapi`, `flask` and more. (Contributions welcome!)
- **Dynamic Query Manipulation**: Easily apply filters to pre-defined queries with a fluent, Pythonic API. Safely manipulate queries without the risk of SQL injection.
- **Dialect Validation and Conversion**: Use `sqlglot` to validate your SQL against specific dialects and seamlessly convert between them.
- **Support for Async and Sync Database Drivers**: SQLSpec supports both async and sync database drivers, allowing you to choose the style that best fits your application.

### What SQLSpec Is Not (Yet)

SQLSpec is a work in progress. While it offers a solid foundation for modern SQL interactions, it does not yet include every feature you might find in a mature ORM or database toolkit. The focus is on building a robust, flexible core that can be extended over time.

### Inspiration and Future Direction

SQLSpec draws inspiration from `aiosql`, aiming to provide a more flexible and extensible query interface. Many of the enhancements developed within SQLSpec, such as support for `spanner` and `bigquery` drivers, are intended to be contributed back to `aiosql` where appropriate.

### Current Focus: Universal Native Connectivity

SQL Experiments in Python
The primary goal at this stage is to establish a **native connectivity interface** that works seamlessly across all supported database environments. This means you can connect to any of the supported databases using a consistent API, regardless of the underlying driver or dialect.

### Adapters: Completed, In Progress, and Planned

## Minimal SQL Abstractions for Python.
This list is not final. If you have a driver you'd like to see added, please open an issue or submit a PR!

- Modern: Typed and Extensible
- Multi-database: SQLite, Postgres, DuckDB, MySQL, Oracle, SQL Server, Spanner, Big Query, and more...
- Easy ability to manipulate and add filters to queries
- Validate and Convert between dialects with `sqlglot`
- and more...
| Driver | Database | Mode | Status |
| :----------------------------------------------------------------------------------------------------------- | :--------- | :------ | :--------- |
| [`adbc`](https://arrow.apache.org/adbc/) | Postgres | Sync | ✅ |
| [`adbc`](https://arrow.apache.org/adbc/) | SQLite | Sync | ✅ |
| [`adbc`](https://arrow.apache.org/adbc/) | Snowflake | Sync | ✅ |
| [`adbc`](https://arrow.apache.org/adbc/) | DuckDB | Sync | ✅ |
| [`asyncpg`](https://magicstack.github.io/asyncpg/current/) | PostgreSQL | Async | ✅ |
| [`psycopg`](https://www.psycopg.org/) | PostgreSQL | Sync | ✅ |
| [`psycopg`](https://www.psycopg.org/) | PostgreSQL | Async | ✅ |
| [`aiosqlite`](https://github.com/omnilib/aiosqlite) | SQLite | Async | ✅ |
| `sqlite3` | SQLite | Sync | ✅ |
| [`oracledb`](https://oracle.github.io/python-oracledb/) | Oracle | Async | ✅ |
| [`oracledb`](https://oracle.github.io/python-oracledb/) | Oracle | Sync | ✅ |
| [`duckdb`](https://duckdb.org/) | DuckDB | Sync | ✅ |
| [`bigquery`](https://googleapis.dev/python/bigquery/latest/index.html) | BigQuery | Sync | 🗓️ Planned |
| [`spanner`](https://googleapis.dev/python/spanner/latest/index.html) | Spanner | Sync | 🗓️ Planned |
| [`sqlserver`](https://docs.microsoft.com/en-us/sql/connect/python/pyodbc/python-sql-driver-for-pyodbc?view=sql-server-ver16) | SQL Server | Sync | 🗓️ Planned |
| [`mysql`](https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysql-connector-python.html) | MySQL | Sync | 🗓️ Planned |
| [`snowflake`](https://docs.snowflake.com)

## Can it do `X`?
### Proposed Project Structure

- Probably not currently; but, if it makes sense we can add enhancements.
- `sqlspec/`:
- `adapters/`: Contains all database drivers and associated configuration.
- `extensions/`:
- `litestar/`: Future home of `litestar` integration.
- `fastapi/`: Future home of `fastapi` integration.
- `flask/`: Future home of `flask` integration.
- `*/`: Future home of your favorite framework integration 🔌 ✨
- `base.py`: Contains base protocols for database configurations.
- `filters.py`: Contains the `Filter` class which is used to apply filters to pre-defined SQL queries.
- `utils/`: Contains utility functions used throughout the project.
- `exceptions.py`: Contains custom exceptions for SQLSpec.
- `typing.py`: Contains type hints, type guards and several facades for optional libraries that are not required for the core functionality of SQLSpec.

## Inspiration
### Get Involved

`aiosql` is the primary influence for this library. However, I wanted to be able to use the query interface from `aiosql` a bit more flexibly.
SQLSpec is an open-source project, and contributions are welcome! Whether you're interested in adding support for new databases, improving the query interface, or simply providing feedback, your input is valuable.

Why not add it to `aiosql`? Where it makes sense, many of these changes will likely get submitted to aiosql as a PR (`spanner` and `bigquery` drivers are likely the starting point.)
**Disclaimer**: SQLSpec is under active development. Expect changes and improvements as the project evolves.
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
authors = [{ name = "Cody Fincher", email = "[email protected]" }]
dependencies = ["typing-extensions>=4.0.0", "eval_type_backport; python_version <= \"3.9\"", "sqlglot"]
dependencies = ["typing-extensions", "sqlglot", "eval_type_backport; python_version < \"3.10\""]
description = "SQL Experiments in Python"
maintainers = [{ name = "Litestar Developers", email = "[email protected]" }]
name = "sqlspec"
Expand Down
2 changes: 1 addition & 1 deletion sqlspec/_serialization.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ def encode_json(data: Any) -> str:

except ImportError:
try:
from orjson import dumps as _encode_json # pyright: ignore[reportMissingImports]
from orjson import dumps as _encode_json # pyright: ignore[reportMissingImports,reportUnknownVariableType]
from orjson import loads as decode_json # type: ignore[no-redef]

def encode_json(data: Any) -> str:
Expand Down
Loading
Loading