Commit 670eecc
feat: Add Catalog and Table implementations for PostgreSQL (#5487)
## Changes Made
Allows us to do
```
catalog = daft.catalog.Catalog.from_postgres("postgresql://user:pass@uri:5432/postgres")
table = catalog.get_table("myschema.mytable")
df = table.read()
...
output_table = catalog.create_table_if_not_exists("myschema.output")
output_table.write(df)
```
Depends on #5471 for
distributed writes and the `df.write_sql()` interface.
## Key decisions, limitations, and notes
- Embeddings support is currently provided via the pgvector extension
and the pgvector python client library.
- The following types are currently not supported:
- Duration
- Interval
- The following types currently fallback to JSONB:
- Multidimensional lists
- Structs
- Maps
- Images
- Tensors
- No connection pooling
- Writes happen on a single node
(#5471 implements distributed
writes)
- Uses psycopg and `COPY` for writes. ADBC would potentially be a better
choice here but we ignore it for now due to its limited type support.
- Instead of assuming that `public` is the default schema, we delegate
table resolution to Postgres' [Schema Search
Path](https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>1 parent 8d62057 commit 670eecc
File tree
5 files changed
+1413
-5
lines changed- daft/catalog
- tests/integration/sql
- docker-compose
5 files changed
+1413
-5
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
312 | 312 | | |
313 | 313 | | |
314 | 314 | | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
315 | 344 | | |
316 | 345 | | |
317 | 346 | | |
| |||
326 | 355 | | |
327 | 356 | | |
328 | 357 | | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
329 | 374 | | |
330 | 375 | | |
331 | 376 | | |
| |||
796 | 841 | | |
797 | 842 | | |
798 | 843 | | |
| 844 | + | |
| 845 | + | |
799 | 846 | | |
800 | 847 | | |
801 | 848 | | |
| |||
0 commit comments