Skip to content

Commit 7aac83b

Browse files
authored
Merge pull request #34 from PostHog/feat/ast-transpiler
Add AST-based SQL transpiler for PostgreSQL to DuckDB conversion
2 parents 40c5310 + f1b2c16 commit 7aac83b

23 files changed

+4468
-725
lines changed

CLAUDE.md

Lines changed: 26 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,14 @@ PostgreSQL Client → TLS → Duckgres Server → DuckDB (per-user database)
1818
- **server/server.go**: Server struct, connection handling, graceful shutdown
1919
- **server/conn.go**: Client connection handling, query execution, COPY protocol
2020
- **server/protocol.go**: PostgreSQL wire protocol message encoding/decoding
21-
- **server/catalog.go**: pg_catalog compatibility (views, functions, query rewriting)
21+
- **server/catalog.go**: pg_catalog compatibility views and macros initialization
2222
- **server/types.go**: Type OID mapping between DuckDB and PostgreSQL
2323
- **server/ratelimit.go**: Rate limiting for brute-force protection
2424
- **server/tls.go**: Auto-generation of self-signed TLS certificates
25+
- **transpiler/**: AST-based SQL transpiler (PostgreSQL → DuckDB)
26+
- `transpiler.go`: Main API, transform pipeline orchestration
27+
- `config.go`: Configuration types (DuckLakeMode, ConvertPlaceholders)
28+
- `transform/`: Individual transform implementations
2529

2630
## PostgreSQL Wire Protocol
2731

@@ -41,22 +45,27 @@ The server implements the PostgreSQL v3 protocol:
4145
### Extended Query Protocol
4246
Supports prepared statements (Parse/Bind/Execute) for parameterized queries and binary result formats.
4347

44-
## pg_catalog Compatibility (server/catalog.go)
48+
## pg_catalog Compatibility
4549

4650
psql and other clients expect PostgreSQL system catalogs. We provide compatibility by:
4751

48-
1. **Creating views** in main schema that mirror pg_catalog tables:
52+
1. **Creating views** in main schema (server/catalog.go `initPgCatalog()`):
4953
- `pg_database`, `pg_class_full`, `pg_collation`, `pg_policy`, `pg_roles`
5054
- `pg_statistic_ext`, `pg_publication`, `pg_publication_rel`, `pg_inherits`, etc.
5155

52-
2. **Creating macros** for PostgreSQL functions:
56+
2. **Creating macros** for PostgreSQL functions (server/catalog.go):
5357
- `pg_get_userbyid`, `pg_table_is_visible`, `format_type`, `pg_get_expr`
5458
- `obj_description`, `col_description`, `pg_get_indexdef`, etc.
5559

56-
3. **Query rewriting** to replace PostgreSQL-specific syntax:
57-
- `pg_catalog.pg_class``pg_class_full`
58-
- `OPERATOR(pg_catalog.~)``~`
59-
- `::pg_catalog.regtype``::VARCHAR`
60+
3. **AST-based SQL transpilation** (transpiler/ package):
61+
The transpiler parses PostgreSQL SQL into an AST using pg_query_go (PostgreSQL's C parser),
62+
applies transforms, and deparses back to DuckDB-compatible SQL. Transforms include:
63+
- **PgCatalogTransform**: `pg_catalog.pg_class``pg_class_full`, strips schema prefix from functions
64+
- **TypeCastTransform**: `::pg_catalog.regtype``::VARCHAR`
65+
- **VersionTransform**: `version()` → PostgreSQL-compatible version string
66+
- **SetShowTransform**: Converts SET/SHOW commands, marks ignored parameters
67+
- **DDLTransform**: (DuckLake mode) Strips PRIMARY KEY, UNIQUE, REFERENCES, SERIAL types
68+
- **PlaceholderTransform**: Counts $1, $2 parameters for prepared statements
6069

6170
## COPY Protocol (server/conn.go)
6271

@@ -95,12 +104,17 @@ PGPASSWORD=postgres psql "host=127.0.0.1 port=35437 user=postgres sslmode=requir
95104

96105
### Adding a new pg_catalog view
97106
1. Add view creation SQL in `initPgCatalog()` in `catalog.go`
98-
2. Add regex pattern to rewrite `pg_catalog.viewname` to `viewname`
99-
3. Add the replacement in `rewritePgCatalogQuery()`
107+
2. If the view needs query rewriting (e.g., `pg_catalog.viewname` `viewname`):
108+
- Add mapping in `transpiler/transform/pgcatalog.go` in `pgCatalogViewMappings`
100109

101110
### Adding a new PostgreSQL function
102111
1. Add `CREATE MACRO` in the `functions` slice in `initPgCatalog()`
103-
2. Add function name to `pgCatalogFunctions` slice for query rewriting
112+
2. The transpiler automatically strips `pg_catalog.` prefix from function calls
113+
114+
### Adding a new transform
115+
1. Create a new file in `transpiler/transform/` implementing the `Transform` interface
116+
2. Register the transform in `transpiler/transpiler.go` `New()` function
117+
3. Add tests in `transpiler/transpiler_test.go`
104118

105119
### Adding protocol support
106120
1. Add message type constant in `protocol.go`
@@ -110,6 +124,7 @@ PGPASSWORD=postgres psql "host=127.0.0.1 port=35437 user=postgres sslmode=requir
110124
## Dependencies
111125

112126
- `github.com/duckdb/duckdb-go/v2` - DuckDB Go driver
127+
- `github.com/pganalyze/pg_query_go/v6` - PostgreSQL SQL parser (CGO, uses libpg_query)
113128
- `gopkg.in/yaml.v3` - YAML config parsing
114129

115130
## Known Limitations

go.mod

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ require (
2424
github.com/klauspost/compress v1.18.0 // indirect
2525
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
2626
github.com/lib/pq v1.10.9 // indirect
27+
github.com/pganalyze/pg_query_go/v6 v6.1.0 // indirect
2728
github.com/pierrec/lz4/v4 v4.1.22 // indirect
2829
github.com/zeebo/xxh3 v1.0.2 // indirect
2930
golang.org/x/exp v0.0.0-20250408133849-7e4ce0ab07d0 // indirect
@@ -32,4 +33,5 @@ require (
3233
golang.org/x/sys v0.35.0 // indirect
3334
golang.org/x/tools v0.36.0 // indirect
3435
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
36+
google.golang.org/protobuf v1.36.8 // indirect
3537
)

go.sum

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,12 @@ github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9L
2828
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
2929
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
3030
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
31+
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
3132
github.com/golang/snappy v1.0.0 h1:Oy607GVXHs7RtbggtPBnr2RmDArIsAefDwvrdWvRhGs=
3233
github.com/golang/snappy v1.0.0/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
3334
github.com/google/flatbuffers v25.2.10+incompatible h1:F3vclr7C3HpB1k9mxCGRMXq6FdUalZ6H/pNX4FP1v0Q=
3435
github.com/google/flatbuffers v25.2.10+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
36+
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
3537
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
3638
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
3739
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
@@ -48,6 +50,8 @@ github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 h1:AMFGa4R4MiIpsp
4850
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8/go.mod h1:mC1jAcsrzbxHt8iiaC+zU4b1ylILSosueou12R++wfY=
4951
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 h1:+n/aFZefKZp7spd8DFdX7uMikMLXX4oubIzJF4kv/wI=
5052
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3/go.mod h1:RagcQ7I8IeTMnF8JTXieKnO4Z6JCsikNEzj0DwauVzE=
53+
github.com/pganalyze/pg_query_go/v6 v6.1.0 h1:jG5ZLhcVgL1FAw4C/0VNQaVmX1SUJx71wBGdtTtBvls=
54+
github.com/pganalyze/pg_query_go/v6 v6.1.0/go.mod h1:nvTHIuoud6e1SfrUaFwHqT0i4b5Nr+1rPWVds3B5+50=
5155
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
5256
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
5357
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
@@ -68,10 +72,15 @@ golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
6872
golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
6973
golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg=
7074
golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s=
75+
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
7176
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da h1:noIWHXmPHxILtqtCOPIhSt0ABwskkZKjD3bXGnZGpNY=
7277
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90=
7378
gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk=
7479
gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E=
80+
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
81+
google.golang.org/protobuf v1.31.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
82+
google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
83+
google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
7584
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
7685
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
7786
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=

0 commit comments

Comments
 (0)