setup

Author: pyramation

packages/transform/AST_RESEARCH.md

@@ -0,0 +1,145 @@
+# PostgreSQL AST Research
+
+This document summarizes differences in the AST type definitions shipped with the `libpg-query-node` project. Each PostgreSQL version directory under `types/` contains TypeScript definitions generated from PostgreSQL's protobuf specification.
+
+## Versioned type files
+
+| Version | Lines in `types.ts` | Lines in `enums.ts` |
+|---------|--------------------|---------------------|
+| 13 | 2098 | 61 |
+| 14 | 2159 | 61 |
+| 15 | 2219 | 63 |
+| 16 | 2325 | 68 |
+| 17 | 2484 | 75 |
+
+Line counts increase steadily and provide a proxy for growing AST complexity.
+
+## Diff sizes between versions
+
+Measured using `diff -u` on the generated `types.ts` and `enums.ts` files.
+
+| Versions compared | Diff lines `types.ts` | Diff lines `enums.ts` |
+|-------------------|----------------------|----------------------|
+| 13 → 14 | 287 | 55 |
+| 14 → 15 | 304 | 47 |
+| 15 → 16 | 2634 | 43 |
+| 16 → 17 | 401 | 58 |
+| 13 → 17 | 2911 | 91 |
+
+Versions 13–15 show relatively small changes. A dramatic increase occurs between PG15 and PG16 with over 2600 changed lines, reflecting large parser changes. PG17 differs from PG16 by about 400 lines.
+
+## Observed differences
+
+A brief inspection of the diffs highlights:
+
+- New enum values across versions (e.g. additional `WCOKind` options in 15, `JsonEncoding` variants in 16).
+- New node interfaces such as `ReturnStmt`, `PLAssignStmt`, and JSON-related constructs appearing in later versions.
+- Some existing structures rename or move fields (for example `relkind` → `objtype` in `AlterTableStmt`).
+### Enum differences by version
+
+| Version | New enum types |
+|---------|----------------|
+| 14 | SetQuantifier |
+| 15 | AlterPublicationAction, PublicationObjSpecType |
+| 16 | JsonConstructorType, JsonEncoding, JsonFormatType, JsonValueType, PartitionStrategy |
+| 17 | JsonBehaviorType, JsonExprOp, JsonQuotes, JsonTableColumnType, JsonWrapper, MergeMatchKind, TableFuncType |
+
+### Enum value shifts
+
+The numeric assignments within several enums changed between releases. The table
+below lists notable examples.
+
+| Enum | Changed in | Notes |
+|------|------------|-------|
+| `A_Expr_Kind` | PG14 | Removed `AEXPR_OF` and `AEXPR_PAREN`, causing indices to shift |
+| `RoleSpecType` | PG14 | Added `ROLESPEC_CURRENT_ROLE` at position 1 |
+| `TableLikeOption` | PG14 | Added `CREATE_TABLE_LIKE_COMPRESSION` at position 1 |
+| `WCOKind` | PG15 | Added `WCO_RLS_MERGE_UPDATE_CHECK` and `WCO_RLS_MERGE_DELETE_CHECK` |
+| `ObjectType` | PG15 | Inserted `OBJECT_PUBLICATION_NAMESPACE` and `OBJECT_PUBLICATION_REL` before existing entries |
+| `JoinType` | PG16 | Added `JOIN_RIGHT_ANTI`, shifting subsequent values |
+| `AlterTableType` | PG16–17 | Many values renumbered; PG17 introduces `AT_SetExpression` |
+| `Token` | multiple | Token list grows each release, with new codes inserted |
+
+Counting all enums, roughly **11** changed between PG13 and PG14, **8** changed from PG14 to PG15, **8** changed from PG15 to PG16, and **10** changed from PG16 to PG17.
+
+
+### Scalar node changes
+
+The basic scalar nodes were refactored in PG15. Prior to that release the `String` and `BitString` nodes carried a generic `str` field, while `Float` relied on `str` as well. From PG15 onward these nodes were split into
+
+- `String` with field `sval`
+- `BitString` with field `bsval`
+- `Float` with field `fval`
+- A new `Boolean` node with field `boolval`
+
+| Version | String field | BitString field | Float field | Boolean field |
+|---------|--------------|-----------------|-------------|---------------|
+| 13–14 | `str` | `str` | `str` | n/a |
+| 15+ | `sval` | `bsval` | `fval` | `boolval` |
+
+These nodes keep the same role but use more explicit property names. Translating from PG13/14 to PG17 therefore requires renaming these fields when constructing the newer AST representation.
+
+These changes indicate incremental evolution in the ASTs, with PG16 introducing the most significant updates.
+### Renamed fields
+
+| From | To | Node type | Introduced in |
+|------|----|-----------|--------------|
+| `relkind` | `objtype` | AlterTableStmt / CreateTableAsStmt | PG14 |
+| `tables` | `pubobjects` | CreatePublicationStmt / AlterPublicationStmt | PG15 |
+| `tableAction` | `action` | AlterPublicationStmt | PG15 |
+| `varnosyn` & `varattnosyn` | `varnullingrels` | Var | PG16 |
+| `aggtranstype` | `aggtransno` | Aggref | PG16 |
+
+### Enum representation changes
+
+Historically libpg_query exposed enum fields in the JSON output as **numeric**
+codes. Starting with the PG15 bindings this switched to returning the **string**
+name of each enum value. The TypeScript type definitions reflect string literal
+unions across all versions, but the underlying JSON changed in PG15.
+
+| Version | Enum format |
+|---------|-------------|
+| 13–14   | integers    |
+| 15–17   | strings     |
+
+
+## Version similarity
+
+Based on diff sizes, PG13 and PG14 are close, as are PG14 and PG15. PG16 introduces major differences, likely due to language features such as the SQL/JSON enhancements. PG17 again adjusts the AST but retains most PG16 structures. Thus PG13–15 form one similar group and PG16–17 another.
+
+## Viability of translation (PG13 → PG17)
+
+Going forward only, translating PG13 ASTs to PG17 is plausible. Many node types remain compatible, and differences are largely additive. A translation layer would need to
+
+1. Map renamed fields (e.g. `relkind` to `objtype`).
+2. Populate newly introduced fields with defaults or derived values.
+3. Handle removed or deprecated fields when present in PG13.
+
+Because PG16 introduced large changes, direct translation from PG13 to PG17 may require bridging PG16 first. Still, each version’s ASTs are defined in TypeScript, so programmatic transforms are feasible.
+### New interface nodes
+
+| Version | Interfaces added |
+|---------|-----------------|
+| 14 | CTECycleClause, CTESearchClause, PLAssignStmt, ReturnStmt, StatsElem |
+| 15 | AlterDatabaseRefreshCollStmt, Boolean, MergeAction, MergeStmt, MergeWhenClause, PublicationObjSpec, PublicationTable |
+| 16 | JsonAggConstructor, JsonArrayAgg, JsonArrayConstructor, JsonArrayQueryConstructor, JsonConstructorExpr, JsonFormat, JsonIsPredicate, JsonKeyValue, JsonObjectAgg, JsonObjectConstructor, JsonOutput, JsonReturning, JsonValueExpr, RTEPermissionInfo |
+| 17 | JsonArgument, JsonBehavior, JsonExpr, JsonFuncExpr, JsonParseExpr, JsonScalarExpr, JsonSerializeExpr, JsonTable, JsonTableColumn, JsonTablePath, JsonTablePathScan, JsonTablePathSpec, JsonTableSiblingJoin, MergeSupportFunc, SinglePartitionSpec, WindowFuncRunCondition |
+
+## Generating AST Samples
+
+To fully understand structural differences we will compile **libpg-query** for
+each supported PostgreSQL version and capture JSON output for a library of
+representative queries. This multi-runtime parser setup lets us record actual
+ASTs from PG13 through PG17. These samples are essential for training upgrade
+logic and verifying enum representations:
+
+- PG13 and PG14 output enum values as integers
+- PG15+ output enums as their string names
+
+The generated samples will live under a dedicated directory and can be compared
+programmatically to spot changes beyond what the protobuf types reveal.
+
+
+## Conclusion
+
+The repository already provides versioned definitions which can be compared programmatically. Diff metrics suggest PG13–15 are most similar, while PG16 marks a major jump and PG17 follows that design. Building an automated translation will require detailed mapping but appears viable, particularly when only upgrading ASTs.

packages/transform/AST_TESTS.md

@@ -0,0 +1,60 @@
+# AST Translation Test Plan
+
+This document outlines a set of incremental tests for building and validating a PostgreSQL AST translation layer. The goal is to ensure ASTs parsed from older versions can be upgraded to the latest version without changing semantic meaning.
+
+## 0. Generate Example ASTs from simple queries
+
+make a new directory called transform/ inside of the root __fixtures__ dir. 
+
+in here, we should have a folder for each version 13/,14/,15/,16/,17/
+
+then, create a series of example .json files that are the resulting output of a handful of sql queries. 
+
+This will be the basis for how we can do our testing, and understand, truly understand the differences in the ASTs.
+
+
+## 1. Baseline Parsing
+
+1. **Parse basic queries for each version**
+   - Verify `parseSync` and `parse` return a single statement for simple queries (`SELECT 1`, `SELECT NULL`).
+2. **Round-trip parsing**
+   - Parse a query, deparse it back to SQL, and parse again. The ASTs should match after removing location data.
+
+## 2. Enum Handling
+
+1. **Integer to string conversion (PG13/14)**
+   - Feed known enum codes to the translation layer and assert the upgraded AST uses the correct enum names.
+2. **Preserve string enums (PG15+)**
+   - Ensure enums already represented as strings remain unchanged after translation.
+
+## 3. Scalar Node Changes
+
+1. **Field rename checks**
+   - Confirm that `String.str` becomes `String.sval`, `BitString.str` becomes `BitString.bsval`, and `Float.str` becomes `Float.fval` when translating from PG13/14.
+2. **Boolean node introduction**
+   - Translating `A_Const` nodes containing boolean values should yield the new `Boolean` node starting in PG15.
+
+## 4. Renamed Fields
+
+Create fixtures demonstrating renamed fields such as `relkind` → `objtype` and `tables` → `pubobjects`. Tests should confirm the new field names and that values are correctly copied.
+
+## 5. Sequential Upgrade Steps
+
+For each release boundary (13→14, 14→15, 15→16, 16→17):
+1. Apply the specific upgrade function to a representative AST.
+2. Validate that required fields are present and obsolete ones removed.
+3. Verify that running all steps in sequence produces the same result as any direct upgrade path (once implemented).
+
+## 6. Full Query Upgrade
+
+1. Parse a library of real-world queries using the oldest supported version.
+2. Upgrade the resulting ASTs to the latest version.
+3. Deparse the upgraded ASTs and execute them against a running PostgreSQL instance to ensure semantics are preserved.
+
+## 7. Future Regression Tests
+
+As translation functions evolve, capture edge cases that previously failed and assert they remain fixed.
+
+---
+
+These tests build confidence incrementally: start with simple node transformations, then cover whole query upgrades. The plan emphasizes functional, deterministic checks that can run in CI.

packages/transform/AST_TRANSLATION.md

@@ -0,0 +1,74 @@
+# AST Translation Strategies
+
+This document explores approaches for translating PostgreSQL ASTs between the versioned type definitions under `types/`.
+
+## Goals
+
+- Upgrade an AST produced for an older PostgreSQL release so that it conforms to the latest definitions
+- Avoid a downgrade path; only translation forward is needed
+- Keep the process transparent and manageable as new versions appear
+
+## Design options
+
+### Functional transforms
+
+One model is to create a set of pure functions, each responsible for upgrading a single node type. These functions would:
+
+1. Accept an instance of a node from the older version
+2. Produce the equivalent structure in the newer version
+3. Rename fields or populate new defaults as required
+
+Benefits:
+
+- Fine grained and testable; each function does one thing well
+- Easier to reason about complex nodes such as `String` or `Var`
+- Composable: an overall upgrade is just a pipeline of node-level transforms
+
+### Nested deparser / reparser
+
+Another idea is to build a new deparser that understands multiple versions simultaneously. The deparser would parse using the old types and re-emit using the newest ones. This could be structured as a visitor that walks the AST, writing out SQL and immediately reparsing with the updated parser.
+
+Benefits:
+
+- Eliminates manual field mapping by relying on the parser to create valid nodes
+- Might handle edge cases automatically where semantics changed
+
+Trade-offs:
+
+- Performance hit due to serializing and parsing again
+- Potential loss of fidelity if certain node properties are not round-trippable
+
+### Handling enums
+
+Older versions of `libpg_query` (PG13 and PG14) emitted numeric codes for enum
+fields. From PG15 onward the JSON output uses the enum **name** as a string.
+Translation code must therefore convert numeric enums to their string
+equivalents when upgrading from PG13/14. When moving between PG15–17 the
+representations already match.
+
+## Translation step ordering
+
+### Sequential upgrades
+
+A straightforward approach is to perform sequential upgrades: 13 → 14 → 15 → 16 → 17. Each step focuses on the incremental changes in that release. This keeps functions small and reuses existing transforms when supporting new versions.
+
+### Direct upgrades
+
+Alternatively, we could implement direct translations for each older version to the newest (13 → 17, 14 → 17, 15 → 17). This avoids running multiple steps but requires larger, more complex functions because they must handle every change introduced across several releases at once.
+
+### Which is better?
+
+Sequential upgrades favor simplicity and reuse. The majority of changes between 13 and 15 are minor, while 16 introduces significant restructuring (see `AST_RESEARCH.md`). Incremental steps allow us to focus on these differences in isolation. Direct upgrades may be feasible for the relatively small jumps (15 → 17), but are harder to implement for 13 → 17.
+
+## Recommended plan
+
+1. Implement functional node-level transforms for each release boundary starting with 13 → 14.
+2. Compose those functions so that upgrading from any supported version to 17 is just a series of transformations.
+3. Provide utilities for renaming fields (e.g. `str` → `sval` in `String` nodes) and filling defaults for new enum values or optional fields.
+4. Optionally develop a proof-of-concept reparse approach for comparison, but keep the functional pipeline as the core strategy.
+
+By building translation functions per version we keep the code maintainable and make it easy to add support for future releases.
+
+## Should we implement a deparser?
+
+The nested deparser approach would walk the old AST, generate SQL, and parse it with the newer parser. This mirrors the visitor pattern and can adapt automatically to certain changes, but it adds overhead and may drop information that doesn't round-trip cleanly. Maintaining explicit transform functions keeps upgrades deterministic and easy to test. A deparser prototype might help with tricky cases, yet the primary strategy should be these per-version transform functions.

packages/transform/PQSQL_PARSER.md

@@ -0,0 +1,96 @@
+this is from the source of the `@pgsql/parser` repo so you can see the API.
+
+
+```ts
+const { describe, it, before } = require('node:test');
+const assert = require('node:assert/strict');
+const { Parser } = require('../wasm/index.cjs');
+
+describe('Parser', () => {
+  describe('Dynamic API', () => {
+    it('should parse SQL with default version', async () => {
+      const parser = new Parser();
+      const result = await parser.parse('SELECT 1+1 as sum');
+      assert.ok(result);
+      assert.ok(result.stmts);
+      assert.equal(result.stmts.length, 1);
+    });
+
+    it('should parse SQL with specific version', async () => {
+      // Get available versions from the Parser class
+      const parser = new Parser();
+      const defaultVersion = parser.version;
+      
+      // Test with a different version if available
+      const testVersion = defaultVersion === 17 ? 16 : 15;
+      try {
+        const versionParser = new Parser(testVersion);
+        const result = await versionParser.parse('SELECT 1+1 as sum');
+        assert.equal(versionParser.version, testVersion);
+        assert.ok(result);
+      } catch (e) {
+        // Version might not be available in this build
+        console.log(`Version ${testVersion} not available in this build`);
+      }
+    });
+
+    it('should handle parse errors', async () => {
+      const parser = new Parser();
+      try {
+        await parser.parse('INVALID SQL');
+        assert.fail('Should have thrown an error');
+      } catch (error) {
+        assert.ok(error);
+        assert.ok(error.message.includes('syntax error'));
+      }
+    });
+
+    it('should work with Parser class', async () => {
+      const parser = new Parser();
+      const result = await parser.parse('SELECT * FROM users');
+      assert.ok(result);
+      assert.ok(result.stmts);
+    });
+
+    it('should validate version in constructor', () => {
+      // Test invalid version
+      assert.throws(() => {
+        new Parser(99);
+      }, /Unsupported PostgreSQL version/);
+    });
+
+    it('should support parseSync after initial parse', async () => {
+      const parser = new Parser();
+      
+      // First parse to initialize
+      await parser.parse('SELECT 1');
+      
+      // Now parseSync should work
+      const result = parser.parseSync('SELECT 2+2 as sum');
+      assert.ok(result);
+      assert.ok(result.stmts);
+      assert.equal(result.stmts.length, 1);
+    });
+  });
+
+  describe('Version-specific imports', () => {
+    // Dynamically test available version imports
+    const versions = [13, 14, 15, 16, 17];
+    
+    for (const version of versions) {
+      it(`should parse with v${version} if available`, async () => {
+        try {
+          const versionModule = require(`../wasm/v${version}.cjs`);
+          await versionModule.loadModule();
+          const result = await versionModule.parse('SELECT 1');
+          assert.ok(result);
+          assert.equal(result.stmts.length, 1);
+        } catch (e) {
+          // Version not available in this build
+          console.log(`Version ${version} not available in this build`);
+        }
+      });
+    }
+  });
+});
+```

packages/transform/package.json

@@ -31,6 +31,7 @@
     "test:watch": "jest --watch"
   },
   "devDependencies": {
+    "@pgsql/parser": "1.0.2",
     "pg-proto-parser": "^1.29.1"
   },
   "keywords": []

packages/transform/scripts/pg-proto-parser.ts

@@ -1,7 +1,7 @@
 import { PgProtoParser, PgProtoParserOptions } from 'pg-proto-parser';
 import { resolve, join } from 'path';
 
-const versions = ['13', '17'];
+const versions = ['13', '14', '15', '16', '17'];
 const baseDir = resolve(join(__dirname, '../../../__fixtures__/proto'));
 
 for (const version of versions) {