Skip to content

Commit a096d5c

Browse files
pyramationpyramation
committed
notes
1 parent d814538 commit a096d5c

File tree

9 files changed

+407
-199
lines changed

9 files changed

+407
-199
lines changed

‎packages/transform/AST_NOTES.md‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,153 @@
11
# AST Transformer Implementation
22

3+
IMPORANT: update, improve, and leverage `expectOldAstToTransformToNewAst` in packages/transform/test-utils/expect-ast.ts
4+
5+
6+
## Visitor Pattern Architecture
7+
8+
The deparser in `packages/deparser` demonstrates an elegant visitor pattern that can be leveraged for building AST transformers. The core approach follows these principles:
9+
10+
### Key Components
11+
12+
1. **Main Visitor Class**: A central class that implements the visitor interface
13+
2. **Dynamic Method Dispatch**: Uses node type names to automatically route to handler methods
14+
3. **Context Propagation**: Passes context through the visitor tree for state management
15+
4. **Simple Method Signatures**: Each node handler follows a consistent `NodeType(node, context)` pattern
16+
17+
### Core Pattern from Deparser
18+
19+
```typescript
20+
// Main visitor method - routes to specific handlers
21+
visit(node: Node, context: Context): ResultType {
22+
const nodeType = this.getNodeType(node);
23+
const nodeData = this.getNodeData(node);
24+
25+
const methodName = nodeType as keyof this;
26+
if (typeof this[methodName] === 'function') {
27+
return (this[methodName] as any)(nodeData, context);
28+
}
29+
30+
throw new Error(`Handler not found for: ${nodeType}`);
31+
}
32+
33+
// Individual node handlers - clean, focused methods
34+
SelectStmt(node: SelectStmt, context: Context): ResultType {
35+
// Transform logic here
36+
return transformedResult;
37+
}
38+
39+
A_Const(node: A_Const, context: Context): ResultType {
40+
// Transform logic here
41+
return transformedResult;
42+
}
43+
```
44+
45+
### Benefits for AST Transformation
46+
47+
- **Extensible**: Easy to add new node type handlers
48+
- **Maintainable**: Each transformation is isolated to its own method
49+
- **Debuggable**: Clear method names match AST node types exactly — method names match the Type.
50+
- **Composable**: Handlers can call `visit()` recursively for child nodes, OR directly call another vistor function directly if it's not a `Node` type and inlined.
51+
52+
### Implementation Strategy
53+
54+
Apply this same pattern to build transformers for PG version migrations (13→14, 14→15, etc.) where each transformer class handles the specific changes needed for that version transition.
55+
56+
## Runtime Schema and Enum Tools
57+
58+
Check the `src/13`, `src/14`, `src/15`, `src/16`, `src/16` directories for useful runtime schema utilities that may help with AST generation.
59+
60+
The enum tooling is particularly helpful for earlier PostgreSQL versions (PG13 and PG14) that used numeric enums in their AST instead of strings. These directories contain:
61+
62+
- Type definitions
63+
- Enum mappings
64+
- Enum conversion functions for translating between numeric and string representations
65+
66+
### Scripts
67+
68+
The `scripts/` directory contains several analysis tools for understanding AST differences and enum representations across PostgreSQL versions. These tools depend on fixtures generated from actual PostgreSQL ASTs.
69+
70+
#### Core Scripts
71+
72+
**1. `generate-ast-fixtures.js`** - Foundation for all analysis
73+
- **Purpose**: Generates AST fixtures for PostgreSQL versions 13-17
74+
- **Output**: Creates `__fixtures__/transform/{version}/` directories with JSON files
75+
- **Query Coverage**:
76+
- Basic operations: SELECT, INSERT, UPDATE, DELETE
77+
- DDL: CREATE TABLE, ALTER TABLE
78+
- Advanced: Complex queries with CTEs, JOINs, window functions
79+
- **Usage**: `node packages/transform/scripts/generate-ast-fixtures.js`
80+
- **Note**: Must be run first to create the fixture data that other scripts depend on
81+
82+
**2. `analyze-ast-differences.js`** - Version migration analysis
83+
- **Purpose**: Deep comparison of AST structures between PostgreSQL versions
84+
- **Capabilities**:
85+
- Detects field renames (e.g., `relkind` → `objtype`)
86+
- Identifies structure changes (e.g., A_Const flattening in v14→v15)
87+
- Finds enum value changes
88+
- Categorizes differences by type and affected nodes
89+
- **Output**: Console report + `ast-differences-analysis.json`
90+
- **Usage**: `node packages/transform/scripts/analyze-ast-differences.js`
91+
92+
**3. `analyze-enum-representation.js`** - Enum format analysis
93+
- **Purpose**: Analyzes how enums are represented across versions (numeric vs string)
94+
- **Key Insights**:
95+
- PG13/14 use numeric enums in many cases
96+
- PG15+ moved to string-based enums
97+
- Identifies fields that need enum conversion during transformation
98+
- **Usage**: `node packages/transform/scripts/analyze-enum-representation.js`
99+
100+
**4. `pg-proto-parser.ts`** - Type generation utility
101+
- **Purpose**: Generates TypeScript types and utilities from PostgreSQL protobuf definitions
102+
- **Output**: Creates `src/{version}/` directories with:
103+
- Type definitions
104+
- Enum mappings
105+
- Runtime schema information
106+
- Enum conversion utilities (`enum-to-int.ts`, `enum-to-str.ts`)
107+
108+
#### Analysis Workflow
109+
110+
1. **Generate Fixtures**: Run `generate-ast-fixtures.js` to create base data
111+
2. **Analyze Differences**: Use `analyze-ast-differences.js` to understand version changes
112+
3. **Study Enums**: Run `analyze-enum-representation.js` for enum conversion patterns
113+
4. **Generate Types**: Use `pg-proto-parser.ts` for type definitions and utilities
114+
115+
#### Fixture Quality & Coverage
116+
117+
**Current State**: The fixtures provide good coverage for common SQL operations but could benefit from expansion.
118+
119+
**Improvement Areas**:
120+
- More complex DDL operations (CREATE INDEX, constraints, etc.)
121+
- Advanced SQL features (window functions, recursive CTEs)
122+
- PostgreSQL-specific syntax (arrays, JSON operations, etc.)
123+
- Edge cases and error conditions
124+
125+
**Adding New Test Cases**: Modify the `queries` object in `generate-ast-fixtures.js`:
126+
127+
```javascript
128+
const queries = {
129+
// ... existing queries ...
130+
'new_feature': [
131+
'SELECT ARRAY[1,2,3]',
132+
'SELECT jsonb_path_query(data, "$.key")'
133+
]
134+
};
135+
```
136+
137+
#### Script Limitations
138+
139+
- **Robustness**: These are analysis tools, not production utilities
140+
- **Error Handling**: Limited error recovery for malformed ASTs
141+
- **Performance**: Not optimized for large-scale analysis
142+
- **Dependencies**: Require consistent fixture generation to be meaningful
143+
144+
#### Practical Usage Tips
145+
146+
- **Incremental Analysis**: Focus on specific version transitions (e.g., 14→15)
147+
- **Query Expansion**: Add representative queries for your specific use cases
148+
- **Pattern Recognition**: Look for consistent patterns across multiple query types
149+
- **Validation**: Cross-reference findings with PostgreSQL release notes
150+
3151
### Transformers
4152

5153
#### v13 → v14 (`v13-to-v14.ts`)

‎packages/transform/AST_TESTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ then, create a series of example .json files that are the resulting output of a
1212

1313
This will be the basis for how we can do our testing, and understand, truly understand the differences in the ASTs.
1414

15+
IMPORANT: update, improve, and leverage `expectOldAstToTransformToNewAst` in packages/transform/test-utils/expect-ast.ts
1516

1617
## 1. Baseline Parsing
1718

‎packages/transform/__test__/transform.test.ts‎

Lines changed: 18 additions & 199 deletions
Original file line numberDiff line numberDiff line change
@@ -1,206 +1,25 @@
1-
import { transformPG13ToPG17 } from '../src/index';
1+
// PostgreSQL AST Transformer Tests
2+
//
3+
// Individual transformer tests have been split into separate files:
4+
// - v13-to-v14.test.ts - PG13 → PG14 transformer tests
5+
// - v14-to-v15.test.ts - PG14 → PG15 transformer tests
6+
// - v15-to-v16.test.ts - PG15 → PG16 transformer tests
7+
// - v16-to-v17.test.ts - PG16 → PG17 transformer tests
8+
//
9+
// This file can be used for integration tests or cross-version testing
10+
211
import { Node as PG13Node } from '../src/13/types';
12+
import { Node as PG14Node } from '../src/14/types';
13+
import { Node as PG15Node } from '../src/15/types';
14+
import { Node as PG16Node } from '../src/16/types';
315
import { Node as PG17Node } from '../src/17/types';
416

5-
describe('PG13 to PG17 transformer', () => {
6-
test('transforms basic string node', () => {
7-
const pg13Node: PG13Node = {
8-
String: { sval: 'test' }
9-
};
10-
const result = transformPG13ToPG17(pg13Node);
11-
expect(result).toEqual({ String: { sval: 'test' } });
12-
});
13-
14-
test('transforms integer node', () => {
15-
const pg13Node: PG13Node = {
16-
Integer: { ival: 42 }
17-
};
18-
const result = transformPG13ToPG17(pg13Node);
19-
expect(result).toEqual({ Integer: { ival: 42 } });
20-
});
21-
22-
test('transforms boolean node', () => {
23-
const pg13Node: PG13Node = {
24-
Boolean: { boolval: true }
25-
};
26-
const result = transformPG13ToPG17(pg13Node);
27-
expect(result).toEqual({ Boolean: { boolval: true } });
28-
});
29-
30-
test('transforms A_Const with string value', () => {
31-
const pg13Node: PG13Node = {
32-
A_Const: {
33-
sval: { sval: 'hello' },
34-
location: 0
35-
}
36-
};
37-
const result = transformPG13ToPG17(pg13Node);
38-
expect(result).toEqual({
39-
A_Const: {
40-
ival: undefined,
41-
fval: undefined,
42-
boolval: undefined,
43-
sval: { sval: 'hello' },
44-
bsval: undefined,
45-
isnull: undefined,
46-
location: 0
47-
}
48-
});
49-
});
50-
51-
test('transforms TableFunc with new PG17 fields', () => {
52-
const pg13Node: PG13Node = {
53-
TableFunc: {
54-
ns_uris: [],
55-
ns_names: [],
56-
docexpr: undefined,
57-
rowexpr: undefined,
58-
colnames: [],
59-
coltypes: [],
60-
coltypmods: [],
61-
colcollations: [],
62-
colexprs: [],
63-
coldefexprs: [],
64-
notnulls: [],
65-
ordinalitycol: 0,
66-
location: 0
67-
}
68-
};
69-
const result = transformPG13ToPG17(pg13Node) as { TableFunc: any };
70-
71-
expect(result.TableFunc.functype).toBeUndefined();
72-
expect(result.TableFunc.colvalexprs).toBeUndefined();
73-
expect(result.TableFunc.passingvalexprs).toBeUndefined();
74-
expect(result.TableFunc.plan).toBeUndefined();
75-
expect(result.TableFunc.ns_uris).toEqual([]);
76-
expect(result.TableFunc.location).toBe(0);
77-
});
78-
79-
test('transforms List with nested nodes', () => {
80-
const pg13Node: PG13Node = {
81-
List: {
82-
items: [
83-
{ String: { sval: 'item1' } },
84-
{ String: { sval: 'item2' } }
85-
]
86-
}
87-
};
88-
const result = transformPG13ToPG17(pg13Node);
89-
expect(result).toEqual({
90-
List: {
91-
items: [
92-
{ String: { sval: 'item1' } },
93-
{ String: { sval: 'item2' } }
94-
]
95-
}
96-
});
97-
});
98-
99-
test('transforms RangeVar with alias', () => {
100-
const pg13Node: PG13Node = {
101-
RangeVar: {
102-
schemaname: 'public',
103-
relname: 'users',
104-
alias: {
105-
aliasname: 'u',
106-
colnames: []
107-
},
108-
location: 10
109-
}
110-
};
111-
const result = transformPG13ToPG17(pg13Node);
112-
expect(result).toEqual({
113-
RangeVar: {
114-
catalogname: undefined,
115-
schemaname: 'public',
116-
relname: 'users',
117-
inh: undefined,
118-
relpersistence: undefined,
119-
alias: {
120-
aliasname: 'u',
121-
colnames: []
122-
},
123-
location: 10
124-
}
125-
});
126-
});
127-
128-
test('handles PG17-only node types by passing through', () => {
129-
const pg17OnlyNode = {
130-
WindowFuncRunCondition: {
131-
opno: 123,
132-
inputcollid: 456,
133-
wfunc_left: true
134-
}
135-
};
136-
137-
const result = transformPG13ToPG17(pg17OnlyNode as any);
138-
expect(result).toEqual(pg17OnlyNode);
139-
});
140-
141-
test('transforms complex nested structure', () => {
142-
const pg13Node: PG13Node = {
143-
Query: {
144-
commandType: 'CMD_SELECT',
145-
querySource: 'QSRC_ORIGINAL',
146-
canSetTag: true,
147-
utilityStmt: undefined,
148-
resultRelation: 0,
149-
hasAggs: false,
150-
hasWindowFuncs: false,
151-
hasTargetSRFs: false,
152-
hasSubLinks: false,
153-
hasDistinctOn: false,
154-
hasRecursive: false,
155-
hasModifyingCTE: false,
156-
hasForUpdate: false,
157-
hasRowSecurity: false,
158-
isReturn: false,
159-
cteList: [],
160-
rtable: [
161-
{
162-
RangeVar: {
163-
relname: 'test_table',
164-
location: 0
165-
}
166-
}
167-
],
168-
jointree: {
169-
fromlist: [],
170-
quals: undefined
171-
},
172-
targetList: [
173-
{
174-
ResTarget: {
175-
name: 'column1',
176-
val: { String: { sval: 'value1' } },
177-
location: 5
178-
}
179-
}
180-
],
181-
stmt_location: 0,
182-
stmt_len: 25
183-
}
184-
};
185-
186-
const result = transformPG13ToPG17(pg13Node);
187-
expect('Query' in result).toBe(true);
188-
if ('Query' in result) {
189-
expect(result.Query.commandType).toBe('CMD_SELECT');
190-
expect(result.Query.rtable).toHaveLength(1);
191-
expect(result.Query.targetList).toHaveLength(1);
192-
}
17+
describe('AST Transformer Integration', () => {
18+
it('should handle multi-version transformations', () => {
19+
// Test chaining multiple transformers (e.g., PG13 → PG14 → PG15)
19320
});
19421

195-
test('throws error for unknown node type', () => {
196-
const unknownNode = { UnknownType: { someField: 'value' } };
197-
try {
198-
transformPG13ToPG17(unknownNode as any);
199-
throw new Error('Expected function to throw');
200-
} catch (error: any) {
201-
if (!error.message.includes('Unknown node type')) {
202-
throw new Error(`Expected "Unknown node type" error but got: ${error.message}`);
203-
}
204-
}
22+
it('should maintain AST validity across all transformations', () => {
23+
// Test that transformed ASTs are valid for their target version
20524
});
20625
});

0 commit comments

Comments
 (0)