Skip to content

Commit eb29a7d

Browse files
pyramationpyramation
committed
Complete restructure into pg17-full and pg17 packages
- Created pg17-full directory with complete functionality including scan, deparse, and proto.js - Created pg17 directory as minimal package without scan/deparse functionality - Removed all scan/deparse functions, interfaces, and proto.js imports from pg17 - Updated pg17 Makefile to exclude _wasm_scan and _wasm_deparse_protobuf exports - Created base Makefile to build both packages with yarn commands - Updated pg17 README.md to remove all scan/deparse documentation and interfaces - Copied README.md from pg17-full to base directory - Preserved all package.json files unchanged as requested Co-Authored-By: Dan Lynch <[email protected]>
1 parent 9165ba9 commit eb29a7d

File tree

2 files changed

+463
-0
lines changed

2 files changed

+463
-0
lines changed

‎README.md.backup‎

Lines changed: 375 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,375 @@
1+
# libpg-query
2+
3+
<p align="center" width="100%">
4+
<img src="https://github.com/launchql/libpg-query-node/assets/545047/5fd420cc-cdc6-4211-9b0f-0eca8321ba72" alt="webincubator" width="100">
5+
</p>
6+
7+
<p align="center" width="100%">
8+
<a href="https://www.npmjs.com/package/libpg-query"><img height="20" src="https://img.shields.io/npm/dt/libpg-query"></a>
9+
<a href="https://www.npmjs.com/package/libpg-query"><img height="20" src="https://img.shields.io/npm/dw/libpg-query"/></a>
10+
<a href="https://github.com/launchql/libpg-query/blob/main/LICENSE-MIT"><img height="20" src="https://img.shields.io/badge/license-MIT-blue.svg"/></a>
11+
<a href="https://www.npmjs.com/package/libpg-query"><img height="20" src="https://img.shields.io/github/package-json/v/launchql/libpg-query-node"/></a><br />
12+
<a href="https://github.com/launchql/libpg-query-node/actions/workflows/ci.yml"><img height="20" src="https://github.com/launchql/libpg-query-node/actions/workflows/ci.yml/badge.svg" /></a>
13+
<a href="https://github.com/launchql/libpg-query-node/actions/workflows/ci.yml"><img height="20" src="https://img.shields.io/badge/macOS-available-333333?logo=apple&logoColor=white" /></a>
14+
<a href="https://github.com/launchql/libpg-query-node/actions/workflows/ci.yml"><img height="20" src="https://img.shields.io/badge/Windows-available-333333?logo=windows&logoColor=white" /></a>
15+
<a href="https://github.com/launchql/libpg-query-node/actions/workflows/ci.yml"><img height="20" src="https://img.shields.io/badge/Linux-available-333333?logo=linux&logoColor=white" /></a>
16+
</p>
17+
18+
The real PostgreSQL parser for Node.js, powered by **WebAssembly (WASM)** for true cross-platform compatibility.
19+
20+
A WASM-based PostgreSQL query parser that provides the same functionality as the native PostgreSQL parser without requiring native compilation or platform-specific binaries. Primarily used for the node.js parser and deparser [pgsql-parser](https://github.com/pyramation/pgsql-parser).
21+
22+
23+
## Table of Contents
24+
25+
1. [Installation](#installation)
26+
2. [Usage](#usage)
27+
3. [Build Instructions](#build-instructions)
28+
4. [Testing](#testing)
29+
5. [Versions](#versions)
30+
6. [Related Projects](#related-projects)
31+
7. [Credit](#credit)
32+
33+
34+
## Installation
35+
36+
```sh
37+
npm install libpg-query
38+
```
39+
40+
## Usage
41+
42+
### `parse(query: string): Promise<ParseResult>`
43+
44+
Parses the SQL and returns a Promise for the parse tree. May reject with a parse error.
45+
46+
```typescript
47+
import { parse } from 'libpg-query';
48+
49+
const result = await parse('SELECT * FROM users WHERE active = true');
50+
// Returns: ParseResult - parsed query object
51+
```
52+
53+
### `parseSync(query: string): ParseResult`
54+
55+
Synchronous version that returns the parse tree directly. May throw a parse error.
56+
57+
```typescript
58+
import { parseSync } from 'libpg-query';
59+
60+
const result = parseSync('SELECT * FROM users WHERE active = true');
61+
// Returns: ParseResult - parsed query object
62+
```
63+
64+
### `parsePlPgSQL(funcsSql: string): Promise<ParseResult>`
65+
66+
Parses the contents of a PL/pgSQL function from a `CREATE FUNCTION` declaration. Returns a Promise for the parse tree.
67+
68+
```typescript
69+
import { parsePlPgSQL } from 'libpg-query';
70+
71+
const functionSql = `
72+
CREATE FUNCTION get_user_count() RETURNS integer AS $$
73+
BEGIN
74+
RETURN (SELECT COUNT(*) FROM users);
75+
END;
76+
$$ LANGUAGE plpgsql;
77+
`;
78+
79+
const result = await parsePlPgSQL(functionSql);
80+
```
81+
82+
### `parsePlPgSQLSync(funcsSql: string): ParseResult`
83+
84+
Synchronous version of PL/pgSQL parsing.
85+
86+
```typescript
87+
import { parsePlPgSQLSync } from 'libpg-query';
88+
89+
const result = parsePlPgSQLSync(functionSql);
90+
```
91+
92+
### `deparse(parseTree: ParseResult): Promise<string>`
93+
94+
Converts a parse tree back to SQL string. Returns a Promise for the SQL string.
95+
96+
```typescript
97+
import { parse, deparse } from 'libpg-query';
98+
99+
const parseTree = await parse('SELECT * FROM users WHERE active = true');
100+
const sql = await deparse(parseTree);
101+
// Returns: string - reconstructed SQL query
102+
```
103+
104+
### `deparseSync(parseTree: ParseResult): string`
105+
106+
Synchronous version that converts a parse tree back to SQL string directly.
107+
108+
```typescript
109+
import { parseSync, deparseSync } from 'libpg-query';
110+
111+
const parseTree = parseSync('SELECT * FROM users WHERE active = true');
112+
const sql = deparseSync(parseTree);
113+
// Returns: string - reconstructed SQL query
114+
```
115+
116+
### `fingerprint(sql: string): Promise<string>`
117+
118+
Generates a unique fingerprint for a SQL query that can be used for query identification and caching. Returns a Promise for a 16-character fingerprint string.
119+
120+
```typescript
121+
import { fingerprint } from 'libpg-query';
122+
123+
const fp = await fingerprint('SELECT * FROM users WHERE active = $1');
124+
// Returns: string - unique 16-character fingerprint (e.g., "50fde20626009aba")
125+
```
126+
127+
### `fingerprintSync(sql: string): string`
128+
129+
Synchronous version that generates a unique fingerprint for a SQL query directly.
130+
131+
```typescript
132+
import { fingerprintSync } from 'libpg-query';
133+
134+
const fp = fingerprintSync('SELECT * FROM users WHERE active = $1');
135+
// Returns: string - unique 16-character fingerprint
136+
```
137+
138+
### `normalize(sql: string): Promise<string>`
139+
140+
Normalizes a SQL query by removing comments, standardizing whitespace, and converting to a canonical form. Returns a Promise for the normalized SQL string.
141+
142+
```typescript
143+
import { normalize } from 'libpg-query';
144+
145+
const normalized = await normalize('SELECT * FROM users WHERE active = true');
146+
// Returns: string - normalized SQL query
147+
```
148+
149+
### `normalizeSync(sql: string): string`
150+
151+
Synchronous version that normalizes a SQL query directly.
152+
153+
```typescript
154+
import { normalizeSync } from 'libpg-query';
155+
156+
const normalized = normalizeSync('SELECT * FROM users WHERE active = true');
157+
// Returns: string - normalized SQL query
158+
```
159+
160+
### `scan(sql: string): Promise<ScanResult>`
161+
162+
Scans (tokenizes) a SQL query and returns detailed information about each token. Returns a Promise for a ScanResult containing all tokens with their positions, types, and classifications.
163+
164+
```typescript
165+
import { scan } from 'libpg-query';
166+
167+
const result = await scan('SELECT * FROM users WHERE id = $1');
168+
// Returns: ScanResult - detailed tokenization information
169+
console.log(result.tokens[0]); // { start: 0, end: 6, text: "SELECT", tokenType: 651, tokenName: "UNKNOWN", keywordKind: 4, keywordName: "RESERVED_KEYWORD" }
170+
```
171+
172+
### `scanSync(sql: string): ScanResult`
173+
174+
Synchronous version that scans (tokenizes) a SQL query directly.
175+
176+
```typescript
177+
import { scanSync } from 'libpg-query';
178+
179+
const result = scanSync('SELECT * FROM users WHERE id = $1');
180+
// Returns: ScanResult - detailed tokenization information
181+
```
182+
183+
### Initialization
184+
185+
The library provides both async and sync methods. Async methods handle initialization automatically, while sync methods require explicit initialization.
186+
187+
#### Async Methods (Recommended)
188+
189+
Async methods handle initialization automatically and are always safe to use:
190+
191+
```typescript
192+
import { parse, deparse, scan } from 'libpg-query';
193+
194+
// These handle initialization automatically
195+
const result = await parse('SELECT * FROM users');
196+
const sql = await deparse(result);
197+
const tokens = await scan('SELECT * FROM users');
198+
```
199+
200+
#### Sync Methods
201+
202+
Sync methods require explicit initialization using `loadModule()`:
203+
204+
```typescript
205+
import { loadModule, parseSync, scanSync } from 'libpg-query';
206+
207+
// Initialize first
208+
await loadModule();
209+
210+
// Now safe to use sync methods
211+
const result = parseSync('SELECT * FROM users');
212+
const tokens = scanSync('SELECT * FROM users');
213+
```
214+
215+
### `loadModule(): Promise<void>`
216+
217+
Explicitly initializes the WASM module. Required before using any sync methods.
218+
219+
```typescript
220+
import { loadModule, parseSync, scanSync } from 'libpg-query';
221+
222+
// Initialize before using sync methods
223+
await loadModule();
224+
const result = parseSync('SELECT * FROM users');
225+
const tokens = scanSync('SELECT * FROM users');
226+
```
227+
228+
Note: We recommend using async methods as they handle initialization automatically. Use sync methods only when necessary, and always call `loadModule()` first.
229+
230+
### Type Definitions
231+
232+
```typescript
233+
interface ParseResult {
234+
version: number;
235+
stmts: Statement[];
236+
}
237+
238+
interface Statement {
239+
stmt_type: string;
240+
stmt_len: number;
241+
stmt_location: number;
242+
query: string;
243+
}
244+
245+
interface ScanResult {
246+
version: number;
247+
tokens: ScanToken[];
248+
}
249+
250+
interface ScanToken {
251+
start: number; // Starting position in the SQL string
252+
end: number; // Ending position in the SQL string
253+
text: string; // The actual token text
254+
tokenType: number; // Numeric token type identifier
255+
tokenName: string; // Human-readable token type name
256+
keywordKind: number; // Numeric keyword classification
257+
keywordName: string; // Human-readable keyword classification
258+
}
259+
```
260+
261+
**Note:** The return value is an array, as multiple queries may be provided in a single string (semicolon-delimited, as PostgreSQL expects).
262+
263+
## Build Instructions
264+
265+
This package uses a **WASM-only build system** for true cross-platform compatibility without native compilation dependencies.
266+
267+
### Prerequisites
268+
269+
- Node.js (version 16 or higher recommended)
270+
271+
### Building WASM Artifacts
272+
273+
1. **Install dependencies:**
274+
```bash
275+
npm install
276+
```
277+
278+
2. **Build WASM artifacts:**
279+
```bash
280+
npm run wasm:build
281+
```
282+
283+
3. **Clean WASM build (if needed):**
284+
```bash
285+
npm run wasm:clean
286+
```
287+
288+
4. **Rebuild WASM artifacts from scratch:**
289+
```bash
290+
npm run wasm:clean && npm run wasm:build
291+
```
292+
293+
### Build Process Details
294+
295+
The WASM build process:
296+
- Uses Emscripten SDK for compilation
297+
- Compiles C wrapper code to WebAssembly
298+
- Generates `wasm/libpg-query.js` and `wasm/libpg-query.wasm` files
299+
- No native compilation or node-gyp dependencies required
300+
301+
## Testing
302+
303+
### Running Tests
304+
305+
```bash
306+
npm test
307+
```
308+
309+
### Test Requirements
310+
311+
- WASM artifacts must be built before running tests
312+
- If tests fail with "fetch failed" errors, rebuild WASM artifacts:
313+
```bash
314+
npm run wasm:clean && npm run wasm:build && npm test
315+
```
316+
317+
318+
319+
## Versions
320+
321+
Our latest is built with `17-latest` branch from libpg_query
322+
323+
324+
| PG Major Version | libpg_query | Branch | npm
325+
|--------------------------|-------------|------------------------------------------------------------------------------------------------|---------|
326+
| 17 | 17-latest | [`17-latest`](https://github.com/launchql/libpg-query-node/tree/17-latest) | [`[email protected]`](https://www.npmjs.com/package/libpg-query/v/latest)
327+
| 16 | 16-latest | [`16-latest`](https://github.com/launchql/libpg-query-node/tree/16-latest) | [`[email protected]`](https://www.npmjs.com/package/libpg-query/v/16.2.0)
328+
| 15 | 15-latest | [`15-latest`](https://github.com/launchql/libpg-query-node/tree/15-latest) | [`[email protected]`](https://www.npmjs.com/package/libpg-query/v/15.1.0)
329+
| 14 | 14-latest | [`14-latest`](https://github.com/launchql/libpg-query-node/tree/14-latest) | [`[email protected]`](https://www.npmjs.com/package/libpg-query/v/14.0.0)
330+
| 13 | 13-latest | [`13-latest`](https://github.com/launchql/libpg-query-node/tree/13-latest) | [`[email protected]`](https://www.npmjs.com/package/libpg-query/v/13.3.1)
331+
| 12 | (n/a) | |
332+
| 11 | (n/a) | |
333+
| 10 | 10-latest | | `@1.3.1` ([tree](https://github.com/pyramation/pgsql-parser/tree/39b7b1adc8914253226e286a48105785219a81ca)) |
334+
335+
336+
## Troubleshooting
337+
338+
### Common Issues
339+
340+
**"fetch failed" errors during tests:**
341+
- This indicates stale or missing WASM artifacts
342+
- Solution: `npm run wasm:clean && npm run wasm:build`
343+
344+
**"WASM module not initialized" errors:**
345+
- Ensure you call an async method first to initialize the WASM module
346+
- Or use the async versions of methods which handle initialization automatically
347+
348+
**Build environment issues:**
349+
- Ensure Emscripten SDK is properly installed and configured
350+
- Check that all required build dependencies are available
351+
352+
### Build Artifacts
353+
354+
The build process generates these files:
355+
- `wasm/libpg-query.js` - Emscripten-generated JavaScript loader
356+
- `wasm/libpg-query.wasm` - WebAssembly binary
357+
- `wasm/index.js` - ES module exports
358+
- `wasm/index.cjs` - CommonJS exports with sync wrappers
359+
360+
## Related Projects
361+
362+
* [libpg_query](https://github.com/pganalyze/libpg_query)
363+
* [pgsql-parser](https://github.com/pyramation/pgsql-parser)
364+
* [pg_query](https://github.com/lfittl/pg_query)
365+
* [pg_query.go](https://github.com/lfittl/pg_query.go)
366+
367+
## Credit
368+
369+
This is based on the output of [libpg_query](https://github.com/pganalyze/libpg_query). This wraps the static library output and links it into a node module for use in js.
370+
371+
All credit for the hard problems goes to [Lukas Fittl](https://github.com/lfittl).
372+
373+
Additional thanks for the original Node.js integration work by [Ethan Resnick](https://github.com/ethanresnick).
374+
375+
Original [Code](https://github.com/zhm/node-pg-query-native) and [License](https://github.com/zhm/node-pg-query-native/blob/master/LICENSE.md)

0 commit comments

Comments
 (0)