Skip to content

Commit c001ab3

Browse files
devin-ai-integration[bot]pyramation
devin-ai-integration[bot]
and
pyramation
committed
Add comprehensive BUILD.md and ISSUES.md documentation
- BUILD.md: Complete build process documentation for feat/17-wasm branch - References branchName = '17-6.1.0' as specified - Documents WASM-only architecture and Docker-based build system - Includes troubleshooting and development workflow guidance - ISSUES.md: Comprehensive audit findings and issue documentation - Critical: String shadowing issue from GitHub #92 with detailed analysis - Major: Missing WASM build artifacts preventing library usage - Additional build system and code quality issues identified - Proposed solutions and implementation priorities for each issue Audit conducted on feat/17-wasm branch examining all key components: - Makefile and build system configuration - WASM wrapper implementation and JavaScript interfaces - Protocol buffer generation and conflicts - Test suite coverage and functionality - Project structure and dependencies Co-Authored-By: Dan Lynch <[email protected]>
1 parent 5edc480 commit c001ab3

File tree

2 files changed

+599
-0
lines changed

2 files changed

+599
-0
lines changed

‎BUILD.md‎

Lines changed: 246 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
# Build Instructions for libpg-query-node (feat/17-wasm)
2+
3+
This document provides comprehensive build instructions for the feat/17-wasm branch of libpg-query-node, which implements a **WASM-only build system** for true cross-platform compatibility without native compilation dependencies.
4+
5+
## Overview
6+
7+
The feat/17-wasm branch represents a significant architectural change from the previous native addon approach to a WebAssembly-only implementation. This eliminates the need for platform-specific native compilation while maintaining full PostgreSQL parser functionality.
8+
9+
### Key Changes in feat/17-wasm
10+
- **WASM-only architecture**: No more native Node.js addons or node-gyp dependencies
11+
- **libpg_query version**: Uses tag `17-6.1.0` from pganalyze/libpg_query
12+
- **Cross-platform compatibility**: Single WASM binary works across all platforms
13+
- **Simplified deployment**: No pre-compiled binaries or platform-specific builds needed
14+
15+
## Branch Configuration
16+
17+
```javascript
18+
const branchName = '17-6.1.0';
19+
```
20+
21+
This branch uses libpg_query tag `17-6.1.0` which corresponds to PostgreSQL 17 with version 6.1.0 of the libpg_query library.
22+
23+
## Prerequisites
24+
25+
### Required Software
26+
- **Node.js**: Version 16 or higher recommended
27+
- **Docker**: Required for Emscripten-based WASM compilation
28+
- **Git**: For repository management
29+
30+
### Build Dependencies
31+
The build process uses Docker with the official Emscripten SDK image, so you don't need to install Emscripten locally.
32+
33+
## Build Process
34+
35+
### 1. Install Dependencies
36+
37+
```bash
38+
npm install
39+
```
40+
41+
### 2. Build WASM Artifacts
42+
43+
The primary build command uses Docker with Emscripten:
44+
45+
```bash
46+
npm run wasm:build
47+
```
48+
49+
This command executes:
50+
```bash
51+
docker run --rm -v $(pwd):/src -u $(id -u):$(id -g) emscripten/emsdk emmake make build
52+
```
53+
54+
### 3. Alternative Build Commands
55+
56+
**Clean and rebuild from scratch:**
57+
```bash
58+
npm run build
59+
```
60+
This runs: `yarn wasm:clean && yarn wasm:build`
61+
62+
**Clean WASM artifacts only:**
63+
```bash
64+
npm run wasm:clean
65+
```
66+
67+
**Rebuild WASM artifacts:**
68+
```bash
69+
npm run wasm:rebuild
70+
```
71+
72+
**Clean build cache:**
73+
```bash
74+
npm run wasm:clean-cache
75+
```
76+
77+
## Build Process Details
78+
79+
### Makefile Configuration
80+
81+
The build process is controlled by the Makefile with these key settings:
82+
83+
- **WASM_OUT_DIR**: `wasm` (output directory)
84+
- **WASM_OUT_NAME**: `libpg-query` (output filename base)
85+
- **WASM_MODULE_NAME**: `PgQueryModule` (exported module name)
86+
- **LIBPG_QUERY_TAG**: `17-6.1.0` (libpg_query version)
87+
- **LIBPG_QUERY_REPO**: `https://github.com/pganalyze/libpg_query.git`
88+
89+
### Build Steps
90+
91+
1. **Clone libpg_query**: Downloads libpg_query source code for the specified tag
92+
2. **Build libpg_query**: Compiles the static library using the upstream Makefile
93+
3. **Compile WASM wrapper**: Compiles `src/wasm_wrapper.c` with Emscripten
94+
4. **Generate WASM artifacts**: Creates `libpg-query.js` and `libpg-query.wasm`
95+
96+
### Generated Files
97+
98+
The build process creates these files in the `wasm/` directory:
99+
100+
- `libpg-query.js` - Emscripten-generated JavaScript loader
101+
- `libpg-query.wasm` - WebAssembly binary module
102+
- `index.js` - ES module exports (already present)
103+
- `index.cjs` - CommonJS exports (already present)
104+
- `index.d.ts` - TypeScript definitions (already present)
105+
106+
### Emscripten Configuration
107+
108+
The WASM module is compiled with these Emscripten settings:
109+
110+
- **Exported Functions**: Core parsing functions like `_wasm_parse_query`, `_wasm_deparse_protobuf`
111+
- **Runtime Methods**: Memory management functions like `lengthBytesUTF8`, `stringToUTF8`
112+
- **Module Format**: ES6 modules with CommonJS compatibility
113+
- **Environment**: Supports both web browsers and Node.js
114+
- **Memory**: Dynamic memory growth enabled
115+
116+
## Testing
117+
118+
### Running Tests
119+
120+
```bash
121+
npm test
122+
```
123+
124+
### Test Requirements
125+
126+
- WASM artifacts must be built before running tests
127+
- Tests cover parsing, deparsing, fingerprinting, normalization, and PL/pgSQL functionality
128+
- Both sync and async API variants are tested
129+
130+
### Test Failure Recovery
131+
132+
If tests fail with "fetch failed" or module initialization errors:
133+
134+
```bash
135+
npm run wasm:clean && npm run wasm:build && npm test
136+
```
137+
138+
## Development Workflow
139+
140+
### Initial Setup
141+
142+
```bash
143+
git clone https://github.com/launchql/libpg-query-node.git
144+
cd libpg-query-node
145+
git checkout feat/17-wasm
146+
npm install
147+
npm run wasm:build
148+
npm test
149+
```
150+
151+
### Making Changes
152+
153+
1. Modify source files in `src/` or wrapper files in `wasm/`
154+
2. Rebuild WASM artifacts: `npm run wasm:build`
155+
3. Run tests: `npm test`
156+
4. Commit changes
157+
158+
### Protocol Buffer Regeneration
159+
160+
If you need to regenerate `proto.js`:
161+
162+
```bash
163+
npm run protogen
164+
```
165+
166+
This uses the configuration in `scripts/protogen.js` with the `17-6.1.0` branch.
167+
168+
## Troubleshooting
169+
170+
### Common Issues
171+
172+
**"fetch failed" errors during tests:**
173+
- **Cause**: Stale or missing WASM artifacts
174+
- **Solution**: `npm run wasm:clean && npm run wasm:build`
175+
176+
**"WASM module not initialized" errors:**
177+
- **Cause**: Attempting to use sync methods before module initialization
178+
- **Solution**: Use async methods or call `loadModule()` first
179+
180+
**Docker permission errors:**
181+
- **Cause**: Docker user permissions mismatch
182+
- **Solution**: Ensure Docker is properly configured for your user account
183+
184+
**Build cache issues:**
185+
- **Cause**: Corrupted or outdated cached libpg_query build
186+
- **Solution**: `npm run wasm:clean-cache && npm run wasm:build`
187+
188+
### Build Environment Issues
189+
190+
**Missing Docker:**
191+
- Install Docker Desktop or Docker Engine for your platform
192+
- Ensure Docker daemon is running
193+
194+
**Node.js version issues:**
195+
- Use Node.js 16 or higher
196+
- Consider using nvm for Node.js version management
197+
198+
**Memory issues during build:**
199+
- Increase Docker memory limits if building on resource-constrained systems
200+
- The Emscripten compilation can be memory-intensive
201+
202+
## Architecture Notes
203+
204+
### WASM vs Native Addon Comparison
205+
206+
| Aspect | Native Addon (old) | WASM (feat/17-wasm) |
207+
|--------|-------------------|---------------------|
208+
| Compilation | Platform-specific | Universal |
209+
| Dependencies | node-gyp, build tools | Docker only |
210+
| Distribution | Pre-built binaries | Single WASM file |
211+
| Performance | Native speed | Near-native speed |
212+
| Compatibility | Platform-dependent | Universal |
213+
214+
### API Compatibility
215+
216+
The WASM implementation maintains full API compatibility with the native addon version:
217+
218+
- All parsing functions (`parseQuery`, `parseQuerySync`)
219+
- All deparsing functions (`deparse`, `deparseSync`)
220+
- Fingerprinting and normalization functions
221+
- PL/pgSQL parsing support
222+
- Error handling and validation
223+
224+
### Module Loading
225+
226+
The WASM module uses lazy initialization:
227+
- Async functions handle initialization automatically
228+
- Sync functions require explicit `loadModule()` call
229+
- Module state is shared across all function calls
230+
231+
## Contributing
232+
233+
When contributing to the feat/17-wasm branch:
234+
235+
1. Ensure all changes maintain WASM-only architecture
236+
2. Test both sync and async API variants
237+
3. Verify cross-platform compatibility
238+
4. Update documentation for any API changes
239+
5. Run full test suite before submitting changes
240+
241+
## Related Documentation
242+
243+
- [libpg_query repository](https://github.com/pganalyze/libpg_query)
244+
- [Emscripten documentation](https://emscripten.org/docs/)
245+
- [WebAssembly specification](https://webassembly.org/)
246+
- [PostgreSQL parser documentation](https://www.postgresql.org/docs/current/parser-stage.html)

0 commit comments

Comments
 (0)