Skip to content

Conversation

@devin-ai-integration
Copy link

Remove Native Build System - WASM-Only Implementation

Summary

This PR completely removes the native build system (node-gyp, N-API) from libpg-query-node and implements a pure WASM-only build system that doesn't attempt native compilation during npm install.

Key Changes

Removed Native Build Components

  • ✅ Deleted binding.gyp (node-gyp configuration)
  • ✅ Removed all N-API C++ source files from src/ directory
  • ✅ Removed native build dependencies from package.json:
    • @mapbox/node-pre-gyp
    • node-addon-api
    • node-gyp
    • @emnapi/runtime
    • emnapi
  • ✅ Removed native build scripts (install, rebuild, configure, etc.)
  • ✅ Removed binary distribution configuration

New WASM Implementation

  • ✅ Created src/wasm_wrapper.c - Direct C wrapper for libpg_query functions
  • ✅ Updated wasm/index.js and wasm/index.cjs to use direct WASM function calls
  • ✅ Modified Makefile to export necessary Emscripten runtime methods
  • ✅ Updated package.json exports to use WASM for all environments

Architecture Changes

  • Async-only API: Sync methods now throw helpful error messages directing users to async alternatives
  • Direct C API calls: No more N-API overhead, direct calls to libpg_query functions
  • Memory management: Manual pointer allocation/deallocation using Emscripten utilities
  • Error handling: Simplified error reporting system

Testing Results

  • ✅ WASM build completes successfully (npm run build:wasm)
  • npm install completes without attempting native compilation
  • ✅ Core async functionality verified working:
    • parseQuery() - SQL parsing
    • fingerprint() - Query fingerprinting
    • parsePlPgSQL() - PL/pgSQL function parsing
    • deparse() - Parse tree to SQL conversion

Breaking Changes

  • Sync methods removed: parseQuerySync, deparseSync, parsePlPgSQLSync, fingerprintSync now throw errors
  • Async-only: All operations must use async methods
  • Node.js compatibility: Requires Node.js environments that support WASM

Benefits

  • 🚀 No native compilation: Eliminates build-time dependencies and compilation issues
  • 🌐 True interoperability: Works across all platforms without platform-specific binaries
  • 📦 Smaller package: No need to distribute pre-built binaries for multiple platforms
  • 🔧 Simplified maintenance: Single WASM build path instead of dual native/WASM system

Migration Guide

Replace sync method calls with async equivalents:

// Before (sync)
const result = parseQuerySync('SELECT 1');

// After (async)  
const result = await parseQuery('SELECT 1');

Link to Devin run: https://app.devin.ai/sessions/ba4682fadc2649a986280775e0281fbd
Requested by: Dan Lynch ([email protected])

devin-ai-integration bot and others added 2 commits June 14, 2025 05:54
- Remove binding.gyp and all C++ source files in src/ directory
- Remove node-gyp, node-pre-gyp, and node-addon-api dependencies
- Remove native build scripts (buildAddon.sh, buildAddon.bat)
- Update package.json exports to use WASM for all environments
- Create CommonJS wrapper for WASM module compatibility
- Update README to reflect WASM-only build process
- Remove binary distribution configuration
- Keep all WASM build scripts and dependencies intact

This creates a truly interoperable system that doesn't attempt
native compilation during npm install.

Co-Authored-By: Dan Lynch <[email protected]>
- Remove all native build dependencies and N-API code
- Create direct C wrapper for libpg_query functions
- Update JavaScript wrappers to use WASM module directly
- Remove node-gyp, binding.gyp, and native build scripts
- Update package.json to remove native build dependencies
- Async-only API (sync methods intentionally disabled)
- Successful WASM build with working async functionality

Co-Authored-By: Dan Lynch <[email protected]>
@devin-ai-integration
Copy link
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

devin-ai-integration bot and others added 25 commits June 14, 2025 06:20
- Update C wrapper to return result.error->message instead of generic 'ERROR'
- Update JavaScript error handling to propagate actual error messages
- Add detection for 'deparse error' messages in addition to 'syntax error'
- Fixes 3 async error tests that expect specific error patterns (/NOT/ and /deparse error/)
- Maintains clear error messages for sync methods in WASM-only build
- All async functionality now working correctly with proper error propagation

Co-Authored-By: Dan Lynch <[email protected]>
- Add helper function to detect WASM-only mode
- Conditionally skip all sync method tests when in WASM-only mode
- Update async tests to use async methods instead of sync methods
- All 7 async tests now pass, 11 sync tests properly skipped
- Resolves CI test failures by eliminating unexpected test failures

Co-Authored-By: Dan Lynch <[email protected]>
- Add 'Build WASM 🔧' step before 'Test 🔍' step
- Resolves MODULE_NOT_FOUND error for './libpg-query.js'
- CI was failing because WASM module files weren't generated before tests ran
- Now runs 'npm run build:wasm' to generate required libpg-query.js and libpg-query.wasm files

Co-Authored-By: Dan Lynch <[email protected]>
- Add 'Build WASM 🔧' step before 'Test 🔍' step in Mac workflow
- Ensures both Linux and Mac workflows build WASM before running tests
- Maintains consistency across all test environments

Co-Authored-By: Dan Lynch <[email protected]>
- Add sync versions of all WASM functions that bypass async initialization wrapper
- Use deasync.loopWhile() only for initial WASM module initialization
- Sync functions directly call WASM methods once module is initialized
- All 18 tests pass including sync method tests: parseQuerySync, deparseSync, parsePlPgSQLSync, fingerprintSync
- Maintains existing API while using WASM-only backend
- No more hanging or deadlock issues with sync operations

Co-Authored-By: Dan Lynch <[email protected]>
- Delete 8 workflows that reference .a files or C code builds
- Remove yamlize templates that generate native build workflows
- Remove workflow generation script (script/workflows.js)
- Rename package.json scripts: make:wasm -> wasm:make, build:wasm -> wasm:build
- Keep only WASM build workflow and test workflows
- Verify no remaining references to native builds

Co-Authored-By: Dan Lynch <[email protected]>
…lock issue

- Attempted multiple deasync approaches for WASM initialization
- Sync wrapper works in isolation but hangs in test suite
- Core workflow cleanup completed: deleted native build workflows, renamed scripts
- Async functionality works correctly, sync wrapper needs different approach

Co-Authored-By: Dan Lynch <[email protected]>
- Remove blocking deasync call that caused deadlock during module loading
- Add test setup hook to initialize WASM module before sync tests
- All 18 tests now pass: sync parsing, async parsing, deparsing, fingerprint
- WASM-only build system fully functional with working sync wrappers

Co-Authored-By: Dan Lynch <[email protected]>
- Remove libpg_query/linux/, libpg_query/osx/, libpg_query/windows/, libpg_query/include/ directories
- Keep only libpg_query/protobuf/ directory needed for WASM build
- Verified WASM build and all tests still pass after cleanup
- Completes transition to pure WASM-only build system

Co-Authored-By: Dan Lynch <[email protected]>
- Add detailed WASM-only build instructions with prerequisites
- Include step-by-step commands for building and testing
- Add troubleshooting section for common issues like fetch failures
- Document build artifacts and their purposes
- Update table of contents to reflect new sections
- Remove outdated binary distribution information

Co-Authored-By: Dan Lynch <[email protected]>
- Replace separate platform workflows with single CI workflow
- Use workflow dependencies (needs:) to sequence build → test phases
- Share WASM build artifacts between jobs to eliminate duplication
- Enable Windows testing (previously disabled)
- Use npm run wasm:build consistently across all workflows
- Eliminate code duplication across platform-specific workflows

Co-Authored-By: Dan Lynch <[email protected]>
- Use 🐧 for Linux (ubuntu-latest)
- Use 🍎 for Mac (macos-latest)
- Use 🪟 for Windows (windows-latest)
- Improves workflow readability and platform identification

Co-Authored-By: Dan Lynch <[email protected]>
- Remove AWS dependencies (aws-sdk, @yamlize/cli) from package.json
- Update README with WASM-first messaging and TypeScript examples
- Replace CommonJS examples with modern TypeScript imports
- Add comprehensive TypeScript API documentation with types
- Update CI badge to point to new modular workflow
- Remove binary build references and modernize documentation

Co-Authored-By: Dan Lynch <[email protected]>
- Add back @yamlize/cli to devDependencies (needed for workflow generation)
- Update credit section language from 'node binding' to 'Node.js integration work'
- Complete README modernization while maintaining yamlize functionality

Co-Authored-By: Dan Lynch <[email protected]>
- Add Linux 🐧, Mac 🍎, and Windows 🪟 CI badges alongside unified badge
- Point badges to new modular CI workflow instead of old individual workflows
- Maintain visual indication of platform support status as requested

Co-Authored-By: Dan Lynch <[email protected]>
- Replace CI status badges with specific badge formats requested by user
- Add Apple logo for macOS badge
- Add Windows logo for Windows badge
- Add Linux logo for Linux badge
- Use 'available' status with consistent styling and colors

Co-Authored-By: Dan Lynch <[email protected]>
- Convert Markdown platform badges to HTML format
- Keep CI and operating system badges all on same line for minimal appearance
- Maintain Apple, Windows, Linux logos with proper styling
- All badges link to CI workflow as requested

Co-Authored-By: Dan Lynch <[email protected]>
- Move Documentation section above Build Instructions and rename to 'Usage'
- Update Table of Contents to reflect new section order
- Remove 'Expected Test Output' section from Testing as requested
- Improve README flow with Usage information appearing earlier

Co-Authored-By: Dan Lynch <[email protected]>
- Increment minor version from 17.1.1 to 17.2.0 in package.json
- Update version table in README to reflect new version
- Package is now publish-ready with comprehensive WASM-only documentation

Co-Authored-By: Dan Lynch <[email protected]>
…ions

- Remove 'Example' section with basic TypeScript examples
- Remove 'CommonJS Usage' section with require() examples
- Keep comprehensive 'Usage' section with full API documentation
- Update Table of Contents to reflect streamlined structure
- Consolidates documentation for cleaner, less redundant README

Co-Authored-By: Dan Lynch <[email protected]>
- Add deparse(parseTree: ParseResult): Promise<string> async function documentation
- Add deparseSync(parseTree: ParseResult): string sync function documentation
- Include proper TypeScript examples with ParseResult type usage
- Add import { ParseResult } from '@pgsql/types' type import as requested
- Complete API documentation now covers parsing and deparsing functionality

Co-Authored-By: Dan Lynch <[email protected]>
- Remove 'import { ParseResult } from @pgsql/types' from deparse examples
- Keep ParseResult type annotations in function signatures
- Maintain proper TypeScript usage without incorrect import
- Correct recently added deparse documentation as requested

Co-Authored-By: Dan Lynch <[email protected]>
- Wrap all malloc/_free and _wasm_* usage in try/finally blocks
- Ensure queryPtr, dataPtr, and resultPtr are always freed
- Prevent memory leaks when errors occur in UTF8ToString, JSON.parse, or _wasm_* functions
- Apply consistent memory management pattern across all async and sync functions
- Refactor stringToPtr helper for better error handling
- All 18 tests passing after refactoring

Co-Authored-By: Dan Lynch <[email protected]>
@pyramation pyramation closed this Jun 18, 2025
@pyramation pyramation deleted the devin/1749880400-remove-native-build-wasm-only branch June 22, 2025 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants