Community Packages with Native Modules

[confused-Techie]

This list will continue to be updated, as the process is only semi-automated at this time.
This issue will contain a checklist of community packages that contain Native Modules. As many native modules will stop working once we officially role out PulsarNext.

So this list will be a list of the most at risk community packages that will require testing in PulsarNext to verify if they are broken or continue to work.

If a package still works on PulsarNext we can check it off the list to indicate it's still functional, and for everything still on the list we will then want to look even closer to see what can be done to get that package working, or if we will need to anticipate it breaking after the PulsarNext release.

(If more information is needed for this list I have it in the logs of the scripts I'm using to check packages, and it can easily be added along with package names, so feel free to request it's addition)

---

(For clarity, the names here are the packages name on the Pulsar Package Registry, it links to the packages GitHub, and they are listed in order of downloads. The text after each package is a comma separated list of dependencies that caused them to be flagged)

Note: A warning, if a package fails to install, we won't be able to determine it's native modules and it won't appear on this list, so there is a chance of those packages becoming false negatives for inclusion. Additionally, a package's devDependencies may cause it showing up here to be a false positive.

• platformio-ide-terminal: napi-build-utils, node-pty-prebuilt-multiarch
• linter-ui-default: lmdb-store, msgpackr-extract, @parcel/source-map, @parcel/watcher
• minimap: lmdb-store, msgpackr-extract, @parcel/source-map, @parcel/watcher
• intentions: lmdb, msgpackr-extract, @parcel/source-map, @parcel/watcher
• hydrogen: zeromq
• atom-ide-ui: nuclide-prebuilt-libs/ctags, nuclide-prebuilt-libs/fuzzy-native, nuclide-prebuilt-libs/keytar, nuclide-prebuilt-libs/pty
• atom-typescript: @parcel/source-map, @parcel/watcher
• ide-python: zadeh
• teletype: @newrelic/native-metrics
• terminal-plus: pty.js
• autocomplete-paths: lmdb, msgpackr-extract, @parcel/source-map, @parcel/watcher
• seti-ui: ttf2woff2
• markdown-preview-plus: @parcel/source-map, @parcel/source-map
• julia-client: napi-build-utils, node-pty-prebuilt-multiarch
• tree-view: pathwatcher
• language-javascript: tree-sitter-javascript, tree-sitter-jsdoc, tree-sitter-regex
• language-python: tree-sitter-python
• symbols-view: ctags
• language-ruby: napi-build-utils, tree-sitter-ruby
• fuzzy-finder: @atom/fuzzy-finder
• spell-check: @atom/fuzzy-native
• language-go: tree-sitter-go
• language-c: tree-sitter-c, tree-sitter-cpp
• bracket-matcher: oniguruma
• language-html: tree-sitter-embedded-template, tree-sitter-html
• language-css: tree-sitter-css
• language-java: tree-sitter-java-dev
• language-shellscript: napi-build-utils, tree-sitter-bash
• ide-html: zadeh
• language-json: tree-sitter-json
• atom-ide-outline: zadeh
• atom-ide-markdown-service: zadeh
• ide-haskell: @parcel/source-map, @parcel/watcher
• vk-terminal: pty.js
• ide-css: zadeh
• node-debugger: dtrace-provider
• jupyter-notebook: pathwatcher, ws
• magicpython: oniguruma
• ide-rust: zadeh
• terminal-tab: napi-build-utils, node-pty-prebuilt-multiarch
• pymakr: napi-build-utils
• script-runner: node-pty
• ide-yaml: zadeh
• github: keytar, leveldown, napi-build-utils, superstring, napi-macros/example
• language-typescript: tree-sitter-typescript
• termination: napi-build-utils, node-pty-prebuilt-multiarch
• x-terminal: napi-build-utils, node-pty-prebuilt-multiarch
• terminus: napi-build-utils, node-pty-prebuilt-multiarch
• ide-ruby: zadeh
• atom-quokka: keytar, napi-build-utils, quokka-atom/build/extension/dist/wallaby -> bufferutil, quokka-atom/build/extension/dist/wallaby -> keytar, quokka-atom/build/extension/dist/wallaby -> napi-build-utils, quokka-atom/build/extension/dist/wallaby -> utf-8-validate, quokka-atom/build/extension/dist/wallaby -> weak,
• atom-ide-vue: zadeh
• dash-ui: ttf2woff2
• process-palette: napi-build-utils, node-pty-prebuilt-multiarch
• dart: ws
• arduino-upload: napi-build-utils, usb-detection, @serialport/bindings-cpp
• json-schema: tree-sitter, tree-sitter-json, @tree-sitter-grammars/tree-sitter-yaml, @swagger-api/apidom-parser-adapter-yaml-1-2 -> tree-sitter
• quick-query: ssh2
• remote-edit2: cpu-features, ssh2
• autocomplete-ctags: ctags
• linter-xo: fs-admin, napi-build-utils, pathwatcher, superstring (@savetheclocktower edited to add: devDependencies lists a dependency on text-buffer, which at least explains the pathwatcher and superstring usages; not sure about the others)
• remote-edit-ni: platform-folders
• atom-keyboard-macros: pathwatcher
• tokamak-terminal: pty.js
• javascript-regex: tree-sitter-regex-js, tree-sitter-regex-unicode-js
• language-javascript-semantic: oniguruma
• iv-terminal: napi-build-utils, node-pty-prebuilt-multiarch
• lang-gdscript: napi-build-utils, tree-sitter-gdscript, tree-sitter-godot-resource, zadeh
• atom-import-js: sqlite3
• atom-ide-debugger-native-gdb: nuclide-prebuilt-libs/ctags, nuclide-prebuilt-libs/fuzzy-native, nuclide-prebuilt-libs/keytar, nuclide-prebuilt-libs/pty
• xterm: pathwatcher, ptyw.js
• font-viewer: freetype2
• ide-svelte: zadeh, @parcel/watcher
• ide-kotlin: zadeh
• language-lua-plus: napi-build-utils, tree-sitter-lua
• chlorine: bufferutil, utf-8-validate
• motepair: ws
• language-marko: zadeh
• latex-wordcount: oniguruma
• language-agda: tree-sitter-agda
• ide-crystal: fuzzaldrin-plus-fast
• mcfunction-lang: tree-sitter-mcfunction-lang
• seti-black-ui: ttf2woff2
• jump-to-import: ctags
• symbols-view-plus: ctags
• slack-chat: ws
• language-legesher-python: lzma-native, tree-sitter-legesher-python
• nrepl: pathwatcher
• language-latex2e: napi-build-utils, tree-sitter-biber
• atomic-terminal: lmdb-store, msgpackr-extract, napi-build-utils, node-pty-prebuilt-multiarch, @parcel/source-map, @parcel/watcher
• atom-sassc-live: pty.js
• compile-watch: node-sass
• atom-reveal-md: ws
• broadcast: oniguruma
• markdown-preview-kramdown: oniguruma
• serenade: bufferutil, utf-8-validate
• language-ags-script: napi-build-utils, tree-sitter-ags-script
• language-circuitpython: napi-build-utils, @serialport/bindings
• language-chialisp: tree-sitter-chialisp
• ide-sourcekit: zadeh
• atom-ide-deno: zadeh
• ide-intelephense: zadeh
• spell-checker-improved: spellchecker
• voicecode: ws
• game-data-explorer: lzo
• language-prisma: tree-sitter-prisma
• athena-ide-atom: grpc, secp256k1
• js-refactor-atom: approvals -> edge
• atom-pgp: keyboard-layout
• ide-haskell-hls: zadeh
• motepair2: ws
• language-roff: oniguruma
• nola-sdk: lzma-native, napi-build-utils, @serialport/bindings
• latex-log: oniguruma
• markmon-preview: ws
• linter-biber: napi-build-utils, tree-sitter-biber
• pulsar-package-manager: git-utils, keytar, napi-build-utils, npm -> node-gyp/test -> hello_world, npm-lifecycle -> node-gyp/test -> hello_world
• tree-sitter-syntax-visualizer: napi-build-utils, tree-sitter
• derive-beta: napi-build-utils, sharp
• language-zig-treesitter: tree-sitter-zig
• language-python-treesitter: tree-sitter-python
• build-haxe: node-expat
• language-lsl2: tree-sitter-lsl2
• linkist: keytar, napi-build-utils
• ide-d: lmdb, lzma-native, msgpackr-extract, zadeh, @parcel/source-map, @parcel/watcher
• agsbs-atom-package: shell.js/src -> ws
• greyatom-tree-view: ws
• atom-dione: bufferutil, utf-8-validate
• spell-check-case: spellchecker
• language-4dm: tree-sitter-4dm
• hydrogen-next: deasync, zeromq
• neo-language-javascript: tree-sitter-javascript, tree-sitter-jsdoc, tree-sitter-regex
• language-zig-zls: zadeh
• atom-xterm-plugin: pty.js
• shrink-whitespace: fs-admin, napi-build-utils, pathwatcher, superstring
• pulsar-ide-ui: nuclide-prebuilt-libs/ctags, nuclide-prebuilt-libs/fuzzy-native, nuclide-prebuilt-libs/keytar, nuclide-prebuilt-libs/pty
• language-janet: tree-sitter-janet
• markdown-spec-formatter: lmdb, msgpackr-extract, @parcel/source-map, @parcel/watcher
• atom-ide-volar: zadeh
• notify-atom: bufferutil, utf-8-validate
• language-carp: tree-sitter-carp
• language-elm-treesitter: napi-build-utils, tree-sitter-elm
• dsp-atom-plugin: bufferutil, utf-8-validate, ws
• molecule: nuclide-prebuilt-libs/fuzzy-native, nuclide-prebuilt-libs/keytar, nuclide-prebuilt-libs/pty
• language-howl: tree-sitter-howl
• live-emojing: websocket
• significant-other: git-utils
• jump-to-line: nuclide -> ws, nuclide/pkg/nuclide-ctags-rpc/VendorLib/ctags-prebuilt, nuclide/pkg/nuclide-fuzzy-native/VendorLib/fuzzy-native, nuclide/pkg/nuclide-debugger-node-rpc/VendorLib -> v8-debug, nuclide/pkg/nuclide-debugger-node-rpc/VendorLib -> v8-profiler
• rid-preview: libxmljs
• pdmsg: napi-build-utils, tree-sitter
• puddletown-language-javascript: tree-sitter-regex
• language-lux: tree-sitter-lux
• terminal-tab-fork: napi-build-utils, node-pty-prebuilt-multiarch
• language-4dpreferences: tree-sitter-4dxml
• atomforces: lzma-native, node-pty
• recognizer: bufferutil, utf-8-validate
• pulsardo: bufferutil, utf-8-validate
• build-ibmstreams: keytar, napi-build-utils
• codesync: bufferutil, utf-8-validate
• language-4dform: tree-sitter-4djson
• do-not-track: @parcel/watcher
• language-fudomo: fudomo-transform
• auto-sass: @parcel/watcher
• atom-templatebuilder-tools: ws
• gdx-viewer: node-gdx
• language-4dcatalog: napi-build-utils, tree-sitter, tree-sitter-4dxml
• coconut: socket.io-client -> engine.io-client -> ws
• language-eu4mod: tree-sitter-eu4mod
• language-4dproject: tree-sitter-4djson
• divinefingers: fsevents
• language-morph: napi-build-utils, tree-sitter, tree-sitter-morph
• pulsar-ide-rust: zadeh
• language-4dlink: napi-build-utils, tree-sitter, tree-sitter-4dxml
• atom-refs: napi-build-utils, tree-sitter, tree-sitter-python
• camel-lsp-client-atom: zadeh
• language-haiku: tree-sitter-haiku
• atom-markshow: ws, engine.io-client -> ws
• language-wms: napi-build-utils, tree-sitter-wms
• parsing: kerberos
• language-forthel: git-utils
• language-apertium: tree-sitter-cg, tree-sitter-lexd, tree-sitter-rtx, tree-sitter-twolc
• amd-navigator: ctags
• language-hclrs: tree-sitter-hclrs
• substrate-plugin: secp256k1, websocket
• language-tablatal: tree-sitter-tablatal
• language-turbowave: tree-sitter-turbowave
• flatpak-builder-runner: pty.js
• language-fuior: napi-build-utils, tree-sitter-fuior
• language-gleam: lxma-native, tree-sitter-gleam
• hide: napi-build-utils, tree-sitter, tree-sitter-firrtl, tree-sitter-mlir, tree-sitter-systemrdl, vcd-stream
• edgedb: keytar, napi-build-utils, oniguruma
• language-regent: napi-build-utils, tree-sitter-regent
• atom-nodeppt-oncemore: ws, engine.io-client -> ws
• apx-test: apx-native-module
• faketerm2: lzma-native
• git-tabular-diff: sqlite3
• language-edl: tree-sitter-edl
• molecules: grpc
• puddletown-language-html: tree-sitter-html
• language-prolog-treesitter: tree-sitter-prolog
• rkamyp2: napi-build-utils, @serialport/bindings
• language-langx: tree-sitter-langx
• kotlin-lsp-adapter: zadeh
• ide-intelephense-forked: zadeh
• 0.1.0: zadeh
• language-alv: tree-sitter-alv
• atom-talky-test: ws
• midi-shortcuts: napi-build-utils, robotjs
• notify: napi-build-utils, sharp
• symbol-provider-ctags: ctags
• language-kython: tree-sitter-kython
• lazuli: bufferutil, utf-8-validate
• language-ptl: napi-built-utils, tree-sitter, tree-sitter-ptl
• language-coccinelle: tree-sitter-coccinelle
• language-tspec: tree-sitter-tspec
• language-ql: tree-sitter-ql
• language-ride: tree-sitter-ride
• language-azor: lzma-native, tree-sitter-azor
• language-spiral: tree-sitter-spiral
• language-applesoft: tree-sitter-applesoft
• pmkbeta: napi-build-utils, @serialport/bindings
• language-yolol: napi-build-utils, tree-sitter-yolol
• language-gcl: tree-sitter-gcl
• language-naz: lzma-native, tree-sitter-naz
• language-kv1: tree-sitter-kv1
• language-zelus: tree-sitter-zelus
• language-wdl-ts: tree-sitter-wdl
• pulsar-ide-biome: zadeh
• language-nwscript: tree-sitter-nwscript
• metals-lsp-adapter: zadeh
• language-raidelve: tree-sitter-delve-language
• zeitgeist: mmmagic
• language-bogl: tree-sitter-bogl
• language-roll20-script: lzma-native, tree-sitter-roll20-script
• language-menhera: tree-sitter-menhera
• language-tis3d: tree-sitter-tis-3d
• language-jacy: lzma-native, tree-sitter-jacy
• atom-layerfile: zadeh

Packages Checked: 10,777/10,777

[asiloisad]

hydrogen doesn't work in PulsarNext, but there is a replacement hydrogen-next

[confused-Techie]

@asiloisad Thanks for your input, good to know there's an alternative we can work with pending the release, so we will want to make sure and leave hydrogen unchecked

[savetheclocktower]

I am curious about which dependency (or sub-dependency) uses native modules for each of these. For instance, linter-ui-default is a surprising one to me. How are you detecting the presence of native modules?

[confused-Techie]

@savetheclocktower I'm using the npm package native-modules.
I can certainly add the dependencies that are triggering the check to flag them.

In linter-default-uis case it's specifically:

• lmdb-store
• msgpackr-extract
• @parcel/source-map
• @parcel/watcher

[savetheclocktower]

@savetheclocktower I'm using the npm package native-modules. I can certainly add the dependencies that are triggering the check to flag them.

In linter-default-uis case it's specifically:

* `lmdb-store`

* `msgpackr-extract`

* `@parcel/source-map`

* `@parcel/watcher`

Does that package distinguish between dependencies and devDependencies? The last two of those in the list, for instance, are things that would make sense to be needed during development, but not at runtime.

[savetheclocktower]

To answer my own question: it doesn't, but that's OK. There will be some number of false positives here for that reason. And some packages with native modules may not actually load them in the renderer; some of them may be devDependencies, and others may run in a worker process (though that's less likely).

[savetheclocktower]

One way to check this would be something like what we'll end up doing in (ideally) a few weeks anyway: produce a Pulsar (not PulsarNext) build on Electron 30 so that it loads from .pulsar rather than .pulsar-next. At that point it'll consume a bunch of packages that were installed under Electron 12.2.3… and some number of them will fail to load, or need rebuilding, or whatever.

So the best possible way to test this would be to actually install each of the X most popular packages (leaving out the ones that fail to install properly, of course) under Pulsar 1.130.1… and then upgrade to Electron 30 and see what happens.

(To be clear: this first pass is still useful, though!)

[confused-Techie]

Exactly, we will still have to manually test, but this at least gives us the list of packages that we should have installed when we do that test. And while I'm still not nearly through with our packages, I'm glad it's just a handful of them out of the 270 most popular that I've checked so far. Plus a lot of these are original Atom publications that we already have migrated, and can likely be removed from the registry entirely. But great point about the false positive possibilities from devDependencies

[confused-Techie]

I have now finished checking every single community package for native modules.
Like mentioned there may be some false negatives and positives, but hopefully this list comprises of the bulk of packages we need to be on the lookout for with the upcoming PulsarNext release.

[confused-Techie]

I've tested the following packages through the PulsarNext upgrade process and encountered zero issues. Attempting to confirm functionality as well as possible:

• linter-ui-default
• minimap
• intentions
• autocomplete-paths
• language-jacy
• atom-layerfile

I'll go ahead and checkmark them on the above list to indicate they should be fine.

[savetheclocktower]

I checked off a handful of the packages listed above because they're built-in packages — so they'll get built correctly for the target Electron version automatically as part of the binary building process.

Even if that weren't so: the language packages that currently list tree-sitter-* dependencies no longer have those dependencies (at least not if they're built-in), since we've converted those to WASM and web-tree-sitter. Relatedly: any non-built-in language packages that use Tree-sitter will stop working (or at least the legacy Tree-sitter grammars will), and there's no way around that.

[confused-Techie]

@savetheclocktower Good call with the community tree-sitter grammars.

Do you think there's any value in reaching out to anyone who's published a tree-sitter package to warn them of this and maybe point them to our docs or your blog series on converting to WASM grammars? Or maybe it's more important we find a way to indicate they will be broken on the package registry.

[confused-Techie]

Also checked for anymore built-in packages, checking off:

• fuzzy-finder
• spell-check
• bracket-matcher
• github
• pulsar-package-manager

Since even if those need to be fixed, it wouldn't be through this method of testing, but more than likely they are already totally updated after all the work that's being done on this front.

[savetheclocktower]

Or maybe it's more important we find a way to indicate they will be broken on the package registry.

This is definitely the most important part. The phrasing is important, too: any package can have any number of grammars inside of it, so if a package's legacy Tree-sitter grammar isn't going to work anymore, that doesn't necessarily mean that the package itself isn't worth installing. So the badge we give them shouldn't necessarily signify “broken” — more “this might affect you, so be sure to look into it.”

My suspicion is that these grammars are somewhere on the “long tail” of package usage, and I have no idea if anyone is still using them. I'd be very surprised if anyone were maintaining any of these grammars, but I guess we'll find out.

If I think of it, I'll look through this list and figure out if any of them have had updates published in the Pulsar era.