Built-in HTML parser and DOM

[vjpr]

I wonder if we could offer built-in support for an HTML parser and the DOM.

Use cases

• Web scraping
  • Quite common, and perf is important here because it's usually done at scale.
• Testing
  • A lot of testing is done in a Node.js environment using js-dom. Speeding up test suites is always cool.
  • With SSR on the rise, this is becoming more important, rather than traditionally relying on client-side E2E testing in a real browser or wrapper such as Cypress or using Puppeteer or Playwright.

Why Bun should do this?

• Bun already has WebKit in its source tree. We could just re-use the existing implementations there. E.g. https://github.com/WebKit/webkit/blob/main/Source/WebCore/dom/DOMImplementation.cpp / https://github.com/WebKit/webkit/blob/main/Source/WebCore/html/parser/HTMLDocumentParser.cpp

• Demonstrating perf benefits for web scraping and DOM testing could push adoption.

• Seamless access to web platform from server. Currently you have to think: "am I in browser env or node env?" when doing web stuff.

Existing HTML/DOM Libraries

A lot is built around htmlparser2: https://github.com/fb55/htmlparser2/#ecosystem

https://cheerio.js.org/ - parse html and run jquery-like selectors

Cheerio wraps around parse5 parser and can optionally use @fb55's forgiving htmlparser2 (TypeScript). Cheerio can parse nearly any HTML or XML document.

https://github.com/jsdom/jsdom - subset of web browser for testing and scraping and can execute js

https://github.com/fb55/htmlparser2

htmlparser2 is the fastest HTML parser

See: https://github.com/fb55/htmlparser2/#performance

If you need strict HTML spec compliance, have a look at parse5.

https://github.com/inikulin/parse5

As of August 2021, parse5 is the only fully spec compliant HTML parser available for NodeJS
From: https://github.com/AndreasMadsen/htmlparser-benchmark

Benchmarks

Here are the benchmarks to beat:

• https://github.com/AndreasMadsen/htmlparser-benchmark
  • Current results: https://github.com/fb55/htmlparser2/#performance

Existing Testing Libraries

Jest

Custom environments

Jest allows you to choose an env (node vs jsdom) or use a custom one

https://jestjs.io/docs/configuration#testenvironment-string

Sandboxing

TestEnvironment is sandboxed. Each test suite will trigger setup/teardown in their own TestEnvironment.

Jest creates a new jsdom environment for each test

Many testing libraries do this which could be quite processor intensive - hence more opportunities for perf wins.

Renderer

If we bring in the parser and DOM now, in addition to JS engine, the only thing left now is the renderer.

After bringing in parsing/dom, the two parts left would be:

• Headless browser
  • already quite a crowded space (https://github.com/dhamaniasad/HeadlessBrowsers)
• Desktop browser
  • Bun replacement for Electron/Tauri.
    • A layer for Bun.js to talk to a browser renderer.
    • Existing discussion: Desktop support (Electron replacement) #790
  • Testing browser
    • Cypress is pretty cool - a desktop Chromium wrapper with a nice driver api.
      • It doesn't feel as integrated as it could be though.
      • I feel like the goal should be: "here is a chunk of TypeScript code, and here is where I want to run, and run it as fast as possible".

I guess the advantage of doing a browser in Bun would be to offer seamless interaction with the entire web platform. At present it's quite patchwork - there is a lot of existing native WebKit code that has been re-implmented (slower) in userland JS. And to access browser environments you need a lot of additonal libraries.

[vjpr]

tricked:

bun has this in deps https://github.com/cloudflare/lol-html so i would think its maybe somewhere already not sure tho

---

jarred:

LOLHTML doesn’t really have a full-fledged DOM
It’s designed for streaming and rewriting
rather than ASTs
I think something in here is a good ide a
But probably not for awhile
CSS is higher priority

[mquandalle]

Deno has https://deno.land/manual@v1.36.4/advanced/jsx_dom/deno_dom

[ethanchristensen01]

I was using HTMLRewriter (LOLHTML) to work with an XML document and ran into problems when the document had Style tags. Children of Style were processed as raw text instead of as elements. I would like to work with LOLHTML to add support for xml documents.

[The-Code-Monkey]

@vjpr did you find a solution for this as I need to use parseFromString from DOMParser but cannot as it doesn't exist in bun.

[The-Code-Monkey]

How would I use JSDOM to parse a html string into html elements

…

On Sat, 9 Dec 2023, 18:15 guest271314, ***@***.***> wrote: There is JSDOM https://github.com/guest271314/jsdom-extension. — Reply to this email directly, view it on GitHub <#1522 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHOFWX5UPVMGA27DWTI6S3LYISTDPAVCNFSM6AAAAAASEPLCL6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMBYGI2TK> . You are receiving this because you commented.Message ID: ***@***.***>

[The-Code-Monkey]

In the browser, I have a rich text editor, that requires html code but it stores the data as a string of that code, so I need to parse e.g. "<p>some text<p>" to an actual html element. Which is what DOMParser.parseFromString does, but DOMParser isn't part of bun yet, so I need some way to do this.

…

On Sat, 9 Dec 2023, 23:02 guest271314, ***@***.***> wrote: HTML elements where, in what environment? — Reply to this email directly, view it on GitHub <#1522 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHOFWX3KUL43QFEQQZX2MPTYITUZDAVCNFSM6AAAAAASEPLCL6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMBZGI2TK> . You are receiving this because you commented.Message ID: ***@***.***>

[The-Code-Monkey]

I need a browser based solution not a node one.

…

On Sun, 10 Dec 2023, 12:39 guest271314, ***@***.***> wrote: @The-Code-Monkey <https://github.com/The-Code-Monkey> Have you tried using the JSDOM code that I compiled for use in a browser extension ServiceWorker, linked above? Should work. — Reply to this email directly, view it on GitHub <#1522 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHOFWXZZ6VZ3CSF23VHTEY3YIWURXAVCNFSM6AAAAAASEPLCL6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMJRGM4TO> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[The-Code-Monkey]

DOMParser doesn't work with bun as far as I could tell unless I just have my typescript settings wrong

[vinyll]

Upgrading Lego to V2 we are trying to minimize external dependencies.
Bun was favorited to achieve this, however it is missing a dom parser like Deno has.
Is there any plan to embed this in the short/mid term?

[21e8]

+1 would love this

[user451421541757324]

CSS is higher priority

I was thinking whether, since the CSS feature was implemented with Bun 1.2, we could think about a DOM parser.

[MrDanielHarka]

+1! Would be nice!

[jamesgarrick]

+1

[mefistofelix]

For a modern native dom/xpath/xslt implementation instead of libxml2 which supports only xpath 1.0 see
https://blog.startifact.com/posts/xee/

About an html5 parser the latest versions of PHP are using
https://github.com/kostya/lexbor

Also adding a builtin json query language will be really cool, the most powerfull one (also used in the aws stack in some place) is:
https://github.com/jsonata-js/jsonata
Will be cool to try use some ai to port the JS to zig and jscore object access to have a builtin native implementation, because is a JS only lib

This additions will be really cool imho, Not only for web scraping but are a super powerfull toolbox for a wide range of common requirements nowdays, in one way or another.

[mefistofelix]

If using dom and HTML parser from WebKit i think you will inherit the dom event system, the mutation observer etc in a more native way than the one exposed in browsers to js... This could open to implementing in bun some interesting GUIs (os native or Not) that have the dom as underlying markup and source of truth/state/GUI/game scene graph...something like xaml and qml (Qt)

[peterlustig78]

as bun build is able to read html files for used script imports and other files to properly bundle them, bun should be able to parse it already using some internal feature.
Not sure how it does it, and if it is useable in JS code.