GS Datastream

[CedricGuillemet]

• Decompression support for Gaussian Splatting data
• Plugin is named DataStream (shall it be a polyfill ?) and register a few new objects: DecompressionStream, ReadableStream, Response, TextDecoder
• Adding miniz and libdeflate depencies. miniz only support 1 type of gzip compression. libdeflate doesn't support file hierarchy. So, adding both for gzip (SPZ) and zip (SOG)
• It's not meant to be a full implementation but rather a working solution so this code runs : https://github.com/BabylonJS/Babylon.js/blob/master/packages/dev/loaders/src/SPLAT/splatFileLoader.ts#L252-L267
• more PRs to come with testing and documentation
• Extend Canvas Image with new methods. SOG format is based on webp.

Performances 1

I use this js code:

console.log(fflate.unzipSync); // confirm expected version is being used

BABYLON.Tools.LoadFile("https://raw.githubusercontent.com/CedricGuillemet/dump/master/sog-files/MardiGrasDragon.sog", (data) => {
    const datau8 = new Uint8Array(data);
    const t0 = Date.now();
    for (let i = 0; i < 1000; i++) {
        const unzipped = fflate.unzipSync(datau8);
    }
    const t1 = Date.now();
    console.log("Call took", (t1 - t0), "ms");
}, undefined, undefined, true);

and switch between Native and JS versions by calling (or not) this cpp code:

loader.LoadScript("https://unpkg.com/fflate/umd/index.js");

.sog file is a 14Mb zip containing webp textures with a ratio close to 1:1. 1000 decompressions.

JSC / macOS / M3

JS: 2630 ms
Native: 1671 ms

Native is faster but I expected way more difference due to JIT missing. Might be because of compression ratio.

Chakra / W11 / IntelXeon

JS: 7530 ms
Native: 6600 ms

V8 / W11 / IntelXeon

JS 12500 ms
Native: 12300ms

V8 slower than Chakra. Why?

Profiles

V8/W11

Chakra/W11

Performances 2

As I expected, SOG files with poor compression don't show potential for Native decompression.
Test2 is a zip file containing Babylon.max.js and babylon.module.d.ts. 19.4Mb compressed into a 3.37Mb zip.
Replace LoadFile line with:

BABYLON.Tools.LoadFile("https://raw.githubusercontent.com/CedricGuillemet/dump/master/sog-files/bjs.zip", (data) => {

1 single decompression.

Chakra / W11 / IntelXeon

JS: 5600ms
Native: 51ms

V8 / W11 / IntelXeon

JS 202 ms
Native: 45 ms

2nd round with 100 decompressions instead of 1
JS : 11400 ms
Native: 5483 ms

JSC / macOS / M3

JS: 840 ms
Native: 28 ms

2nd round with 10 decompressions instead of 1
JS: 8700 ms
Native: 240 ms

TLDR

When compression ratio is low like with SOG (zipped webp), decompression performance difference between JS and Native is less visible as unpacking is mainly memory copy.
The better the compression, the faster in comparison the Native version is. Up to 100X faster than Chakra, 30X faster than NoJIT JSC and 2X than V8 + JIT.
No idea why 1st performance test makes V8 2 times slower than Chakra.

[ryantrem]

@alexchuber introduced a more general NativeEncoding plugin a few weeks ago. It currently only has image encoding, but the idea was for it to be a general purpose plugin for different encoding/decoding related things. To make sure we could still get the full benefits of not linking anything that is not used, we talked about the idea of having multiple header includes for the plugin rather than the more typical single header. This way you could choose specific parts of the NativeEncoding plugin that you want. Seems like since this is very much an encoding/decoding feature, it should just go in NativeEncoding rather than introducing a new plugin. Thoughts @alexchuber and @bghgary?

[alexchuber]

Not very familiar with the domain, but my take:

Since decompression is isolated to DecompressionStream and the web API is narrowly scoped (it only handles gzip/deflate), using it as a polyfill seems reasonable.

For the unzipping functionality in the DataStream plugin, I don't think it fits under NativeEncoding-- nor does it have an equivalent web API, as far as I know-- so I guess having it in its own plugin makes sense.