Use squasfs for conda environment

[martenrichter]

Still, Jupyter Lite with Xeus kernel suffers from long startup times.
I concluded that little tweaks would not help overcome the problem.

One main time-consuming part for the startup is downloading and decompressing the tar archives.
So it would be a better idea to store the environment in a squashfs. (Only download files really required!)
I have looked into the Emscripten source code, and the new WASMFS should allow a relatively simple implementation (though it has to be included in Emscripten).
One caveat is that it is only performant if the browser and server allow caching of HTTP range downloads. This may add some complexity.
It will also add a decompression step for file access.

I do not know if and when I would invest time in this idea. Before the question, does this solution sound worth pursuing?

[martinRenou]

Thanks @martenrichter for your interest in making jupyterlite-xeus and mambajs more performant, and thanks for your contributions that helped a lot in that direction!

Just an FYI, we are working on making use of smart caching logic to speed up kernel startups, I'll keep you posted!

[martenrichter]

@martinRenou Thanks for the information; speeding up kernel startup sounds interesting.
Last weekend, I got libsquashfs to compile inside emscripten. This weekend I am migrating my package to 0.6.1.
I was curious whether loading only necessary packages with conda install would result in a gain in startup time. (Note: I am running a complete notebook at startup for my application.) Surprisingly, it took almost the same time to preload (with many standard packages) or load the necessary package dynamically. (The profiler showed that there was more waiting for kernel startup in the dynamic case).
So I will still try to pursue the squash fs idea..., but it will need a careful balancing of download size and delays.

[martenrichter]

First attempt towards squashfs support in emscripten:
emscripten-core/emscripten#24670

[martenrichter]

Now as external port at:

https://github.com/martenrichter/emscripten_wasm_squashfs
next stop try it on the jupyter lite universe.

[martenrichter]

A draft PR for xeus
jupyterlite/xeus#267

[martenrichter]

A draft PR for empack:
emscripten-forge/empack#127
all at an early stage. Yet to be determined, if it will be worth pursuing it.

[martenrichter]

A draft PR for mambajs:
#129
I am still not at the point to run tests.

[martenrichter]

OK, I have problems with _PyEM_TrampolineCall_Javascript which seems to come from:
https://github.com/python/cpython/blob/7636a66635a0da849cfccd06a52d0a21fb692271/Python/emscripten_trampoline.c#L194
this looks like assembler code. So I would say I do not expect asyncify to work here.
If the python interpreter actually does WASM compiling, then asyncify can not work. JSPI may work, but no webkit support..
So, back to the drawing board, how async fetching can be integrated to the wasmfs call to fetch.

[martenrichter]

Ok, next idea I will disable asyncify and use a worker which uses Atomics to communicate with the main thread and a sharedarraybuffer. (But what happens in insecure webcontexts?).