Optimization of included JS files

[firai]

I noticed that the size of the NiceGui package, not including dependencies, increased from 13 MB to 49 MB between 1.2.x to 1.3.x. I understand that this is due to #658, which copied in the full JS packages rather than the minified versions. Upon further inspection, the copied packages (at least for mermaid, plotly and aggrid) appear to also contain the minified versions. If the full packages files are being used now, can/should the minified files be removed to eliminate duplication?

My JS knowledge is definitely rudimentary, so I apologize if what I'm suggesting above doesn't make sense given how the packages are set up.

[falkoschindler]

Very interesting observation, @firai!

Honestly, I didn't check which files are provided for every single dependency. Sure, in case we have both versions and use only one, we could remove the other one. But we also wanted to provide source maps for easier debugging. As far as I know, you need the non-minified file for source maps to work. So we ultimately need to provide *.js, *.min.js and *.js.map, growing the package size even more. Or is *.min.js and *.js.map enough?

[firai]

Thanks for your response, @falkoschindler! Bard and Bing seem to tell me that you could still get somewhat readable debugging information without the full package, but I don't really debug JS/TS. @dclause, do you know the answer to @falkoschindler's question? It seems like Mermaid stopped building source maps into the npm/CDN package since 10.2 though.

Looking at Mermaid as an example (since that's the biggest package), it seems to me that 45 versions of the package are actually being shipped with NiceGui right now:

• ESM full (mermaid.esm.mjs and its tree of files)
• ESM minified (mermaid.esm.min.mjs and its tree of files)
• ESM core bundle (mermaid.core.mjs and its tree of files)
• UMD full (mermaid.js)
• UMD minified (mermaid.min.js)

If we need the full packages for debugging and if we have switched to ESM, maybe we can at least get rid of the UMD packages? Ideal scenario would be getting down to just one version.

EDIT: Just thinking out loud, could we somehow point to the CDNs for the full versions for debugging? Might be unlikely that debugging is being done on a machine with no network access? And if that's desired by the user, maybe we can tell the user (or provide a script) to download the full versions as needed, instead of distributing them to everyone?

[falkoschindler]

Super exciting thoughts!

@dclause I think if a user really wants to debug into JS code, the requirement of an internet connection is fine.
And yes, I could imagine to have the JS stack as part of the dev dependencies.
But what about users that want to checkout and use the current main branch? Are they "devs", i.e. would need the JS stack and build the dependencies themselves? This sounds not very convenient. Or should we keep prebuilt JS dependencies in the repo for such a use case?

@firai Even though I couldn't allow all the details, it sounds like a reasonable and very powerful approach for reducing the dependencies to what is actually needed. Do you think this method would be applicable to future versions of Mermaid etc., or will we need to make adjustments to, e.g., the regex when updating Mermaid?

[firai]

@falkoschindler I was originally suggesting to traverse the Mermaid ESM module tree bottom up while only reading the first few lines of each file, but it turns out there are a couple of modules (array and init) that don't point back to the head. The more reliable way is therefore to traverse the tree top down and read the files in their entirety. See working code at https://gist.github.com/firai/4d2c981869a067a12ce08c2e355ffc18 to find all minified ESM files for both Mermaid 10.3.0 and 10.2.4 (the version currently copied into NiceGui). Takes about 10 seconds to find all of the files for each of the builds. The lists seem to be correct as:

• All modules are included.
• Included modules point back to the version of the main mermaid-xxx.js file that mermaid.esm.min.mjs refers to (except for array and init mentioned above, which don't import from the main module for some reason).
• The versions of array and init included are clearly the minified versions when you compare them against the other versions in the folder.

Note that I wrote this code based on Mermaid, since that's the biggest package in NiceGui right now, but it may work for other packages as well as long as the right root directories and entry points are chosen. As long as Mermaid uses the current directory structure (all of their js files are in the dist root directory), the code should work. If they change the directory structure, the code probably still works, as long as they prepend the imported paths with ./, otherwise minor adjustments to the regex may be required. To foolproof and future-proof this code, we could use urllib.parse to resolve the relative urls in the imports.

[falkoschindler]

@firai This sounds very promising. Would you like to create a PR to provide optimized Mermaid dependencies as a proof-of-concept? Or should we tackle all dependencies at once?

[firai]

Is there a conclusion about how you want to deal with source maps for packages that don't provide them as build artifacts, such as Mermaid? Can we leave it as an upstream issue and not provide source maps if upstream doesn't provide them? Or do you want to minify the scripts ourselves?
(By the way, webpack is a bit beyond my realm of knowledge. I can try to take a stab at implementing the optimization for Mermaid without providing source maps, but someone may have to take this over if source maps need to be provided for all packages in this project.)

[falkoschindler]

Let's first focus on removing unneeded files. We can tackle missing source maps in a separate, later PR. This approach avoids cluttering a single PR and eases the review process.