Minifying modules with ADVANCED_OPTIMIZATIONS

[espruino-discuss3]

Posted at 2016-02-28 by luwar

While minifying modules currently the Google closure compiler will be used with SIMPLE optimization (=default without the @compilation_level ADVANCED_OPTIMIZATIONS annotation).

Moray gave an example of using advance optimization while he was refactoring the BME module. → http://forum.espruino.com/conversations/277165/

There is still a little imperfection already stated: The basic rule is refer to anything you don't want minified as an indirect property reference, so me.property should be written as me['property'].
This blights the code a little bit and can be avoided be giving the closure compiler the knowledge which code you want to keep. Moray already mentioned the "extern" approach http://forum.espruino.com/comments/12636552/. A simple usage scenario (without parameters) would be enough. Here is an example of the HTU21D module:

var htu = exports.connect( null );
var t1 = htu.RH_12_BITS_TEMP_14_BITS;
var t2 = htu.RH_8_BITS_TEMP_12_BITS;
var t3 = htu.RH_10_BITS_TEMP_13_BITS;
var t4 = htu.RH_11_BITS_TEMP_11_BITS;
htu.isMeasuring();
htu.readTemperature();
htu.getTemperature( null );
htu.readHumidity();
htu.getHumidity();
htu.getCompensatedHumidity( null, null );
htu.getDewPoint( null, null );
htu.softReset();
htu.getResolution();
htu.setResolution( null );
htu.isEndOfBattery();
htu.isHeaterOn();
htu.setHeaterOn( null );
htu.getSerialNumber();

Now you can leave all the module code and advanced optimization works fine. But from where should we take this usage code for every module?

1. Write it manually
2. Extract it from the code annotations
   Google itself has an @export notation but then you have to use the Google Closure Library → not applicable.
   Parse/Grep would be possible but it's a lot of work until reaching a stable version for all possible cases.
3. Using a precompiler like TypeScript. Type information is available and it's seems Google is enhancing the minifier with some TypeScript support. For me this is the most promising option because I write modules in TypeScript and the advantage of using TypeScript at all is huge. But I think many module developers do not share this opinion - up to now. We'll see us in one year. ;-) But there is currently no out-of-the-box solution.

Currently I prefer 1) because it's straightforward and easy realizable and additionally leaves option 2) and 3) open.

Later it would be possible to enable advanced optimization in the IDE. This would be really great to remove all unused code during save().

What do you think about adding a simple extern/usage file for every module and compile the modules with advanced optimization? This would not be a huge one-step-solution. Every module can decide by itself:

• No extern file available → simple optimization
• extern file available → advanced optimization

A few numbers for the motivation behind:

• HTU21D.js: 14391 bytes, simple minified: 3802 bytes, advanced minified: 2400 bytes
• MLX90614.js: 4867 bytes, simple minified: 1714 bytes, advanced minified: 1166 bytes
• MAG3110.js: 8219 bytes, simple minified: 2547 bytes, advanced minified: 1542 bytes
• MCP9808.js: 12928 bytes, simple minified: 3745 bytes, advanced minified: 2427 bytes

Code sizes shrink about 30% to 40% when switching from simple to advanced optimization.

BTW: The Google webservice is slow with ADVANCED_OPTIMIZATIONS and denied after using too often. I wrote a local build script. It uses exactly the same directory/file convention so no customization is required. It's faster and no additional installation except Java is required (Google Closure Compiler is a Java program). I would make it available if somebody is interested...

And that could be the roadmap:

• Write or generate an espruino_extern.js file, still mentioned http://forum.espruino.com/comments/12636552/
• Create a file XXX_extern.jsfile for every module step-by-step

Any thoughts?

[espruino-discuss3]

Posted at 2016-02-28 by @allObjects

@luwar, appreciate your investigative work. I would go for an xxx_extern... or xxx.exj or alike fie for each xxx.js module file. As you point out, it provides what is needed with the least disruption / change, most freedom AND options.

---

Posted at 2016-02-29 by DrAzzy

I remember having lots of trouble making anything minified with advanced minification actually run - Have you checked that the code you're having it minify actually runs after going through advanced minification?

---

Posted at 2016-02-29 by @gfwilliams

Looks like you're getting some good compression there - it seems like a lot of work for each module though, unless it can somehow be done automatically.

I'd imagine that your example of using var htu = exports.connect( null ); may cause some problems though. Imagine the module is actually:

exports.connect = function(options) {
  if (!options) throw new Error("Needs options!");
  // ...
}

I think advanced optimisations could actually optimise everything out of the module, because it knows how you're using it. If it doesn't do it now, it may well do when it gets improved a bit more.

Also, when the modules themselves are minified, they get wrapped with (function() { ... })(), which allows the compiler to rename any non-exported functions/variables, even with simple optimisations.

Is that what you're doing for your figures here? Last I checked, the difference in size was nowhere near as big as you're getting - there does seem to be a difference between the modules at http://www.espruino.com/modules/ and the figures you have...

I think the biggest 'win' at the moment would be to 'bundle' the whole program up into one JS file (modules and all) in the Web IDE and send that to be minified. Hopefully then it could automatically rename absolutely everything that is possible, and could even remove unused functions (which is probably the biggest source of wasted space at the moment).

I'm not quite sure how that'd be done, but it might be that just defining require as function require(x) { return modules[x]; } would be enough - the closure compiler is pretty smart.

(Of course the Web IDE has an offline compiler built in - UglifyJS IIRC - and if that would do similar tricks as well it'd be much faster)

---

Posted at 2016-02-29 by luwar

@drazzy

I remember having lots of trouble making anything minified with advanced minification actually run - Have you checked that the code you're having it minify actually runs after going through advanced minification?

Yes, the minified code is working. But you are right. We should not enable advanced optimization blindly. Every module has to be examined and tested separately. This can be done step-by-step.

What kind of trouble do you mean? Did you use an extern/usage file? First I tried to avoid this additional file and - yes - also with JSDoc and Google annotation it never worked in all cases.

---

Posted at 2016-02-29 by luwar

@gfwilliams

I'd imagine that your example of using var htu = exports.connect( null ); may cause some problems though. ...

Extern/usage files are handled differently by the optimizer. It's enough to reference the functions. First I thought a usage scenario is better readable but it's not.

htu.isMeasuring;
htu.readTemperature;
htu.getTemperature;
htu.readHumidity;
htu.getHumidity;
htu.getCompensatedHumidity;
htu.getDewPoint;
htu.softReset;
htu.getResolution;
htu.setResolution;
htu.isEndOfBattery;
htu.isHeaterOn;
htu.setHeaterOn;
htu.getSerialNumer;

Also, when the modules themselves are minified, they get wrapped with (function() { ... })(), which allows the compiler to rename any non-exported functions/variables, even with simple optimisations.

I saw the wrapping in the build script. It's not nessecarry in advance mode anymore. It's even counterproductive. After wrapping, minifying and unwrapping the HTU21D module is a few bytes larger (in advanced mode). I will investigate it when the next steps are clear.

Is that what you're doing for your figures here? Last I checked, the difference in size was nowhere near as big as you're getting - there does seem to be a difference between the modules at espruino.com/modules/ and the figures you have...

I didn't try to cheat. I found that the difference between the current minified files on the website and my numbers in the posting was in the compiler settings and the (sometimes) missing wrapping in my build settings.
And I tricked myself: The minified version is even smaller, e.g. MCP9808:

• Earlier posting: simple minified: 3745 bytes
• Earlier posting: advanced minified: 2427 bytes
• Website: 3429 bytes
• Now minified with advanced optimization: 1994 bytes (> 40% smaller)

I'm not quite sure how that'd be done, but it might be that just defining require as function require(x) { return modules[x]; } would be enough - the closure compiler is pretty smart.

I will try and report back. But it will take a few days. And without ensuring that every used module is "advanced optimization"-safe we cannot enabled advance optimization for whole programs.

What should be the next step? Perhaps selecting one module as an example, specify the exact steps, writing the extern file, customizing the build script, ...?

---

Posted at 2016-02-29 by @gfwilliams

Those stats are interesting - if MCP9808 still works fine, that's a really big saving. I'm just worried about the extra work involved in maintaining all the modules and their extern files. Usually contributed code needs a bit of fiddling to get the documentation/module in the right format, and it's just going to be even more difficult with something like that.

"advanced optimization"-safe

Have you come across anything that just doesn't work with advanced optimisation? If it causes an error in Espruino and you can figure out what bit of syntax causes it, it'd be worth filing a bug for.

What should be the next step?

Well, if you're interested in getting the finished program as small as possible, modifying the Web IDE (the EspruinoTools command-line would be the best start) to do whole program minification would be absolutely amazing.

The problem with changing the minification script used to minify the modules is I don't honestly know who is going to have the time to test all the modules - and if there's all that extra complexity and then only 5 modules end up getting advanced minification set up, I'm not sure if it's worth it.

---

Posted at 2016-02-29 by luwar

Ok, that's a plan. I share your concern about the extra complexity and the missing acceptance while developing a module. There is still a switch in the build script for @compilation_level ADVANCED_OPTIMIZATIONS. A module developer has to rewrite his whole module code with me['property'] syntax and therefore only one module is using it. This special case could be removed with the new approach.
It will be really easy to add a small extern file with round about 10 lines and don't change the code at all. The modification to the build script will be minimal - I'll try to prove ;-)

---

Posted at 2016-02-29 by @allObjects

I see the extra effort that has to go in - especially the testing effort, because it needs a setup of not just a basic board but also the peripheral components and all wiring. Initially, I wanted to mention a flag in the external file that would enable/disable/control the advanced minification. An option could be to start out with this flag set to false to use the current minification. On need - when running out of memory on use - individual contributions of the community into the external file could over time lead to the desired coverage of the modules with enhance minification. The enhanced file may even have a different name or extension in order to leave the existing process along. Having though an enhanced option integrated into the processing, crates the base for growing 'this extra leg'.