Espruino plugin for Rollup module bundler

[espruino-discuss]

Posted at 2017-01-11 by @joakim

I've made a plugin for Rollup that lets you push the resulting code straight to Espruino devices.

Rollup is a module bundler, which takes JS modules (preferably ES modules) and bundles them into one file. While doing so, it employs tree-shaking and static analysis to remove unused code, resulting in a file that contains only the bare minimum. Perfect for Espruino :)

It supports plugins, so you can easily add minification of code, linting, support for CoffeScript – and thankfully, support for CommonJS modules, which is the format of Espruino's modules.

Basically, this allows you to structure your Espruino projects using modern ES modules, and have Rollup bundle it all up and send the resulting code to Espruino in one command.

I've only tested it on Puck.js so far, but it's based on code from EspruinoTools' CLI, so hopefully it works for all supported devices. There's also one more feature I'd like to add, the ability to save after sending.

If this sounds useful, try it out and let me know how it works (or doesn't work) :)

https://www.npmjs.com/package/rollup-plugin-espruino

[espruino-discuss]

Posted at 2017-01-11 by @joakim

In order to use Espruino modules (and other CommonJS modules) a few extra steps are required, especially if you also want to uglify/minify the code. This assumes you already have Node, npm and Rollup installed.

First, install the plugins…

npm install rollup-plugin-espruino
npm install rollup-plugin-commonjs
npm install rollup-plugin-uglify

You need to use the "harmony" branch of the uglify-js library for the uglify plugin to support ES modules. This will fetch it from GitHub:

npm install mishoo/UglifyJS2#harmony

Finally, add the following configuration to your Rollup config (rollup.config.js if using the CLI):

import espruino from 'rollup-plugin-espruino'
import commonjs from 'rollup-plugin-commonjs'
import uglify from 'rollup-plugin-uglify'
import { minify } from 'uglify-js'

export default {
  entry: 'src/main.js',
  dest: 'dist/bundle.js',
  format: 'es',
  plugins: [
    commonjs(),
    uglify({}, minify),
    espruino(),
  ],
}

Here I'm passing the "harmony" branch of uglify-js as minify to the uglify plugin.

And with that, it should work as expected. In harmony.

Luckily, other plugins are more straight forward.

[espruino-discuss]

Posted at 2017-01-11 by @joakim

And with that configuration, here's the DS18B20 module's example rewritten in modern JS:

import { connect } from './modules/DS18B20.js'

let ow = new OneWire(A1)
let sensor = connect(ow)

setInterval(() => {
  sensor.getTemp(temp => console.log('Temp is ' + temp + '°C'))
}, 1000)

The same code bundled with the module, uglified and sent to my Espruino after running rollup -c:

function DS18B20(t,e){if(this.bus=t,void 0===e?this.sCode=this.bus.search()[0]:parseInt(e).toString()==e&&e>=0&&e<=126?this.sCode=this.bus.search()[e]:this.sCode=e,!this.sCode)throw new Error("No DS18B20 found");this.type=parseInt(this.sCode[0]+this.sCode[1])}var C={CONVERT_T:68,COPY:72,READ:190,WRITE:78};DS18B20.prototype._r=function(){var t=this.bus;return t.select(this.sCode),t.write(C.READ),t.read(9)},DS18B20.prototype._w=function(t,e,s){var r=this.bus;r.select(this.sCode),r.write([C.WRITE,t,e,s]),r.select(this.sCode),r.write(C.COPY),r.reset()},DS18B20.prototype.setRes=function(t){var e=this._r();t=[31,63,95,127][E.clip(t,9,12)-9],this._w(e[2],e[3],t)},DS18B20.prototype.getRes=function(){return[31,63,95,127].indexOf(this._r()[4])+9},DS18B20.prototype.isPresent=function(){return this.bus.search().indexOf(this.sCode)!==-1},DS18B20.prototype.getTemp=function(t){function e(e){for(var s=e._r(),r=0,i=0;i<8;i++){r^=s[i];for(var o=0;o<8;o++)r=r>>1^140*(1&r)}var n=null;return r==s[8]&&(n=s[0]+(s[1]<<8),32768&n&&(n-=65536),n/=10==e.type?2:16),t&&t(n),n}if(this.bus.select(this.sCode),this.bus.write(C.CONVERT_T,!0),!t)return e(this);var s={9:94,10:188,11:375,12:750};setTimeout(e,s[this.getRes()],this)},DS18B20.prototype.searchAlarm=function(){return this.bus.search(236)},DS18B20.prototype.setAlarm=function(t,e){t--,t<0&&(t+=256),e<0&&(e+=256);var s=this._r();this._w(e,t,s[4])};var connect=function(t,e){return new DS18B20(t,e)};let t=new OneWire(A1),e=connect(t);setInterval(()=>{e.getTemp(t=>console.log("Temp is "+t+"°C"))},1e3);

Not so pretty anymore..

[gfwilliams]

Posted at 2017-01-12 by @gfwilliams

Wow, thanks - this looks great! It might well be able to minify it even further as well (changing the name of DS18B20 for instance).

So is it also able to fetch the modules from the Espruino website as well?

If you want to do save, you might just be able to set Espruino.Config.SAVE_ON_SEND to 1 before sending.

[espruino-discuss]

Posted at 2017-01-12 by @joakim

Yea, I noticed that it could be minified even further. That's up to the Uglify plugin to do, I'll see whether it can be configured to be more ugly.

At the moment you have to manually download the modules, but I think it might be possible to automatically download Espruino modules like the Web IDE does, I'll see what I can do! Would be very nice if this could work just like the Web IDE, that's a good benchmark :)

I don't think setting SAVE_ON_SEND would work, as I'm skipping the part of EspruinoTools that does minification, which is what calls the plugin for saving, if I recall correctly. Can I call that plugin directly somehow?

[gfwilliams]

Posted at 2017-01-12 by @gfwilliams

If you want to skip minification (which shouldn't be on by default anyway?) you can set the config variables for it - you probably shouldn't try and cut bits of the pipeline as there might well be some other stuff in there that's required (like the newline fiddling)...

[espruino-discuss]

Posted at 2017-01-12 by @joakim

You're right of course! I think I skipped the transformForEspruino processor call when I was trying to get things to work, and honestly hadn't really understood how the plugin system works until now. Now that the processor call is back my code follows the same pipeline as sendCode, and it seems to be doing just the same (I'm not calling it directly because I want more control over output).

Some more changes:

• Better options
  • Now supports reset, save, setTime, BLE, audio, baudRate and throttle (all with Espruino's defaults)
  • Any other Espruino config options may be set using their uppercase names directly
  • Added output option for showing the output from Espruino when sending code
• Detects and reports any uncaught JS errors after code has been sent (very useful)
• Prettier CLI with colours and all

I also figured out how to minify it even further, and added a tip for how to configure Uglify to be even uglier on the project page.

[espruino-discuss]

Posted at 2017-01-12 by @joakim

I've looked into loading Espruino modules straight from espruino.com/modules, and while that seems possible, I think It will have to wait untill I actually need is, I have enough on my plate as it is.

If someone else wants to give it a shot, the relevant Rollup hooks appear to be resolveId and load.

[espruino-discuss]

Posted at 2017-01-13 by @joakim

I made a noob mistake when publishing the package to NPM, it probably didn't work for anyone else but me :/ It should be fixed now, hopefully it works for everyone. It does require Node v6 or greater.

[espruino-discuss]

Posted at 2017-02-21 by CriscoCrusader

Couldn't you automate detection of the Espruino so that the port doesn't have to be entered manually? Even just changing to a different USB port will cause this value to change. In the script that I wrote for this, I copied the code that the web IDE uses to identify Espruino boards and connect to the corresponding port:
https://github.com/stokebrain/morra-build/blob/master/main.js#L23

Webpack also has tree-shaking. When I first tried rollup, I concluded it wasn't a good choice because I saw that it was placing top-level variables from all modules, all together in the same scope. I was too concerned about name collision and went back to webpack.

Also, I don't see the smaller footprint that rollup is claiming.
https://vkbansal.me/blog/rollup-first-impressions/

I continually see people writing that rollup configuration is easier, but that has not been true for me. Webpack just worked, and rollup continues to require debugging.

I still have some complaints and issues to solve with webpack. For my first project it all worked well, but in my current MQTT test, I'm getting errors when using my build-and-save script. One thing that concerns me is the use of arrays in the build, because I don't know if these will cause performance issues for Espruino.

[espruino-discuss]

Posted at 2018-05-21 by Wilberforce

Hi @joakim, I'm getting this:

root@office-pc:~/builder/rollup# rollup -c
[!] Error: Cannot find module 'lodash.forEach'
Error: Cannot find module 'lodash.forEach'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:571:15)
    at Function.Module._load (internal/modules/cjs/loader.js:497:25)
    at Module.require (internal/modules/cjs/loader.js:626:17)
    at require (internal/modules/cjs/helpers.js:20:18)
    at Object.<anonymous> (/root/node_modules/rollup-plugin-espruino/dist/index.cjs.js:8:31)
    at Module._compile (internal/modules/cjs/loader.js:678:30)
    at Module._extensions..js (internal/modules/cjs/loader.js:689:10)
    at Object.require.extensions..js (/root/.nvm/versions/node/v10.1.0/lib/node_modules/rollup/bin/rollup:2834:17)
    at Module.load (internal/modules/cjs/loader.js:589:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:528:12)

Any ideas?
Thanks

[espruino-discuss]

Posted at 2018-05-21 by @joakim

Oops, I merged a pull request too quickly.

I haven't released the latest changes on NPM though, so if you fetch the module using npm install rollup-espruino it should work (that is, use v0.2.1).. The updated example configuration in the README should be correct though, while the one in v0.2.1 may no longer work.

I'll be working on this plugin soon, and will iron out any issues then. Hopefully in a few days!

[espruino-discuss]

Posted at 2018-05-21 by Wilberforce

Thanks.

[espruino-discuss]

Posted at 2018-09-15 by opichals

I've looked into loading Espruino modules straight from
espruino.com/modules, and while that seems possible, I think It will
have to wait untill I actually need is, I have enough on my plate as
it is.

If someone else wants to give it a shot, the relevant Rollup hooks
appear to be resolveId and load.

https://github.com/opichals/rollup-plugin-espruino-modules (not published to npm yet).

It is generates a default config for Espruino which currently also does minification by default. It is just a first shot as it needs refinement with regards to configuration possibilities but it works quite nicely. It downloads the modules into the local ./modules folder

I converted the webpack/babel example to use this and babel 7.x here https://github.com/opichals/espruino-rollup-babel-sample

By using the import statements in the app code one can get use of tree shaking so that e.g. the SSD1306 driver's connectSPI is not bundled when not used.

caveat: I haven't tried the resulting bundles on-board yet.

[espruino-discuss]

Posted at 2018-09-15 by @MaBecker

wow greate job, will give it a try in the next weeks

I have decided to include some modules directly into the build like SSD1306 and tinyMqtt see ESP8266_4MB how to.

[espruino-discuss]

Posted at 2018-09-15 by opichals

Using the external tools like rollup and the terser used to minify the bundle in the rollup plugin one can avoid hitting es6 (arrow functions) related minifcation issues for example. Also does the module contents merging mentioned in espruino/EspruinoTools#27.

Although it will require some work I think we should be able to get rollup (and perhaps even the babel functionality if deemed interesting) including the necessary plugins into the EspruinoTools and therefore into the EspruinoIDE. There is no reaction on the EspruinoTools google-closure-compiler-js PR . @gfwilliams what would be your position on that?

@MaBecker I'd rather not make the default firmware images bigger by including stuff people won't use. I'd instead aim for the 'Storage' module assigned flash space expansion for the boards where possible (like the ESP8266_4MB) and use tools like rollup to put stuff in.

[gfwilliams]

Posted at 2018-10-01 by @gfwilliams

That's great - thanks! I must have totally forgotten to keep that up to date.

[espruino-discuss]

Posted at 2018-10-03 by opichals

@gfwilliams There is espruino/EspruinoTools#80 which I'd appreciate getting feedback on. Would that be a way to go to integrate or are other ideas, requirements or comments.

One known issue:

• It looks like the terser config is tool agressive and although I researched the options I didn't find a reasonable way for it to not to strip the onInit() method if not called (having the top_level mangling on at the same time for the best compression ratio).

[espruino-discuss]

Posted at 2018-10-03 by @allObjects

Fool the tool! - works even w/ the smartest one! - What would be a the half-liner doing so? ...like a if (getTime()<0) onInit(); as last item in top-level code?

I hope the win using the tool chain exceed by far the losses caused by above code half-line...

[espruino-discuss]

Posted at 2018-10-04 by opichals

@allObjects I actually do such a thing here , but I do it only in the rollup flow and not for the minification only. Also I don't like it too much as it is too big of a hack IMO (also troublesome to remove from the resulting bundle). Perhaps there is another way to make sure this function is never inlined or renamed by changing the terser configuration .

I have yet to try the babel-minify as that one sounds like the one with the most prospective future.

[espruino-discuss]

Posted at 2018-11-07 by opichals

Made some progress, fixed some stuff in rollup-plugin-commonjs and had other problem corrected in terser minifier and now it looks like it is doing what I would expect.

@gfwilliams The tools PR (https://github.com/espruino/EspruinoTool­s/pull/80 ) is now working in the IDE as well through espruino/EspruinoWebIDE#197.

[gfwilliams]

Posted at 2018-11-09 by @gfwilliams

That's great - thanks! I'm literally only just back from the conference today so am catching up but I'll try and look at this next week.

[espruino-discuss]

Posted at 2018-11-09 by opichals

@gfwilliams :) Were you on nodeconf.eu? People brought 3 badges from there to our office... I should have gone ;)

[gfwilliams]

Posted at 2018-11-09 by @gfwilliams

I was, yes! Where do you work?

[espruino-discuss]

Posted at 2018-11-09 by opichals

:) I work @ apiary.io (acquired by Oracle almost 2 years ago) based in Prague, Czech Republic

[espruino-discuss]

Posted at 2018-11-25 by opichals

@gfwilliams I did some more polishing of the PRs and they are from my point of view ready for pre-merge review.

Everything is off by default and needs to be explicitly turned on in IDE settings or in the cli job file.

[gfwilliams]

Posted at 2018-11-26 by @gfwilliams

That's great! Thanks - sorry for not looking at these yet, I've been a bit busy here.

[espruino-discuss]

Posted at 2020-01-14 by nistvan.86

I'm trying to use the new rollup-plugin-espruino-modules (the old one is not API compatible with Rollup anymore).

import { espruinoModules } from 'rollup-plugin-espruino-modules'
import commonjs from 'rollup-plugin-commonjs'
import { terser as terserPlugin } from "rollup-plugin-terser";

export default {
  input: 'src/main.js',
  output: {
    file: 'dist/bundle.js',
    format: 'es'
  },
  plugins: [
    commonjs(),
    terserPlugin({
      compress: true,
      mangle: true
    }),
    espruinoModules({
      board: 'ESP8266_4MB'
    }),
  ],
}

Running rollup fails with this:

Error: ENOENT: no such file or directory, open 'C:\dev-esp\espruino-cc1101-q7rf\modules\.board_ESP8266_4MB.json'

Mod: okay, never mind, I've figured out. I needed to create the modules directory by hand, otherwise it fails. It's probably a bug.

Am I correct that this new module cannot upload the code to the board after bundling? I've seen the older plugin had the port/baud/save options for this, but not this.

Also is it okay to run the Espruino plugin after the Terser like this?

[espruino-discuss]

Posted at 2020-01-14 by @MaBecker

Check this link to get the needed JSON file

http://www.espruino.com/json/

[espruino-discuss]

Posted at 2020-01-16 by opichals

@nistvan.86 There is a different philosophy behind the new module. It has been created so that I could separate the JS mangling stuff like module resolution, optimizations and minification from the actual code upload to the boards. We exchanged ideas on this with @gfwilliams earlier here.

So the workflow it has been designed in mind with is:

• use the rollup-plugin-espruino-modules to create the final .js bundle
• use the EspruinoTools (espruino npm module) to send the bundle to the board

Or you could use the EspruinoTools (there a config/job variable for that) to run rollup for you with predefined config suitable for using with Espruino. I admit that this needs to get documented better... but you can have a look here.

[espruino-discuss]

Posted at 2020-01-16 by nistvan.86

Thank you for explaining. I'm trying to put the pieces together, and Rollup seems to be the way to go.

I would like to use Typescript on top of allow these, so that's going to twist this a bit further.
(Interesting that I've found a more complete type definition in this repository for the platform; the @types/espruino definition is missing a bunch of stuff. I don't know where the first one came from, would be good to know. Can't reach the repository owner unfortunately.)