using modules in modules?

[espruino-discuss3]

Posted at 2016-08-11 by dwallersv

Can other modules be used in a module with the require() statement? The writing your own modules page addresses this to some degree, but it wasn't entirely clear to me what would happen if, for instance, the same "core" module is referenced in several different modules.

The scenario: I want to break up the GUI components I've been working on into separate modules so that only the elements being used can be individually referenced, saving space. So, I want to break the code up into a Button module, Slider module, Checkbox, Textbox, etc. All need a "core" set of functionality that would be included in each module.

So, if you were to create an interface that used buttons and sliders, you would include just these components in your code like this:

require("Button");
require("Slider");

Each of those have in them,

require("GUIcommon");

which, of course, is then getting called twice.

Is this handled gracefully, the "right" way so there isn't duplication?

[espruino-discuss3]

Posted at 2016-08-11 by ClearMemory041063

The WIFI modules use the AT module.

---

Posted at 2016-08-11 by @allObjects

...it all depends...

On uploading: no, it's uploaded only once...

On use, it may be different matter: what are your expectations? ...and what's the implementation of the module (Does it create something else than just executable code (and shared (config) object)?

Inverting the require tree could simplify your work... Common is requested first. Each Specific registers with the Common. Then you ask the Common part to create a Specific for you, like a factory.

More to come...

---

Posted at 2016-08-12 by dwallersv

Thanks for the feedback, guys.

Ran into problems. I have successfully split the GUI code into 4 separate modules: Widget, Button, CheckBox, and Slider. Widget must always be included (require("Widget")), and must come first.

I clearly didn't understand scoping and namespace particulars, however. Simply cutting up the source and adding exports lines didn't do the trick. The scope the module is loaded (and run) in is not the same scope as the main console prompt. After looking at other modules, it looks like I have to call a function exported and set everything up there.

So, as a first-pass, I just took the entire "class" for one of these objects, and wrapped it in an init() function that's exported. Constructor function, prototype properties and functions, etc. -- literally indented the entire thing and put "exports.init = function() { ... }" around it. Then, I use it all this way in my demo code:

require("Widget").init();
require("Button").init();
require("CheckBox").init();
require("Slider").init();

var w;
var sld;
function start() {
	LCD.clear();
	new Button({
		style: 'flat',
		area: [58,5,100,50],
		text: 'LED1 on',
		color: new Uint8Array([1, 0, 0, 102]),
		border: 5,
		textColor: new Uint8Array([0,200,200,200]),
		onPress: function() {
			LED1.write(!LED1.read());
			this.text = LED1.read() ? "LED1 off" : "LED1 on";
		},
	});
	new Button({
		area: [163,5,100,50],
		text: 'LED2 on',
		color: new Uint8Array([1, 0, 128, 153]),
		onPress: function() {
			LED2.write(!LED2.read());
			this.text = LED2.read() ? "LED2 off" : "LED2 on";
		},
	});
	new CheckBox({
		area: [70,70,150,30],
		text: 'Backlight',
		textColor: new Uint8Array([0,255,255,0])
	});
	sld = new Slider(
	{
		area: [10,189,300,50],
		min: 0,
		max: 24,
		value: 0,
		range: [0, 1],
		showSlide: false,
		onPress: function() {
			SPI2.send8bit(Uint8Array(this.pixels.buffer, (this.value>>>0)*3, 72), 0b00000011, 0b00011111);
		},
		onHold: function() {
			SPI2.send8bit(Uint8Array(this.pixels.buffer, (this.value>>>0)*3, 72), 0b00000011, 0b00011111);
		},
		draw: function() {
			strips = Math.ceil(this.area[2] / 8);
			for (i = 0; i < strips; i++) {
				hue = i * (this.range[1] - this.range[0]) / strips + this.range[0];
				LCD.setColor2(Uint8Array.prototype.HSBtoRGB([1, hue * 255, 255, 255]));
				LCD.fillRect(
					this.area[0] + i * 8,
					this.area[1],
					this.area[0] + (i + 1) * 8 - 1,
					this.area[1] + this.area[3]
				);
			}
			LCD.setColor2(this.borderColor);
			LCD.drawRect(
				this.area[0],
				this.area[1],
				this.area[0] + this.area[2],
				this.area[1] + this.area[3]
			);
		}
	});
	sld.pixels = new Uint8Array(144);
	for (i=0; i<48; i++) {
		c = (new Uint8Array([1, (i%24)*255/24, 255, 25])).HSBtoRGB();
		sld.pixels[i*3] = c[2];
		sld.pixels[i*3+1] = c[1];
		sld.pixels[i*3+2] = c[3];
	}
	SPI2.setup({baud:6400000, mosi:B15});
	
	new Slider({
		color: new Uint8Array([0,100,100,100]),
		slideColor: new Uint8Array([0,200,200,200]),
		area: [60,120,200,25],
		min: 0,
		max: 1,
		value: LCD.brt,
		onPress: function() { LCD.setBrt(this.value); },
		onHold: function() { LCD.setBrt(this.value); }
	});
}

Not sure if this is the right way to do this (I'm really trying to do what #include does in C). What I found is that it seems to consume twice the memory space just getting the stuff into the system than if the code in those require statements is just literally there rather than being loaded through the module system. This is before even executing the start() function from the demo code, which creates the objects.

I don't get it.

---

Posted at 2016-08-12 by @allObjects

I did not walk through the code yet. But somehow I feel that the elements act too individually and independent. I did not encounter the notion of focus. In a UI all elements interact somehow with each other... like the view (UI) elements are connected through the controller with each other and the model in the MVC concept. May be this interaction is there but I didn't notice.

The other challenge is a typical javascript challenge - but it is only a challenge when thinking C/C++ (or any other) - so called - compiled language... Sorry - and not sorry at all - to repeat myself and pointing to some 3 cents contributed in a different conversation: simple explanation how to save code that espruino run on start?.

What trips most non-Javascript-natives off is the fact that when the code is uploaded, 'things' already happen. I was used to Smalltalk before being 'brain-washed' with Javascript. What is kind of common in both of these languages, that after the code is loaded - there exists a cluster of objects (instances and class instances, so to speak). In Smalltalk, this is called an image, and when something new is defined, for example, a class, it is added to that image of objects. Similar to key words in Javascript, of which one the function f() is a very important one, especially when used as a constructor / 'class' object. The concept of an image - memory with alive objects - is the same in Espruino. On save() Espruino saves all - variables/state AND code/class/function definitions in its current state in RAM - as an image - into the Flash EEPROM. When powered on again, RAM is loaded by copying this image back from EEPROM and execution is kicked of by Espruino invoking onInit() and all ```E.on("init",function(){})-registered functions.

Btw, above aspects made me chose the DOM approach in my UI setup to follow the Browsers' great implementation choice. Last but not least is the Browser the world Javascript was born into:

While HTML(5) is streaming into the Browser, the browser interprets the incoming tags... and so also the script tag. All what follows this tag is directly passed on to the Browser's Javascript engine and interpreted(note1) by it: if it is in top level / level 0 - it is executed directly, such as var v = 5; and var d = new Date(); and require("module"); and - 'worse' - require("module").connect(); (or what ever method/function is invoked on the required module object - static or dynamic). The same happens with function f() { ... }, which creates a function object which is reference-able with fin the object space (syntactically equal with: var f = function() { .... };. The same happens when code is uploaded to - streaming into - Espruino(note2): top/level code is executed immediately(note3).

You mention space / memory concerns... a reason that I separated data from code and chose the cheapo-oo approach for my UI implementation: data is in simple and fast arrays and code are methods on the UI singleton. Your perfect oo representation and legibility just costs a bit too much in memory because of the amount of source code that is required to describe each element. Agreed, it allows you to have a nice omitting and defaulting of values when a property is not provided in the constructor parameter object. On the other hand, practically every time the core attributes show always and create a lot of property name repetition. Yes, the methods can easily be attached to the 'class' with prototype, but that's about it. Similar challenge is faced by the color info... Even though I like very much the freedom, most UI's (have to) use a color palette and a single integer can pick the color. When it comes to MCs, frugality is of the essence - I hated to go down this route and [leave the path of oo-virtue by going for parm list and storage in arrays - but it paid off in terseness without compromising ease of use. I could go even further, and make the UI elements definitions in top-level/level-0 in nested array and thus save even more space...

I could think of your UI implementation to be used as a generator for condensed code to be executed on a runtime implementation (GWT uses such an approach, where stuff is even developed in a different language - Java - and cross-compiled/generated into Javascript and paired with a runtime piece).

I hope some of this brain-dump is useful to you in moving your stuff forward.

(note#)s in next post...

---

Posted at 2016-08-12 by @allObjects

////////////////////////////

(note1): Browser javascript engines just-in-time compile the code block first before executing it, where as Espruino - because running of source code - immediately - 'streaming-ly' - interprets the code... you can see the difference in the way the the browser throws a syntax error before even doing something, where as Espruino already may have executed parts of the code when noticing that something is - syntactically - not adding up (@gfwilliams correct me if I'm wrong or leaning too much out of the window...).

(note2): Before a piece of code is uploaded to Espruino, it is literally grep-parsed for require("moduleName") and referenced module is searched and sent to Espruino with the directly executed Modules.addCached(moduleName,moduleSource);see Reference - which you btw can execute in the Web IDE's console with the same effect... and even for the module source this parsing and Modules.addCached(); is invoked. So when it the comes to the immediate or delayed execution of require("moduleName");, the 'code is found' in the Modules' singleton's cache... Now - afaiu - not the source code is stored in the cache, but the interpreted source code's return result... which means that local things need space only for the process to prepare for caching are then garbage collected if not referenced anymore (globally somehow or by closure).

note3: This is the reason that upload should only contain a minimum of actual code execution. Therefore, it is better to stick ALL execution stuff into onInit(){ ... } and E.on("init",function(){ ... })-registered functions - even setting of pin mode, timeouts, and intervals. Of there is too much executed immediately beyond code (function) definitions, the upload may not have come to the code that in the timeout or callback will be called... (oooops... happened and created great confusion...).

---

Posted at 2016-08-12 by dwallersv

Thanks very much for the detailed discussion! I'm a geek about this sort of stuff, so the more understanding the better :-)

In the end, the problem can be summed up pretty simply: Scope.

Rereading the primer on making modules I think I found the solution -- the section declaring public members. I didn't do that. Giving that a try in a bit...

Also, ran across this again (I need to get kicked in the head twice sometimes to notice I've been kicked at all :-)):

Can I dynamically load (and unload) modules?

Yes. By default each module that is loaded will be cached (to avoid loading modules >twice). However you can call Modules.removeCached('modulename') which will remove >the module from the cache and free the memory that it uses.

I think this is what's doubling the space usage over just loaded the same code directly from the IDE. IOW, there is a cached copy of the code, and another copy in the working space of the interpreter.

I'll try using Modules.removeCached() in the main demo code to see what happens, but I suspect it will make the loaded code non-functional (as if it wasn't there). @gfwilliams?

In the end, I may just have to structure this as a set of code files that must be literally appended to the application source in order to keep from wasting the very precious RAM on the F1, if code duplication because of the Module mechanism wastes all that space.

It's ironic, because the Module caching mechanism seems intent on SAVING rather than WASTING space, and for the problem its intending to solve, it does. However, looks like there's an unintended side-effect of wasted space when there isn't the circumstance of loading the same module more than once.

---

Posted at 2016-08-12 by @allObjects

Regarding modules, the upload for every piece of code happens in two steps:

1. parse the code (or module) to upload for every instance of require("noduleName");. for each found module do 1.

2. upload - code or module - for which the (nested) required modules have just been loaded.

Note: you can 'escape' step one by using a variable... But in order that the code has what it needs to run successfully, the module has to be uploaded separately.

var moduleName = "myModule";
require(moduleName);

Above code will fail with already on upload because it is all in top-level/level 0 and interpreted when arriving at Espruino with this message:

 1v86 Copyright 2016 G.Williams
>echo(0);
ERROR: SD card must be setup with E.connectSDCard first
WARNING: Module "myModule" not found
=undefined
>

You notice two things:

1. ERROR: SD card...
2. WARNING: Module "myModule" not found

Bottom line the 2nd item is what matters. Interestingly though is that when you have an SD memory card connected and mounted and "myModule.js" or "myModule.min.js" (minified version of "myModule.js") on it, the module is found on the SD memory card, loaded and stored evaluated in the cache.

You can actually ask Espruino which modules are cached. Enter in the console Modules.getCached() (command, reference):

>Modules.getCached();

What you would expect is (according to reference documentation):

=[
 ]
>

but you get:

=[
  "myModule"
 ]
>

You will get what you expected with 'empty' Espruino (after you enter command reset(). But for what ever reason, @gfwilliams decided to make the (map) entry anyway on require() but put the value undefined... and unfortunately there is NO (easy and) quiet (programmatic) way to figure out if the module is actually there, successfully loaded... NOT even with a a try catch block (Upload below code - copy-pasted into editor - and enter command onInit():

var moduleName = "myModule";
var myModule = "The module was not found";
function onInit() {
  try {
      myModule = require(moduleName);
  } catch(x) {
      console.log("Programmatically caught exception: " + x);
  }
  var namesOfCachedModules = Modules.getCached();
  console.log("Cached modules:");
  console.log(namesOfCachedModules);
  if (typeof myModule === "string") {
    console.log("myModule = " + myModule);
  } else {
    console.log("myModule was set to " + myModule);
  }
}

@gfwilliams, can that be fixed? ...backward compatible? Not successfully loaded modules should not show in the list or there should be an easy way to get to that info about the module - without Espruino throwing a fit / writing to console (and may be mess up if console not connected...) - or, worst case, with cumbersome try-catch-block that actually works... Try-catch is obviously not working as can be seen by the output produced (or not produces as expected by the catch-block), but sets the variable myModule.

 1v86 Copyright 2016 G.Williams
>echo(0);
=undefined
>onInit()
ERROR: SD card must be setup with E.connectSDCard first
WARNING: Module "myModule" not found
Cached modules:
[
  "myModule"
 ]
=undefined
>

(Since Modules is natively implemented, there is no access to the 'map' that could just return null for not uploaded modules and not-null/module object for successfully uploaded modules.)

Back to hat I wanted to continue with (Sorry for this (even me) surprising detour...): Enter following commands and you get empty list of cached modules.

reset();

Modules.getCached();

Now we 'hand'-fix the issue by entering the issue by entering the following in the console (copy-paste):

Modules.addCached(
    "myModule"
  , "exports = \"a simple String value: 'Hi!'\";");

Then we enter in console (copy-paste):

Modules.getCached();
var myModule = require("myModule");
console.log("myModules is: " + myModule);

And the expected output is:

>Modules.getCached();
=[
  "myModule"
 ]
="a simple String value: 'Hi!'"
>console.log("myModules is: " + myModule);
myModules is: a simple String value: 'Hi!'
=undefined
>

We could not directly upload and execute previous code, because first thing of upload is reset() which just wipes out our 'hand'-fix. But we can add the 'hand'-fix to the code as first thing and then proceed as before.

Modules.addCached(
    "myModule"
  , "exports = \"a simple String value: 'Hi!'\";");
var moduleName = "myModule";
var myModule = "The module was not found";
function onInit() {
  try {
      myModule = require(moduleName);
  } catch(x) {
      console.log("Programmatically caught exception: " + x);
  }
  var namesOfCachedModules = Modules.getCached();
  console.log("Cached modules:");
  console.log(namesOfCachedModules);
  if (typeof myModule === "string") {
    console.log("myModule = " + myModule);
  } else {
    console.log("myModule was set to " + myModule);
  }
}

And here the expected output from output and entering onInit() command:

 1v86 Copyright 2016 G.Williams
>echo(0);
=undefined
>onInit();
Cached modules:
[
  "myModule"
 ]
myModule = a simple String value: 'Hi!'
=undefined
> 

I guess this should put to peace many of the questions in regard of modules.

...another 2 cents from my part.

PS: Read through related Struggling with modules conversation.

---

Posted at 2016-08-15 by @gfwilliams

Just got back from a holiday and I have a million e-mails and posts to get through, so sorry for not reading this fully, but this might help:

• Modules can be used from another module
• Modules are only 'loaded in' once, even if used multiple times. If you put console.log in the main scope of a module, it will only be called once. But it'll be called right at the start, not when you first do require.
• If your module is var A = {}; exports.A = A;, every time you require that module you will get access to the same version of A
• The order of loading should be correct (i think) - that is, if A requires B, B is loaded before A.
• Modules are executed in their own scope (called module IIRC). If you want to stick stuff in the global scope I think you can still use global, but that's probably frowned upon.