Proposed, and implemented, some enhancements (1Mb + 1Mb firmware, resizable jsvars size, ...)

[espruino-discuss]

Posted at 2017-02-25 by Pedro

Hi all,

This looks like a great piece of work, and I am migrating some personal project to Espruino with my ESP cards.

For everything to work fine, I needed some enhancements, so I have done them (on v91), and I think it woud be worth sharing so that these can be merged into the public version:

• Resizable vars : the code is there, for the Linux version. Just patched a bit to enable it in ESP (should also work for other cards now) and fixed also some minor bugs on the save to / load from flash code (as it was, I think it would also fail on Linux, as with RESIZABLE_JSVARS the vars are stored in blocks, while the compress functions, as they were, read and wrote a full contiguous block of memory)

• 1024Kb + 1024Kb memory map : Added few lines to the Makefile to create a new target for the ESP 4Mbyte cards. With this layout, now we have 1024KB for firmware and stored vars and functions. Still neeeded to check where to fix some address mapping for the second image, as in this mode the bootloader maps 0x4020000 to either the first or second Mb of flash, and the image itself is exactly identical for user1 and user2. Just need to check in existing code if there is something harcoded around flash address and fix it.

So now, I get:

reset();
=undefined
... [ Espruino boot banner ] ...
Flash map 4MB:1024/1024, manuf 0xe0 chip 0x4016
Flash code area: 0x76000 (565248 bytes)

process.memory();
={ "free": 106, "usage": 22, "total": 128, "history": 0 }

require("ESP8266").getState();
={
"sdkVersion": "2.0.0(5a875ba)",
"cpuFrequency": 160, "freeHeap": 34232, "maxCon": 10,
"flashMap": "4MB:1024/1024",
"flashKB": 4096,
"flashChip": "0xe0 0x4016"
}

var tester = [];
=[ ]

for(var i=0 ; i < 900 ; i++) tester[i] = i;
=undefined

process.memory();
={ "free": 88, "usage": 936, "total": 1024, "history": 6 }

var more = [];
=[ ]

for(var i=0 ; i < 900 ; i++) more[i] = 2*i;
=undefined

process.memory();
={ "free": 210, "usage": 1838, "total": 2048, "history": 6 }

require("ESP8266").getState();
={
"sdkVersion": "2.0.0(5a875ba)",
"cpuFrequency": 160, "freeHeap": 3344, "maxCon": 10,
"flashMap": "4MB:1024/1024",
"flashKB": 4096,
"flashChip": "0xe0 0x4016"
}

require("ESP8266").reboot();
=undefined
... [ ESP boot info and some garbage :) ] ...
Loading 2048 jsvars from flash...

dump();
var tester = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,;
var i = 900;
var more = [ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, ;
pinMode(D0, "input_pullup", true);
pinMode(D4, "input_pullup", true);
pinMode(D5, "input_pullup", true);
pinMode(D12, "input_pullup", true);
pinMode(D13, "input_pullup", true);
pinMode(D14, "input_pullup", true);
pinMode(D15, "input_pullup", true);
=undefined

TO DO:

• More testing
• Take benefit of this extra memory (make sure we can access code directly on it)

For the second part, what I use is as small piece of code based on something I found a couple of years ago in Internet and which I enhanced bit, so that ESP can access directly the flash mapped memory independently of alignment, or size (1, 2, 4 bytes) with no further modification to the user code. This is possible because ESP8266 is a bit of a monster, and allows virtualizing memory access through HW catchable exceptions for this kind of memory access errors, so a trap function can directly see the failing opcode and target address, access flash with the right alignment, simulate the opcode and put the result on the right register, and return control to the program (at assembler opcode level) after the offending opcode address.

This is seamless to your code, that can now access memory mapped flash as if it was in the rodata area. Of course, there is an impact in speed, which I have not benchmarked, but not noticed either, as we are not using our ESP8266s to do math intensive calculations (are we?)

Will work on other improvements.

So, if there is interest on it, I have a bunch of ".patch" files I will be happy to submit, or share, or whatever is the best way to get these into the main build.

[espruino-discuss]

Posted at 2017-02-25 by @MaBecker

Hi @pedro, looks promising !

Building firmware for ESPs has a lot of compatible setting to make sure it runs for all models.

There was a switch to a single board build

So why not switch to a two board build: 512K and 4MB

Would like to jump in for testing, please share.

---

Posted at 2017-02-26 by Wilberforce

Hi @pedro
The way of patching this stuff is to clone https://github.com/espruino/Espruino into your own name.
Then you can make a branch, say called 4MB, and then do your updates into the branch.

Then when you are ready, you do a pull request to the /espruino/Espruino master.

I think you might need to reserve some heap space for the wifi, as it looks like you don't have much heap space left after the vars increase. The buffer for IP are made dynamically and a free heap of 3344 might not be enough.

---

Posted at 2017-02-26 by Pedro

Thanks @MaBecker, I am following @wilberforce instructions on to create a branch so that the code cen be checked, and once I get feedback and more testing I will do a pullt request. Need to investigate how to do it : not a git master :) even if I remember doing something similar to patch Linux kernel in the past.
For the two board switch, the main issue is that platform_config.h and some ifdefs in the code must behave differently (for the RESIZABLE_JSVARS part), so I did something like the past two board switch, but the whole thing needs to be recompiled (not just relinked to get the different firmwares), so it is somehow different. Will try to submit today so that you can check it.

---

Posted at 2017-02-26 by @MaBecker

After reading through the source code, I think there is no need to have a new board for ESP, because with make option FLASH_4MB=1 it should be possible to handle all stuff.

FLASH_4MB=1

build "flashMap": "4MB:1024/1024" 

not sure about RESIZABLE_JSVARS

---

Posted at 2017-02-26 by Pedro

Thanks @wilberforce, will try to branch the code so that it can be reviewed.
Regarding your comment on free heap, I just pushed it to the limit (2048 jsvars dynamically allocated). I agree at some point I need to put some check to keep a minimum heap available, failing to create a new jsvar block, but the result should be the same : memory full / bad things happen :)
Currently new jsvars are allocated in blocks of 128, so it does not go directly from 31KB heap to 3KB heap.

---

Posted at 2017-02-26 by Wilberforce

Ok, sounds good.

32k heap sounds like a bit more than usual - was that related to your changes?

---

Posted at 2017-02-26 by @MaBecker

@wilberforce normal is 8k heap and 1700 vars

---

Posted at 2017-02-26 by Pedro

You are right. For the 4MB map we do not need a new board. For RESIZABLE_JSVARS, as it is behaving differenty as the standard ESP8266 current version, I wanted to make sure not to mess with it and create a new board. To avoid "forking" then new board just includes the standard ESP8266_BOARD.py and changes a few parameters. If there is a cleaner way, will be happy to do it.

---

Posted at 2017-02-26 by Wilberforce

Ahh - I see why now - 128 vars at startup, that's why so much heap is available.

---

Posted at 2017-02-26 by Pedro

Yes it is. Initially only 128 jsvars are allocated, and it grows as needed.

---

Posted at 2017-02-26 by @MaBecker

@pedro can you please briefly explain how to switch to "flashMap": "4MB:1024/1024"

---

Posted at 2017-02-26 by @MaBecker

found a lot of information on this espressif page in chapter 4 and 5

---

Posted at 2017-02-26 by Pedro

In the Makefile:

ESP_FLASH_SIZE ?= 6
ET_FS ?= 32m-c1

And in user_main.c, change (to avoid a warning):

if ((chip == 0x4013 && map != 0) || (chip == 0x4016 && map != 4)) {

to

if ((chip == 0x4013 && map != 0) || (chip == 0x4016 && map != 4 && map != 6)) {

As soon as I have created the branch I will submit the changes for review (including this in the makefile with the proper conditionals)

In this mode, user1 and user2 bin files are identical (no different ld files) and the bootloader maps the first or second MB of flash to 0x40200000. As commented before, I need to check in the existing code that there is nothing hardcoded in current espruino build for ESP8266 when referencing flash from the first or second image

---

Posted at 2017-02-26 by Pedro

@MaBecker, @wilberforce, how do I create a new branch ? (or do I just fork into my repository). As commented, not a expert github user (mostly a newby here)

---

Posted at 2017-02-26 by @MaBecker

These are the hard coded sections I know so far:

https://github.com/espruino/Espruino/blob/master/boards/ESP8266_BOARD.py#L46

https://github.com/espruino/Espruino/blob/master/libs/network/esp8266/jswrap_esp8266_network.c#L1185

not sure about https://github.com/espruino/Espruino/blob/master/libs/network/esp8266/ota.c

---

Posted at 2017-02-26 by @MaBecker

• Create yourself a github account

• goto https://github.com/espruino/Espruino

• and click on Fork

That's it :)

---

Attachments:

• Bildschirmfoto 2017-02-26 um 11.43.02.JPG

---

Posted at 2017-02-26 by @MaBecker

Next

• git clone your fork

• install a tool like "Github Desktop" an add your fork with "+"

• add your changes and commit them with "Github Desktop"

---

Attachments:

• Bildschirmfoto 2017-02-26 um 11.55.07.JPG

---

Posted at 2017-02-26 by Pedro

Hi @MaBecker,

If I did not get anything wrong :) here's the branch : https://github.com/pedro-es/Espruino/tree/esp8266-enhancements

I only did one commit for all the files, as they are interrelated. The comments for the commit covers all.

---

Posted at 2017-02-26 by @MaBecker

Hi @pedro,

looks great, thanks for sharing.

Will start to implement this into my branch and run some test.

---

Posted at 2017-02-26 by @MaBecker

Your stuff is a great enhancement for ESP8266 with 4MB!

• got "4MB:1024/1024" running with fixed vars.

• used "FLASH_4MB_C1" instead of "ESP8266_BIGFLASH" :-)

next is the "scripts/build_platform_config.py" stuff

---

Posted at 2017-02-26 by @MaBecker

My implementation of FLASH_4MB_C1:

MaBecker/Espruino@b41210b

---

Posted at 2017-02-26 by Pedro

Happy it is being useful :)

Just a couple of remarks:

• No need for two different linker scripts (app1 and app2). By definition in this mode both images should be identical (which is indeed a great advantage), so we only need to generate one, and if we want to keep two for consisntency, I would point both linker scripts to the same file.
• We need to check the upper limit. In the "jswrap esp8266 network.c" file you pointed before as having a hardcoded flash address, I see it is pointing to the last page, looks like) so we may need to adjust the linker script to allow for one page less size, and of course adding 1Mb to the flash address if we are in image 2 (in runtime, getting the base flash address from the corresponding sdk os_* function)

---

Posted at 2017-02-26 by @MaBecker

Thanks

EDIT:

No need for two different linker scripts (app1 and app2)

good point, as it is just a linker script keep both :-)

Shrink firmware size by six pages and always use first 1MB area, five pages to save code and one page for wifi. What do you think ?

---

Posted at 2017-02-26 by @MaBecker

tested wiflash with "4MB:1024/1024" - works - Thanks @tve

---

Posted at 2017-02-27 by Wilberforce

@mabe > tested wiflash with "4MB:1024/1024" - works - Thanks

Please post your complete wiflash line...

---

Posted at 2017-02-27 by @MaBecker

sure

• first step

  ESP8266_BOARD=1 FLASH_4MB_C1=1 make

• second step - manual way

  cd to_your_firmware_dir

  esptool.py --port "your_com_port" --baud 460800 erase_flash

  esptool.py --port "your_com_port" --baud 460800 write_flash
  --flash_freq 80m --flash_mode qio --flash_size 4MB-c1
  0x0000 boot_v1.6.bin 0x1000 espruino_esp8266_user1.bin
  0x3FC000 esp_init_data_default.bin 0x3FE000 blank.bin

• third step - send updates with wiflash.sh

  sh wiflash.sh "ip_of_esp":88 espruino_esp8266_user1.bin espruino_esp8266_user2.bin

---

Posted at 2017-02-27 by @MaBecker

Five pages for code save and one page for wifi save - and still using one board file :)

MaBecker/Espruino@6421b97

---

Posted at 2017-02-27 by Wilberforce

Looking good. I think the change of saved code would need to be documented well.

I'm wondering if there is a better area for the saved code rather than the end of the application user Space?

---

Posted at 2017-02-27 by @gfwilliams

I'd be a bit iffy about the RESIZABLE_JSVARS - it does make things slower, and may possibly also break flat strings/arraybuffers/etc - not to mention make the whole thing just crash with a stack overflow if you run out of memory.

I'm not entirely sure what you gain over just having set up a sensible number of JsVars beforehand.

The whole unaligned access catcher would be great though - while Espruino tries quite hard not to do unaligned accesses, having something to handle the one or two cases where it does would be really handy.

Also, with the flash map - I don't know if all 1MB is really needed at the moment? What would be great is automatically saving code to the extra areas of flash if they were detected - then finally you wouldn't get the 'out of memory' errors when saving

---

Posted at 2017-02-27 by @MaBecker

@wilberforce

I'm wondering if there is a better area for the saved code rather than the end of the application user Space?

• ESP_FLASH_MAX is reduced to have extra pages for code and wifi

• afaik, this is the same technic that is used now for ESPs

@gfwilliams

Also, with the flash map - I don't know if all 1MB is really needed at the moment?

• Me too, but there are only a few KB left and as there are still improvements in Espruino, how long would it last until it does not fit in anymore?

• Using "4MB:1024/1024" still offers additional 2MB flash space

Is it possible to calculate how many space is needed by save() in worst case?

---

Posted at 2017-02-27 by @gfwilliams

there are only a few KB left and as there are still improvements in Espruino

True, and you could probably add a few extra bits and bobs (encryption, etc). With the byte read hack you could simply force all read-only arrays into flash, freeing a lot of RAM. I guess a 4MB build would probably be a good plan then.

Is it possible to calculate how many space is needed by save() in worst case?

You could argue it's JsVars * 16 bytes. But with E.setBootCode - assuming you had a way to actually write the code - you could upload a 1MB JS file if you wanted.

---

Posted at 2017-02-27 by @MaBecker

Great! So 1MB for save code and 1MB for additional flash space, that's what I am going to configure.

---

Posted at 2017-02-27 by @MaBecker

Got all puzzle pieces together.

• wifi save starts at 0x200000 with 1 page

• code save starts at 0x201000 with 255 pages

• getFreeFlash returns "addr": 3145728, "length": 3129344

MaBecker/Espruino@81204d6

---

Posted at 2017-02-28 by Pedro

Hi @MaBecker, 1MB for save code won't work, as far as I have seen in the code. The main reason is that saved code must be first allocated somewhere in RAM, and we do not have so much RAM :)
Also, when we already have saved code in flash, and we try to save state later, the whole code in flash will try to go to the stack (alloca) temporarily ro reflash, which for sure is not good.
In my code you will find an implementation of alloca that will check if we have enough stack and at least return NULL if not enough room, instead of overflowing.
Regarding RESIZABLE_JSVARS, the code is already there for Linux, I did not reinvent the wheel :) The idea is to leave as much heap free as possible when not needed for JsVars (better encryption, more connections, etc), and allow it to grow on demand. Of course there is some work to be done, to make sure we always leave a minimum, but it is just an "if" in the provided malloc function, and a check for NULL in where the code calls malloc (I think I did not see this check in the LINUX code, but it is quite straight forward)

---

Posted at 2017-02-28 by Pedro

Hi @gfwilliams, my comments on RESIZABLE_JSVARS above, in my answer to @MaBecker.

Regarding the unaligned access code, here it is. I can't remember where I found it first on the internet, so I cannot credit the original idea author (even if current code does not seem too much to the original one...).

By the way, it may need some adjustements to compile (this is just copied from another project of mine, and I have only done minor changes for it to compile in Espruino, but not really checked).

Important: change everywhere ## by the a single "## hash" caracter (unless somebedy explains me how to write a single "## hash" without it being interpreted by the forum as some special character...)

To activate the code, you need to call virtual_unaligned_memory_access() early in user_init()

 ##include "espmissingincludes.h"
 ##define RODATA __attribute__((section(".rodata")))
 ##define EXCCAUSE_LOAD_STORE_ERROR 3
 ##define EXCCAUSE_UNALIGNED 9
 ##define __stringify_1(x...) ##x
 ##define __stringify(x...) __stringify_1(x)
 ##define RSR(sr) ({ \
        uint32_t r; \
        __asm__ volatile("rsr %0," __stringify(sr) : "=a"(r)); \
        r; \
})

struct xtos_saved_regs {
        uint32_t pc;    /* instruction causing the trap */
        uint32_t ps;
        uint32_t sar;
        uint32_t vpri;  /* current xtos virtual priority */
        uint32_t a0;    /* when __XTENSA_CALL0_ABI__ is true */
        uint32_t a[16]; /* a2 - a15 */
};

typedef void (_xtos_handler_func)(struct xtos_saved_regs *);
typedef _xtos_handler_func *_xtos_handler;
_xtos_handler _xtos_set_exception_handler( int n, _xtos_handler f );

const char _trap_unaligned_debug_msg[] RODATA = "Unaligned access @%p [%08x %08x]: %p\n";
uintptr_t _trap_unaligned_last;
uint32_t _trap_unaligned_read;

static void CALLED_FROM_INTERRUPT unaligned_debug(uint8_t *pc, void *vaddr) {
        uint32_t *base = (uint32_t *)((uintptr_t) pc & ~0x3);
        ets_printf(_trap_unaligned_debug_msg, pc, base[0], base[1], vaddr);
}

uint8_t inline CALLED_FROM_INTERRUPT read_unaligned_byte(uint8_t *addr) {
        uintptr_t base = (uintptr_t) addr & ~0x3;
        if(base != _trap_unaligned_last) {
                _trap_unaligned_read = * (uint32_t *) base;
                _trap_unaligned_last = base;
        }
        return ((uint8_t *)&_trap_unaligned_read)[((uintptr_t) addr & 0x3)];
}

static void CALLED_FROM_INTERRUPT flash_emul_exception_handler(struct xtos_saved_regs *frame) {
        uint32_t vaddr = RSR(EXCVADDR);
        uint32_t instr = read_unaligned_byte((uint8_t *) frame->pc) | (read_unaligned_byte((uint8_t *) frame->pc + 1) << 8);
        uint8_t at = (instr >> 4) & 0xf;
        uint32_t val;
        uint32_t count;
        uint32_t u2s;

        if (vaddr < 0x40200000 /* 0x3ffe8000 */) {
                unaligned_debug((void *)frame->pc, (void *)vaddr);
                while(1);
        }

        if ((instr & 0xf00f) == 0x0002) {
                // |     imm8       |   r   |   as  |   at  |0 0 1 0|
                // l8ui at, as, imm    r:0
                frame->pc += 3;
                count = 1;
                u2s = 0;
        } else if ((instr & 0x700f) == 0x1002) {
                // |     imm8       |   r   |   as  |   at  |0 0 1 0|
                // l16ui at, as, imm   r:1
                // l16si at, as, imm   r:9
                frame->pc += 3;
                count = 2;
                u2s = (instr & 0x8000);
        } else if ((instr & 0xf00f) == 0x2002) {
                // |     imm8       |0 0 1 0|   as  |   at  |0 0 1 0|
                // l32i at, as, 0..1020
                frame->pc += 3;
                count = 4;
                u2s = 0;
        } else if ((instr & 0x000f) == 0x0008) {
                // |  imm4 |   s   |   t   |1 0 0 0|
                // l32i.n at, as, 0..60
                frame->pc += 2;
                count = 4;
                u2s = 0;
        } else {
                unaligned_debug((void *)frame->pc, (void *)vaddr);
                while(1);
        }

        val = 0;
        do {
                val |= read_unaligned_byte((uint8_t *) vaddr + --count);
                if(count) val <<= 8;
        } while(count);

        /* a0 and a1 are never used as scratch registers */
        frame->a[at - 2] = (u2s) ? (int16_t) val : val;
}

void virtual_unaligned_memory_access(void) {
        _xtos_set_exception_handler(EXCCAUSE_LOAD_STORE_ERROR, flash_emul_exception_handler);
        _xtos_set_exception_handler(EXCCAUSE_UNALIGNED, flash_emul_exception_handler);
}

---

Posted at 2017-02-28 by @MaBecker

Hi @pedro,

1MB for save code won't work, as far as I have seen in the code. The main reason is that saved code must be first allocated somewhere in RAM, and we do not have so much RAM :)

I fully agree, it's a waste of space.

>save();
=undefined
Erasing Flash.................................................................................................................................................................................................................................................................
Writing.....
Compressed 25600 bytes to 2224
Checking...

At the end it will be a solution that calculates JsVars * 16 byte and add the rest as FlashArea.

Regarding RESIZABLE_JSVARS .....The idea is to leave as much heap free as possible when not needed for JsVars

I definitely will try it, because this is too temptingly :)

---

Posted at 2017-02-28 by @gfwilliams

It'd be worth checking that the firmware update over WiFi can still function in the 1MB/1MB config

The main reason is that saved code must be first allocated somewhere in RAM, and we do not have so much RAM

However as I said above, you can potentially execute JS directly from flash, so if there were a way to write larger JS (eg in chunks, or with the bootloader) then it could be an option.

Wrt RESIZABLE_JSVARS - it's fun to play with, but I think it'd be a huge mistake to make the default builds use it. As I said above, the problems with flat string support could really trip people up - not to mention issues with garbage collection (I'm not sure when that happens currently).

There is a branch for 'variable cache' which IMO would be the most exciting if it had a little work done on it. It would use the RAM only for most-used variables, and would journal the rest in flash - giving you potentially 65000 variables.

---

Posted at 2017-02-28 by Wilberforce

@pedro

It'd be worth checking that the firmware update over WiFi can still function in the 1MB/1MB config

Here is the code for the over the air updates... there is a flash map address look up:
https://github.com/espruino/Espruino/blob/6da2ddd05917d98e0693a084a2d64fcfb8b800d1/libs/network/esp8266/ota.c#L171

---

Posted at 2017-03-02 by user74245

This all sounds interesting, I have some questions:

Where are wifi settings stored? Are they now stored twice and thus have to be reset after upgrading?

What about saving javascript code to flash, is that now lost when upgrading? And "EEPROM" settings?

---

Posted at 2017-03-02 by @MaBecker

Hi @user74245,

• switch to new firmware requires erase_flash

• code and wifi save check link in 💾 Espruino 2v25 released! #34

---

Posted at 2017-03-05 by Pedro

Hi @gfwilliams, I agree that if RESIZABLE_JSVARS is not stable, and until it is, the default builds should not use it, but I would definetively suggest to leave it there so that it can be compiled directly as an option from the main branch. Otherwise keeping it up to date and bring it up to stable would be a nightmare.
Just one question : I understand from your comment that it is not really working in LINUX ? Regarding "flat string support", do you mean that not everywhere in the code strings are accessed thorugh iterators ? (sorry if this is a naive question).

At least a couple of my code changes are bug fixes for the existing LINUX RESIZABLE_JSVARS part and should anyway be applied. What is the best way to get this done?

Regarding the "variable cache" branch, it is a different topic as I understand it in your comment : RESIZABLE_JSVARS leaves as much heap as possible free when not used for JSVars, and "variable cache" would allow (as you speak of flash cache) to have more JSvars than would fit on RAM. They are two related things, but different in approach and implementation (and complementary to each other in some way).
By the way, wouldn't this kind of "live" jsvar storage in flash kill it really fast ? (too many refresh cycles) I rendered one ESP8266 unusable by leaving it only one night in a reboot cycle that wrote to the system config area at every reboot.

---

Posted at 2017-03-06 by @gfwilliams

I would definetively suggest to leave it there so that it can be compiled directly as an option from the main branch

It's not going anywhere. Perhaps the build_platform_config.py file could be modified to accept 0 JsVars to mean resizable? That way it's an easy tweak.

I understand from your comment that it is not really working in LINUX ?

No, it does work. However Linux can afford to have quite a big chunk size which means that flat strings can usually be allocated.

Regarding "flat string support", do you mean that not everywhere in the code strings are accessed through iterators ?

Usually they are, but if a big area of memory is needed then sometimes it'll be allocated out of a stretch of JsVars. If the block size for resizable jsvars is too small then that may not happen. At least on Linux for things like TLS/HTTPS it won't crash, but the command will fail if a flat string can't be allocated.

Also flat strings are usually allocated for types arrays to speed up accesses - if they can't be allocated then the access speed for arrays will drop right down.

At least a couple of my code changes are bug fixes for the existing LINUX RESIZABLE_JSVARS part and should anyway be applied. What is the best way to get this done?

That's great - if you could create a branch on your GitHub for those fixes and could issue a Pull Request for them that'd be great.

But it helps a great deal to submit changes in several small groups. If something gets broken it's easy enough to look back and see exactly why things were changed then - if it's part of a massive commit you have no idea what changes go with what improvement.

wouldn't this kind of "live" jsvar storage in flash kill it really fast ?

I'm not sure if it's done yet or not, but the idea was that the variable_cache branch would use journalling when writing to flash. The sale way the FlashEEPROM module works at the moment - it's actually pretty kind on flash memory.

Also in my tests I found that the majority of code didn't cause writebacks - most often you're just loading code in to execute, and aren't changing anything so have no need to write to flash.