E.memoryArea not working on ESP8266 (due to char-iterator ?)

[espruino-discuss]

Posted at 2016-01-04 by JumJum

Tested this command on actual Esp8266 firmware.
Going down to assembler listing it breaks using an 8 bit instruction in iterator.
There could be a problem in reading bytes from Flash, reading works fine for aligned 32bits only (afaik).

[espruino-discuss]

Posted at 2016-01-04 by @gfwilliams

Yes - I think this is something we'd have to look into. It might be worth seeing if you could just tweak jsvStringIterator to read 32 bits at a time and see if that fixes it? (you'd have to change jslex.c too)

I guess seeing if you could use peek32 to access flash would be a quick way to test whether it was that easy.

---

Posted at 2016-01-04 by JumJum

peek32 works fine on flash
there only needs to be an address translation (0x71000 to 0x40271000)
peek16 and peek8 don't work and create an error

Sorry my knowledge is too poor to make the changes by myself :-(
Learned a lot, but I'm still far away from changes like this.
See this, thats my level right now, somewhere around copy/paste from already existing sources.

---

Posted at 2016-01-04 by tve

Out of curiosity, what are you trying to do with E.memoryArea? And where is it documented, I don't see it in http://www.espruino.com/Reference ?

---

Posted at 2016-01-05 by @gfwilliams

I think E.memoryArea is only in recent builds, so won't be in the reference (generally the reference matches the latest 'stable' version).

The idea is that you can put strings/arrays into flash, and can even execute code direct from it - I think this was something you'd actually asked for at some point?

Potential uses:

• On Pico - storing keys/certificates for TLS
• Storing modules (still not quite sure why you'd want this)
• Storing things like Waveform or Bitmap data
• You can even define a function and put its contents into flash

---

Posted at 2016-01-05 by JumJum

In jswrap_espruino.c line 700 and following is a description for E.memoryArea.
In my understanding this creates an additional kind of string for jsvar.
It holds start in memory(flash) and length of string.

---

Posted at 2016-01-05 by JumJum

Based on a lot of copy/paste, trial and error, E.memoryArea works for strings.
function jsvStringIteratorGetChar in jsviterator.h is extend to handle native string

static ALWAYS_INLINE char jsvStringIteratorGetChar(JsvStringIterator *it) {
  if (!it->ptr) return 0;
  if (jsvIsNativeString(it->var)){
    int addr = (int)it->ptr + (int)it->charIdx;
	uint32_t bytes = *(uint32_t*)(addr & ~3);
	return ((uint8_t*)&bytes)[(int)addr & 3];
  }
  else return it->ptr[it->charIdx];
}

In next step function jslNextCh in jslex.c is extended too and first test calling a simple function works fine.

static ALWAYS_INLINE char jslNextCh(JsLex *lex) {
  if(jsvIsNativeString(lex->it.var)){
	int addr = (int)lex->it.ptr + (int)lex->it.charIdx;
	uint32_t bytes = *(uint32_t*)(addr & ~3);
	return ((uint8_t*)&bytes)[(int)addr & 3];
  }
  else return (char)(lex->it.ptr ? lex->it.ptr[lex->it.charIdx] : 0);
}

Pretty sure there is a nicer way doing this, and we will see this in next version.
Anyway, it opened a new door of knowledge for me.

---

Posted at 2016-01-05 by @gfwilliams

That's great! So that works for you on ESP8266?

Personally I'd just #ifdef ESP8266 and access by word all the time (it'll be faster than checking if you're using a native string!).

---

Posted at 2016-01-06 by @gfwilliams

@tve I know you're doing some benchmarking at the moment? Please could you try:

static ALWAYS_INLINE char jsvStringIteratorGetChar(JsvStringIterat­or *it) {
  if (!it->ptr) return 0;
~ifdef ESP8266
  int addr = (int)it->ptr + (int)it->charIdx;
  uint32_t bytes = *(uint32_t*)(addr & ~3);
  return ((uint8_t*)&bytes)[(int)addr & 3];
~else
  return it->ptr[it->charIdx];
~endif
}

// and

static ALWAYS_INLINE char jslNextCh(JsLex *lex) {
  if (!lex->it.ptr) return 0;
~ifdef ESP8266
  int addr = (int)lex->it.ptr + (int)lex->it.charIdx;
  uint32_t bytes = *(uint32_t*)(addr & ~3);
  return ((uint8_t*)&bytes)[(int)addr & 3];
~else
  return (char)(lex->it.ptr[lex->it.charIdx]);
~endif
}

// ~ should be a hash - the forum 'eats' it otherwise

And see if it hurts performance much? My hope is that actually it won't have that much of an effect at all.

If not, it'd be great to get it in as it could really help with the low RAM situation...

---

Posted at 2016-01-06 by tve

OK. This week looks very busy for me, so it will have to wait a few days.

---

Posted at 2016-01-11 by JumJum

I did some testing with new function to use strings in flash.
Testcase is a SHA1-function and an server-object in WebIDE.
For testing a simple function calls SHA1 and creates an object.

• sent to the board with minification free:985
• checked memory some seconds later free:1168
• called testing function free:1101, duration 553ms
• checked memory some seconds later free:1133

Next testcase has SHA1-function and server-object in Flash.
Both, function and object are executed with eval from Flash.

• sent to the board with minification free:1156
• checked memory some seconds later free:1174
• called testing function free:1113, duration 500ms
• checked memory some seconds later free:1144

Result from this test(IMHO) is:

• sending from WebIDE takes some memory which is free later.
• at the end memory consumption does not change much.
• source from Flash executes 10% faster, no idea why
• reading sources from flash can be helpful for state oriented apps, where functions/objects are not needed all the time. Not used functions could then be dropped and others could be reloaded. LUA on ESP8266 supports something like this.
• I would like to have this function, especially for ESP8266 where memory is a problem

---

Posted at 2016-01-11 by JumJum

One more test.
This time the SHA1 was copied to flash without function body.
All variables have been set to undefined at the end of source.
2 more global variables are defined, one for input data, one for the result.

• sent to board with minification free:1330
• checked memory some seconds later free:1347
• called test function free:1286, duration 500ms
• checked memory some seconds later free:1292
• called test function several times, no change in free memory

---

Posted at 2016-01-12 by tve

Interesting tests, but a bit difficult to follow without any code snippets. For example, I don't know what you mean by "This time the SHA1 was copied to flash without function body." If you didn't put the SHA1 function body in flash, then what did you put in flash?

I can see how to execute code from a memory area, what I'm not yet seeing is how to put it all together. One option could be to have a special method to load a bunch of source code into a flash area and tell the interpreter: here, load this. But that now creates a completely separate path from the way the IDE or command line tool work to load code.

Or we could modify some stuff in jsinteractive that puts code into a memory area by default. But now we need to create a memory allocator for the flash area that can also deal with code being replaced, i.e. areas being freed, etc. On top of that then comes save() which somehow needs to make sure that whatever is in RAM that points to the flash is saved (perhaps that's in a JSvar so automatically gets saved, I haven't looked). There's probably a simple solution that goes 90% of the way, like dropping the flash area on reset() and only adding to it, never freeing until the next reset().

---

Posted at 2016-01-12 by JumJum

Let me try to give a better explanation. Its more or less a proof of concept.
The idea behind is to use it like a "firmware-extension".
First of all, I made some changes to Espruino firmware:

• new code for handling strings in flash http://forum.espruino.com/comments/12709163/
• added functions for a kind of housekeeping in a flash page.
• Start of page is 4 bytes for each string, first 2 bytes are relative start, next 2 bytes are length
• http://jumspruino.jumware.com/sources/flashMemory/jswrap_FlashExtension.h
• http://jumspruino.jumware.com/sources/flashMemory/jswrap_FlashExtension.c
  Next created a nodejs application to build a page for flash.
  My idea is to make this available some day.
  Today my learning curve is still steeply rising, got my public virtual server some days ago ....
  Could give access right now on a personal base only :-(
  Writing this to flash is done with nodemcu firmware programmer.
  Data is written to 0x71000 and/or 0x72000, since this is the first empty page I found.
  Source for 1st test with function and object. Only removed module-relevant commands
• SHA1 function http://jumspruino.jumware.com/sources/http/SHA1.js
• http-server object http://jumspruino.jumware.com/sources/http/EspSrv.js
• test itself http://jumspruino.jumware.com/sources/flashMemory/Test1.js
  Source for 2nd test
• SHA1 sourcecode http://jumspruino.jumware.com/sources/flashMemory/SHA1.js
• test itself http://jumspruino.jumware.com/sources/flashMemory/Test2.js

---

Posted at 2016-01-26 by @gfwilliams

I've just added some code to handle this here I'm not able to test on my ESP8266 for some reason so it'd be great if someone could take a look at the builds and see if they still work ok.

---

Posted at 2016-02-11 by Wilberforce

Hi Gordon,
Is there any easy way to tell which builds now have the E.memoryArea update for reading esp8266 flash?
I don't think this is any @tve builds yet?

---

Posted at 2016-02-11 by @gfwilliams

You could look at the ChangeLog on @tve's GitHub and figure out which commits have the changes in (if any). Otherwise the main travis builds for Espruino are auto-built and should be usable.

According to the comments on this bug it may not work though. I'd kind of assumed that after all the requests someone might have tested it, but it seems not :(

---

Posted at 2016-02-11 by Wilberforce

Ah - I'll try one of the auto builds this weekend and report back if it works.
thanks

---

Posted at 2016-02-12 by JumJum

Just tested with sources from github, for me it works fine.
Also tested E.memoryArea, reading a string from flash and executed with eval, works fine too.

peek32(0x3ff00054).toString(16)
="200991a"
>peek16(0x3ff00054).toString(16)
="991a"
>peek16(0x3ff00056).toString(16)
="200"
>peek8(0x3ff00054).toString(16)
="1a"
>peek8(0x3ff00055).toString(16)
="99"
>peek8(0x3ff00056).toString(16)
="0"
>peek8(0x3ff00057).toString(16)
="2"

---

Posted at 2016-02-12 by @gfwilliams

That's awesome - thanks for checking! Nice to know it is all working properly, at least in some builds :)

---

Posted at 2016-02-14 by Wilberforce

I can confirm the peek stuff above works, however I get a reset with E.memoryArea trying to read the Esp8266 flash:

>esp8266 = require("ESP8266");
=function () { [native code] }
>var base=esp8266.getFreeFlash()[2].area;
=1011712
>E.memoryArea(base,10);
="
 ets Jan  8 2013,rst cause:2, boot mode:(3,6)
load 0x40100000, len 1396, room 16
tail 4
chksum 0x89
load 0x3ffe8000, len 776, room 4
tail 4
chksum 0xe8
load 0x3ffe8308, len 540, room 4
tail 8
chksum 0xc0
csum 0xc0
2nd boot version : 1.4(b1)
  SPI Speed      : 80MHz
  SPI Mode       : DIO
  SPI Flash Size & Map: 32Mbit(512KB+512KB)
jump to run user1 @ 1000
don't use rtc mem data

>process.env
={
  "VERSION": "1v84",
  "BUILD_DATE": "Feb 12 2016",
  "BUILD_TIME": "08:57:53",
  "GIT_COMMIT": "8d4df7829bc7ebc8bc701c0f04e2fc645b99bd02",
  "BOARD": "ESP8266_BOARD",
  "CHIP": "ESP8266",
  "CHIP_FAMILY": "ESP8266",
  "FLASH": 0, "RAM": 81920,
  "SERIAL": "5ccf7f00-d975",
  "CONSOLE": "Serial1",
  "EXPORTS": { "jsvLock": 1075994728, "jsvLockAgainSafe": 1075994712, "jsvUnLock": 1075981000, "jsvSkipName": 1075970120,
    "jsvMathsOp": 1075935700, "jsvMathsOpSkipNames": 1075960212, "jsvNewFromFloat": 1076000164, "jsvNewFromInteger": 1076000232, "jsvNewFromString": 1075999068,
    "jsvNewFromBool": 1076000200, "jsvGetFloat": 1075970588, "jsvGetInteger": 1075971844, "jsvGetBool": 1075971036, "jspeiFindInScopes": 1075910904,
    "jspReplaceWith": 1075904248, "jspeFunctionCall": 1076010408, "jspGetNamedVariable": 1075912700, "jspGetNamedField": 1075923460, "jspGetVarNamedField": 1075922944,
    "jsvNewWithFlags": 1075999608 }
 }

It works with the 0x3ff00054 address:

>peek32(0x3ff00054).toString(16)
="20000d9"
>E.memoryArea(0x3ff00054,4);
>E.memoryArea(0x3ff00054,4);
="\xD9\x00\x00\x02"

So this seems to be associated with the address....

https://github.com/espruino/Espruino/wiki/ESP8266-Design-Notes

---

Posted at 2016-02-14 by JumJum

Flash in ESP8266 is mirrored (windowed?) to 0x40200000.
So if you want to use E.memoryArea for 1011712 (0xf7000) please check 0x0402f7000.
BTW, you could check for free flash, since value given by esp8266.getFreeFlash() returns a value which is designed to be free. If firmware does not use all of reserved memory for firmware, there is some more available flash, right now starting at 0x72000(0x40272000)
One more BTW, for more confusion, flash.read does adress calculation internally.

flash = require("Flash");
function mem(){
  for(var i = 0x70000;i < 0x80000;i+=4096){
    console.log(i.toString(16),flash.read(8,i));
  }
}

if this returns 255 for all fields, usually this is available flash

---

Posted at 2016-02-15 by @gfwilliams

hmm, so could it be that the READ_UINT8 macro won't work graphics (As in @MaBecker's bug) because the address used is wrong? That could complicate things :(

I guess we'd need a second Macro with if (addr < RAM+ADDR) addr|=0x40200000;

---

Posted at 2016-02-15 by Wilberforce

Thanks. Using the offset works ok- I'm wondering if there is a way of hiding this transparently like Flash() lib does?

I was expecting the esp8266.getFlash() to return these larger regions, it doesn't look like this is implemented yet.

I also believe these is a restriction of 1mb max, and that this is in the espurino core rather than the esp8266 implementation. Does anyone know why this is the case?

---

Posted at 2016-02-15 by Wilberforce

if (addr < RAM+ADDR) addr|=0x40200000

That makes sense, and would explain it.

---

Posted at 2016-02-15 by @gfwilliams

I also believe these is a restriction of 1mb max, and that this is in the espurino core rather than the esp8266 implementation.

Well, it's not a restriction in Espruino itself - maybe just something about getFlash?

---

Posted at 2016-02-16 by Wilberforce

I also believe these is a restriction of 1mb max, and that this is in the espurino core rather than the esp8266 implementation.
Well, it's not a restriction in Espruino itself - maybe just something about getFlash?

I was going by this comment:
https://github.com/espruino/Espruino/blob/master/targets/esp8266/jshardware.c line 48

The restriction seems to be here:

// Save-to-flash uses 12KB at 0x78000
// The jshFlash functions use memory-mapped reads to access the first 1MB
// of flash and refuse to go beyond that. Writing uses the SDK functions and is also
// limited to the first MB.

#define FLASH_MAX (1024*1024)
#define FLASH_MMAP 0x40200000
#define FLASH_PAGE_SHIFT 12 // 4KB
#define FLASH_PAGE (1<<FLASH_PAGE_SHIFT)

Looking at the fetch:

void jshFlashRead(
    void *buf,     //!< buffer to read into
    uint32_t addr, //!< Flash address to read from
    uint32_t len   //!< Length of data to read
  ) {
  //os_printf("jshFlashRead: dest=%p, len=%ld flash=0x%lx\n", buf, len, addr);

  // make sure we stay with the flash address space
  if (addr >= FLASH_MAX) return;
  if (addr + len > FLASH_MAX) len = FLASH_MAX - addr;
  addr += FLASH_MMAP;

So really #define FLASH_MAX (1024*1024) should be determined by the type of Flash chip in the esp8266 board.

https://github.com/espruino/Espruino/blob/master/targets/esp8266/jswrap_esp8266.c

JsVar *jswrap_ESP8266_getFreeFlash() {
  JsVar *jsFreeFlash = jsvNewArray(NULL, 0);
  // Area reserved for EEPROM
  addFlashArea(jsFreeFlash, 0x77000, 0x1000);

  // need 1MB of flash to have more space...
  extern uint16_t espFlashKB; // in user_main,c
  if (espFlashKB > 512) {
    addFlashArea(jsFreeFlash, 0x80000, 0x1000);
    if (espFlashKB > 1024) {
      addFlashArea(jsFreeFlash, 0xf7000, 0x9000);
    } else {
      addFlashArea(jsFreeFlash, 0xf7000, 0x5000);
    }
  }

So this could be used rather than the #define:
extern uint16_t espFlashKB; // in user_main,c

---

Posted at 2016-02-16 by tve

Maybe E.memoryArea should use flash addresses (starting at 0) instead of memory mapped addresses?
In any case, it can only access the first 1MB of flash due to esp8266 memory mapping limitations.

---

Posted at 2016-02-16 by @gfwilliams

Maybe E.memoryArea should use flash addresses (starting at 0) instead of memory mapped addresses?

I guess we'd need some ESP8266-specific tweaks in there for that, but it could be a plan.

In any case, it can only access the first 1MB of flash due to esp8266 memory mapping limitations.

@wilberforce So jshFlashRead could potentially use more memory as it can use API calls, but memoryArea needs a pointer, so it will never be able to access more than the first 1MB.

---

Posted at 2016-02-16 by JumJum

Would it be possible/an option to move the defines from jsutils to jshardware ?

---

Posted at 2016-02-16 by @gfwilliams

At the moment there's no per-platform .h file apart from platform_config.h - that could be an option?

I agree it's a shame to have loads of ESP8266 ifdefs in jsutils.h, but right now I think even if those moved, there would still be a bunch of stuff for jsException/jsError (which is used to get the error messages moved out of RAM).

---

Posted at 2016-03-07 by Wilberforce

Does using the api get around the memory mapped limitation of accessing > 1mb memory for flash?
https://github.com/espruino/Espruino/blob/9781bacee8185b58438604cb22672389bc3ff42d/targets/esp8266/jshardware.c
Line 1193

For writing the api call is used,but for reading it is just accessing memory. If the Api was used for the reading could we using the flash module for accessing > 1 mb?

---

Posted at 2016-03-07 by @gfwilliams

I would imagine so, but there might be some gotchas. @tve might know?

---

Posted at 2016-03-16 by tve

Speed is the main gotcha, IIRC. Espruino uses 4-byte flash reads when loading stuff like the saved code from flash, so if each one turns into an SDK call which has to tweak the HW memory map back and forth to access the rest of flash it's gonna be real slow.
Why not add an esp8266-specific call to access the rest of flash?

---

Posted at 2016-03-16 by tve

Or perhaps a better option: if the flash region accessed is in the lower 1MB use the existing memory-mapped code, if it's higher use the SDK calls...

---

Posted at 2016-03-17 by @gfwilliams

I think you'd be fine using the API calls for jshFlashRead, if that's what you were thinking of? jshFlashRead isn't (or shouldn't) be called that often so wouldn't hurt performance.

As @tve says, I wouldn't start changing internal stuff (eg. E.memoryArea) to use it, or performance will really suffer.

---

Posted at 2016-03-17 by Wilberforce

Looks like the conditional if < 1mb direct read otherwise api is the way to go.

Now that the flash module returns available flash should the esp8266 call be depreciated?

If I manage to get this stuff working, I'll have to extend the free areas to return the new available space too

---

Posted at 2016-03-17 by @gfwilliams

The esp8266 call is deprecated - it'll say so in the documentation when the next version is released (the online documentation matches the current release).

---

Posted at 2016-03-30 by Wilberforce

The code compiles, but I've not had a chance to test yet:
espruino/Espruino@master...wilberforce:master

---

Posted at 2016-03-31 by Wilberforce

espruino/Espruino#827

Tested ok on 4mb board.