Moving to ESP-IDF release 3.0, and some Bluetooth, and ....

[espruino-discuss3]

Posted at 2018-04-25 by JumJum

For a day or so, V3.0 is released.
We've been waiting long time for this step.
My plan now is, to get Espruino running with actual sources, which are

• Espruino itself
• ESP-IDF V3.0
• Bluetooth support for ESP32
• support PSRAM in WROVER boards giving us 20000 jsVars (or more ?)
• support of BLE serial (NRF UART) is under development. Windows 10 Is not really helpful for that. This will take some time.
  -last not least documents
  Everything will be done first in ESP32 branch.

There are some restrictions like

• reduce jsvars to 2500, I'm just testing with 3000 and got no problems(yet). If you need more, WROVER boards are available. Last not least, there is somebody in the ESP32 forum adding a 8MB PSRAM, priced less than 1$
• Bluetooth eats memory like crazy. Size of binary, including BLE Uart and bonding is 1280KB. There is a build option to build Espruino without Bluetooth, but even that is really big
• do we need OTA for ESP32 ? Flashing more than 1MB over the air, is this really helpful ? It reduces available Flash a lot.

Once this upgrade to V3.0 is done and works fine, next steps are

• merge ESP32 branch into master. For this step @gfwilliams is the man
• upgrade EspruinoBuildTools to V3.0. This is a tool from @wilberforce, so he has the knowhow

Please take my plan as starting point only.
Any comments, especially from guys (hopefully) involved, are welcome.

[espruino-discuss3]

Posted at 2018-04-25 by @gfwilliams

This sounds great - thanks! I don't think OTA works on ESP32 at the moment, so I wouldn't waste time worrying about it.

I imagine most users would prefer BLE to the extra few vars given the choice, and memory usage isn't such a nightmare now it's reasonably easy to upload code directly to flash.

---

Posted at 2018-04-26 by Wilberforce

Your right - the ota stuff was never completed.
My two cents worth... this is a possible plan

1. Make a partition table to use for BLE and non BLE
2. Update espurino build tools - add a espv3 branch and get that working
3. Add an espv3 branch to espruino and get that building against the build tools branch
4. Once this is working, this can be pulled into master
5. Look at merging the esp32 branch

At this stage I'm not sure if the esp32 ble branch builds if the BLE Makefile flags are off?

I believe we should offer a build without BLE, as the increased RAM usage and power requirements might not suit everyone.

Perhaps it can be one firmware with the option to turn BLE on from JavaScript?
Or the other options to offer two firmware - one with BLE and one without?

How does this sound?

---

Posted at 2018-04-26 by Spocki

sounds ble-eding good ;-)

---

Posted at 2018-04-27 by JumJum

@wilberforce, thanks for your feedback

• So we have agreement for removing OTA, great.
• I've seen, you already created an espv3 branch. Would it be correct, to change line 3 to v3 and line 4 to esp32 branch in build-idf.sh ? I will do my best to bring up a version. Pretty sure, I've to contact you to get a better understanding of BuildTool. For example, right now, I've no idea how to test my changes.
• For compiling we will need a new compiler version. How/where would we tell which compiler to use ?
• with/without BLE is a problem. Creating 2 firmware binaries would mean 2 different precompiled templates, more maintenance, etc. etc. An option to switch BLE on/off during runtime would not help, since we would need a kind of jsVar-renumber to increase or decrease number of jsVars.
• and, number of jsVars, hmm, using V3 already eats a lot of RAM, much more than V2.1. My assumption is, maximum will be somewhere between 3500 and 4000 jsVars. If there is anybody interested in that and having the time to check this seriously, please go ahead and give us some feedback.
• BTW, do you have experience with Gordons point to upload code directly to flash ?
• I had a big discussion with Gordon in Nürnberg. He has a great idea, how to reduce RAM usage for jsVars and store them in Flash. This is future talk, but will be a big step.

---

Posted at 2018-04-27 by Wilberforce

I’ve started an issue page here - it’s probably best to track there.
espruino/Espruino#1387

I have done a pull request on the master branch which updated the compiler. This is done in the provision script.

Will add more answers to your questions today.

---

Posted at 2018-04-28 by Wilberforce

At this stage I still can't get a build running, as it's expecting the psram, and that not available on the standard boards. I need to get the build working with a standard board and v3 before we can move on...

---

Posted at 2018-04-28 by Spocki

http://esp-idf.readthedocs.io/en/latest/api-reference/kconfig.html#spiram-support perhaps?

---

Posted at 2018-04-29 by Wilberforce

I've got a plain build with v3 and 4000 vars booting. Please see the github post above.

Next step is to update the build tools so this can be part of master, then look at bring in the Bluetooth

---

Posted at 2018-05-08 by SergeP

Hi! I've tried to build ESP32 branch, and I have no success.
First of all, I have found that 'source scripts/provision.sh ESP32' (as in docs) gets app and esp-idf from master branch of EspruinoBuildTools while code uses newer version already. I have corrected it to get ESP32 branch.
Then I have got an error about unknown gcc option '-mfix-esp32-psram-cache-issue'. After removing it from makefile (in any case I have no board with psram) I have all files compiled. But at linking stage I have got "undefined reference to `esp_ble_gap_set_security_param'" error.
While I think the build will not work without the call, I have commented all calls of esp_ble_gap_set_security_param() to look for next stop. It is "cp: cannot stat '/home/serge/Projects/ESP32/Espruino/app/build/partitions_espruinoBLE.bin': No such file or directory"
My build commands are:

source scripts/provision.sh ESP32
make clean && BOARD=ESP32 DEBUG=1 USE_BLUETOOTH=1 make

---

Posted at 2018-05-08 by Wilberforce

Sorry - this stuff is in flux at the moment. I have created a new branch to resolve the issues, then will push once all the testing is done. This branch updates the tool chain and solves all the issues you are seeing.
I then have the task to merge the two branches into master...

Can you wait a bit longer?

If it is firmware you want to try with Ble compiled, I can provide a link for that.

---

Posted at 2018-05-09 by SergeP

Thank you. My idea is to transfer Puck.JS BLE-MQTT bridge from Raspberry-like system to ESP32 and use one of public MQTT services with it. It may be cheap and simple solution.

---

Posted at 2018-05-09 by Wilberforce

@sergep

@jumjum published a build here:

https://github.com/espruino/Espruino/tree/545cce3a201ca65cea73a1faf49c92dae2264c00/targets/esp32/BLE/Firmware

You will need to use all bin files, as the partition table is different. Please note that it will be changing from what is in the firmware linked above. You might want to do a flash erase.

If you do get a puck linked to the esp32 as a bridge, please start a new topic and share!

---

Posted at 2018-06-16 by MarkT

PS, as a commercial entity, OTA is critical to update client devices.. That's a showstopper if its not available.

---

Posted at 2018-06-16 by DrAzzy

What is the state of the jsvars-in-psram thing for wrover boards? What will the maximum number of jsvars be, assuming we accept that we can't fill it all the way and then save() it? There s got to be some maximum at which it needs more bits to address it than there are in the 16-byte jsvar....

---

Posted at 2018-06-17 by Wilberforce

I think @jumjum was talking about 20,000 at one stage.

---

Posted at 2018-06-18 by JumJum

Actual ESP-IDF release supports WROOM or WROVER boards.
Problem is the or, you cannot build a binary which runs WROOM and WROVER
There was an simple extension in master for some month before release, but it did not make it to release 3.0
Since Espressif does not give any information like a roadmap, we have to wait and hope for 20000 or even more jsVars.
Obviously you could build your own firmware for WROVER boards. You only have to switch PSRAM support on in menuconfig. Espruino itself is ready for that. See line 51 to 54 in targets/esp32/main.c

---

Posted at 2018-06-18 by DrAzzy

@jumjum - I'm a bit confused by that post.

Is is currently possible to build for WROVER? It's not clear to me if the thing we're hoping for from Espressif is to allow WROOM or WROVER to be supported with the same binary, or if something is needed from them to support the PSRAM at all.

---

Posted at 2018-06-18 by Wilberforce

Espressif is to allow WROOM or WROVER to be supported with the same binary

They do support this, however it was not in the v3.0 release. It is the working branch ( unstable) at the moment.

To build with wrover with the v3.0, you have to make a change to the library, and then the same binary can't used on a regular board.

---

Posted at 2018-06-20 by Wilberforce

@jumjum

https://github.com/espressif/esp-idf/releases/tag/v3.0.1

There is a new release - not sure if this addresses any of your concerns.

I can try a build with espruino-build tools and see if it works directly or if there are other changes required.

---

Posted at 2018-06-21 by JumJum

@wilberforce
V3.01 does not, but there is a prerelease V3.1 which supports with/without PSRAM
I'll take a look to that to see how to get it running with Espruino.
Already known is renaming of directories and header files
Prerelease comes very soon after release, much faster than in the past.
Lets see how long we have to wait for release of V3.1

---

Posted at 2018-06-21 by Wilberforce

@jumjum
Do you think it's worth updating to 3.01 ? Looks like there are some bug fixes.

---

Posted at 2018-06-21 by Wilberforce

@jumjum

Please can you explain what the minHeap means here?

>ESP32.getState()
={
  "sdkVersion": "v3.0",
  "freeHeap": 38528, "BLE": true, "Wifi": true, "minHeap": 32720 }

---

Posted at 2018-06-21 by JumJum

@wilberforce
There is a call in API to get the minimal size of heap (heap_caps_get_minimum_free_size)
AFAIK, this is the minimum size which happened during processing.
The idea is to find out, which function uses how much heap.
For example, create http-server and check this value, reset, create https and check, etc. etc.
Up to now, I did not work on this, with other words, no experience yet ;-)

---

Posted at 2018-06-26 by Wilberforce

@jumjum

I have updated the build tools to use esp-idf 3.0.1

Let's keep a watch here:

https://github.com/espressif/esp-idf/releases

There is a v3.1-beta1 there now.

Should be update the old esp32 branch to current and use that? The v3.0 branch is now named incorrectly for the current v3.1.

I suppose we could drop that branch and add a 3.1 that we can then advance as the beta becomes live?

You mentioned Makefile and path changes that will need to be sorted.

---

Posted at 2018-06-26 by JumJum

@wilberforce,
I started to get Espruino compiled with V3.1-beta.
There is a change in DPORT handling, NEOpixel uses that.
Some makefile and path changes need to be done.
Right now I'm at a point where linker tells about multiple definitions around mbedtls_sha256_init and other mbedtls_sha256_ functions.
Message is like this:
***/home/esp32/V3_1_prerelease/template/build/mbedtls/libmbedtls.a(sha256.o): In function mbedtls_sha256_init': /home/esp32/V3_1_prerelease/esp-idf/components/mbedtls/mbedtls/library/sha256.c:82: multiple definition of mbedtls_sha256_init'
libs/crypto/mbedtls/library/sha256.o:/home/esp32/V3_1_prerelease/Espruino/libs/crypto/mbedtls/library/sha256.c:79: first defined here
/home/esp32/V3_1_prerelease/template/build/mbedtls/libmbedtls.a(sha256.o): In function mbedtls_sha256_free': sha256.c:(.text.mbedtls_sha256_free+0x0): multiple definition of mbedtls_sha256_free'
libs/crypto/mbedtls/library/sha256.o:sha256.c:(.text.mbedtls_sha256_free+0x0): first defined here

---

Don't know, how to get rid of this :-( Hopefully somebody out there knows, how to get rid of this

---

Posted at 2018-06-27 by Wilberforce

libmbedtls.a
Looks like this is being referenced in two different paths.

They might have moved it to a new location, and the Makefile, and your build, has both, so it's finding both and getting confused.