Sending code to ESP32 slow using EspruinoTools -- says "Set Slow Write = true"

[espruino-discuss3]

Posted at 2018-03-11 by jugglingcats

I tracked this down to versionChecker.js in the EspruinoTools code. This checks for firmware version but also that env.CONSOLE is one of USB, Bluetooth or Telnet.

In my case env.CONSOLE is equal to Serial1. If I hack this in as an option I get fast writes -- whoop!

Just wondering if this is deliberate or a bug/missing feature of EspruinoTools. I saw a comment somewhere that comms over serial has no flow control, but this is virtual serial over USB to the ESP32 and not sure if same applies.

I'm going to leave my hack in for a while and see if there are any issues. It makes a big difference to speed of dev cycle when using espruino --watch.

Thanks

[espruino-discuss3]

Posted at 2018-03-11 by jugglingcats

Hmm well so far not so good...

New interpreter error: FIFO_FULL
>Guru Meditation Error of type LoadProhibited occurred on core  0. Exception was unhandled.

Get this error on second transfer -- every time. Interestingly this doesn't happen with all code. If I create a simple hello world app I can change it multiple times no problem and the transfer is ok.

Could be something to do with code size or maybe something about my app itself (it has a heavy interval timer).

EDIT: I assume ESP32 serial over USB is less capable flow-control wise than the Espruino Pico and this is simply a buffer overflow issue. I played around with the slow serial block size in the code and found 128 bytes to be a happy medium between speed of transfer and stability. Default (slow send) is 19.

---

Posted at 2018-03-12 by @gfwilliams

Yeah, it's because:

• There's no flow control by default (it can be enabled) and some boards can't chew the data fast enough as it is uploaded. Every other Espruino board uses 9600 baud, but ESP32/ESP8266 don't because it's 'faster' - but then the IDE has to deliberately slow everything down to near 9600 baud speeds anyway because in some cases it can't be handled fast enough :(
• On Chrome, the serial API actually loses data if you send too much data at a time on some platforms - it's totally unrelated to the board, and seems something to do with the serial port interface chips on some of the boards.

There could be an option for it: You can force the 'slow write' on using the Throttle Send option - I guess that could be a drop-down allowing the option to only throttle various amounts.

But honestly, it may work fine for you, but running in Chrome, on a different platform or with a different ESP32 with a different interface chip it'll probably break - which is why it's done that way. Having something slightly slower than it could be is worth it in order to have it 'just work' for as many people as possible.

---

Posted at 2018-03-12 by jugglingcats

Thanks. I actually added an option on my local CLI, -s --serial-block-size. I can send you a pull request if useful. I found that on the ESP32 128 works well enough. I didn't look at adding this in the Web IDE because I don't tend to use it except for testing tiny things (partly due to my Typescript pipeline...)

---

Posted at 2018-03-12 by @gfwilliams

Thanks, but I think anything I merged would have to include a proper option in the Web IDE as well.

There's already Throttle Send so I'd prefer to just change that rather than adding another option.

---

Posted at 2018-03-12 by jugglingcats

Happy to modify and have a go at the Web IDE if you can suggest right approach...

For the ESP32 env.CONSOLE is coming through as Serial1, so setSlowWrite(true) is called in versionChecker.js. In this case SERIAL_THROTTLE_SEND has no effect. I think the options are either to provide an explicit block size for slow sends only, which is what I've done or provide an explicit block size for all sends slow or not.

@wilberforce, Gordon says that flow control "can be enabled". Do you think this is a possible option for ESP32? In which case we wouldn't need hacky options.

---

Posted at 2018-03-12 by Wilberforce

Not sure about the serial console, @jumjum wrote this code. I don't know if all boards have flow control enabled.

Your other option is to use the telnet port 23 to send code up to the esp32, bypassing serial. In the web ide you can cannot via ip - not sure if the command line tools are set up for this.

---

Posted at 2018-03-12 by jugglingcats

Sending over TCP is quick from the Web IDE, but wifi doesn't seem to be completely stable (then again my ESP32 doesn't seem completely stable in general!).

There is an issue open for enabling TCP in the CLI... espruino/EspruinoTools#32.

---

Posted at 2018-03-12 by Wilberforce

This is from the readme...

espruino --board PUCKJS --minify mycode.js -o output.js

Then you could use nc or similar to send output.js to the telnet port...

---

Posted at 2018-03-12 by jugglingcats

You gave me an idea. At the moment I'm running webpack --watch in one terminal and espruino --watch in another. I could write a simple webpack plugin to run espruino CLI (or integrate with the lib) after each build and push the result to the device.

---

Posted at 2018-03-12 by @gfwilliams

@jugglingcats it would be very cool if you could implement the CLI TCP issue. It should be reasonably straightforward.

In terms of throttling, I guess you'd make the option a drop-down with something like:

• Auto (default)
• Force throttling 19 bytes
• Force throttling 128 bytes
• Force no throttling (with a warning)

For flow control, you just need to add jshSetFlowControlEnabled to jshUSARTSetup. It's done in software so should work on all platforms (it's just that ESP32/8266 don't expose it). Ideally you'd modify built_platform_config.py to check for a default_console_cts option that then made a DEFAULT_CONSOLE_CTS_PIN definition that the ESP32 initialisation then checked, so it could be tweaked based on the board.

---

Posted at 2018-03-12 by @gfwilliams

But... it also depends whether the IDE requests hardware flow control or not - it may not.

---

Posted at 2018-03-12 by jugglingcats

Hi @gfwilliams I tried adding the call and it didn't cause any issues but didn't seem to help either - still seeing New interpreter error: FIFO_FULL. This is from the command line. Is there a way I can check if hardware flow control is being attempted?

---

Posted at 2018-03-13 by @gfwilliams

You'd have to dig around in the code I'm afraid - serial_nodeserial.js for the CLI afaik. Honestly though I have to focus my time on the tools for folks that actually buy Espruino hardware.

From my point of view, I make nothing out of ESP32. Making uploads 50% faster doesn't change anything for me at all - but if 1 in 100 uploads fails with the new code then people come to the forum, email, twitter - and that ends up costing loads of time in support - so this is all super far down my priority list.

---

Posted at 2018-03-13 by jugglingcats

100% understand -- read your Makezine article... will try and stay out of your hair.

Hopefully things are working for you financially with the range of hardware you have and that having the other ports in the ecosystem at least adds some value.

---

Posted at 2018-03-13 by @gfwilliams

Thanks - yes, it's doing ok. I'd be earning more in a 'proper' job, but obviously I have a lot more freedom over what I do this way. There are all kinds of pros and cons of the ESPxxx devices, but it's impossible to say either way if there's a net benefit - I wish it was a bit more clear-cut!

Anyway, it's great to see people using Espruino and pushing it as far as they can - I just won't always be able to reply when I'm mentioned in a non-official board thread :)

Things that can be done that make Espruino faster/better as a whole (like your TCP/IP CLI mods) are fantastic - it's just more of a grey area if it's not actually doing anything for official boards, and it's turning much darker grey if it's also making code harder to maintain or impacting negatively on other devices :)

---

Posted at 2018-03-19 by JumJum

flow control for ESP32 is set to UART_HW_FLOWCTRL_DISABLE
Just checked all options for flow control

• UART_HW_FLOWCTRL_DISABLE
• UART_HW_FLOWCTRL_RTS
• UART_HW_FLOWCTRL_CTS
• UART_HW_FLOWCTRL_CTS_RTS
• UART_HW_FLOWCTRL_MAX

All with RTS don't work.
All others have same speed.

---

Posted at 2018-04-22 by Wilberforce

@jugglingcats

Thanks. I actually added an option on my local CLI, -s --serial-block-size. I can send you a pull request if useful. I found that on the ESP32 128 works well enough.

I would like to look at what is involved with this, as I need to use throttle on send when using the linux espruino build on telnet on the web ide. If throttle on send is off:

New interpreter error: FIFO_FULL

I would like to get an idea about where to increase the buffer size to 128.
When throttle on send is used, it is overly throttled, so would be great to be able to tune this.

Thanks.

---

Posted at 2018-04-22 by jugglingcats

Hi @wilberforce, here are my changes:

espruino/EspruinoTools@gh-pages...jugglingcats:serial-block-size

---

Posted at 2018-04-22 by Wilberforce

@jugglingcats

Thanks for posting.

---

Posted at 2018-04-23 by @gfwilliams

@wilberforce the best bet would be to tweak Telnet to add flow control in the firmware itself?

It should be very easy. Just make: https://github.com/espruino/Espruino/blob/master/libs/network/telnet/jswrap_telnet.c#L292

Check jshHasEventSpaceForChars.

Maybe lower the buffer size a bit to make it a bit more fine-grained, but something like:

  char buff[64];
  if (!jshHasEventSpaceForChars(sizeof(buff))) return false;
  int r = netRecv(net, ST_NORMAL, tnSrv.cliSock-1, buff, sizeof(buff));
  if (r > 0) {
    jshPushIOCharEvents(EV_TELNET, buff, (unsigned int)r);
  } else if (r < 0) {
    telnetRelease(net);
  }

would probably be perfect, and then TCP/IP should handle all the flow control stuff itself.

---

Posted at 2018-04-24 by Wilberforce

@gfwilliams

Thanks for the hint. I'll give that a try and if it works out ok I'll do a pull request.

---

Posted at 2018-04-26 by Wilberforce

@gfwilliams

I had a chance to try it - the 64 byte buffer works a treat on the esp32. The upload is so fast that if you blink you miss it, and that was pulling in 3 modules!

---

Posted at 2018-04-26 by @gfwilliams

Fantastic! PR? :)