How to send a large amount of data faster ? #6003
Replies: 1 comment
-
Posted at 2018-01-21 by @allObjects ...no surprise to me... Arduino leaves the ESP8266 alone to do just its (Wifi) work... running now the serving and everything on ESP8266 - and in JS - ... it could be a buffer size issue (in general and in particular in comm w/ Espruino and ESP8266 Wifi firmware... RAM is after all limited... now it does not only serve the Wifi send and receive, but all Espruino. --- I amy be totally mistaken, though... Posted at 2018-01-21 by Polypod You mean there is no work around or better way to send faster on espruino ? To download its usable 3 mb of data, it would take ... 50 minutes. :( Posted at 2018-01-22 by Wilberforce The drain event might not send until a buffer is full... What happens if you send all of the sequence in one go? Posted at 2018-01-22 by Polypod Is it possible to flush this buffer to be sent immediately somehow ? or get called until the buffer is really full ? I can't send 20kb in one go, I get "New interpreter error: LOW_MEMORY,MEMORY". If I send only 1 chunk of 5bk, it still takes 5 seconds even straight away without ondrain:
It actually seems hanging few seconds before sending anything. Posted at 2018-01-22 by ClearMemory041063 I'm trying to visualize your hardware setup.
The first case would be slower because the uP has to handle more tasks (marshall data to send, and send data) Posted at 2018-01-22 by Polypod I am in the first case: Esp8266 with Espruino flashed into the Esp8266. By marshalling data, I assume you refer to the E.toString. It's not much different without marshalling. The loop finish in only 13ms with a straight string (instead of 26ms for uint8array), but the 5kb of data still make 5 seconds to finish loading in the browser, in at least ten pieces :
Posted at 2018-01-22 by ClearMemory041063 We you test it using Arduino, is the Arduino flashed onto the Esp8266? Posted at 2018-01-22 by Polypod @ClearMemory041063, not sure I understand correctly, yes I tested with an arduino program flashed on the esp8266. With arduino, I believe there is no other way than to flash my program in there. Posted at 2018-01-22 by ClearMemory041063 Please post the Arduino code for comparison. Here is another approach: Try moving the new UintArray(chk) to a Global scope
Posted at 2018-01-22 by ClearMemory041063 Try moving some data to see how much is being moved
Saved the page as test.txt and it contains 20 kB of data. Attachments: Posted at 2018-01-23 by Polypod @ClearMemory041063, I am surprised it goes a bit faster on your setup. I still got 17 sec for the 20kb, when I tried the two programs you posted. I tried with/out AP/station, no change. What version of espruino are you running ?
Posted at 2018-01-23 by Polypod The arduino code I used is from this tutorial : https://www.hackster.io/rayburne/nodemcu-esp8266-remote-temperature-using-websockets-5956c4 Posted at 2018-01-23 by ClearMemory041063 Problem identifiedthe res.on('drain') is slow to trigger. The following code was executed on a Pico connected via serial port to an Esp8266 at 115 Kbaud. It uses a setTimeout function to issue emit commands at a higher rate. It doesn't test for the amount of data in the drain buffer and thus is an incomplete fix.
@gfwilliams any comments on the hypothesis? @Polypod please give it a try on your Esp8266 with flashed Espruino and see if you can get similar results. I just saw your latest post about the Arduino using Websockets and not HTML. My limited experience with websockets left me with the impression that they are much faster than using HTML. Posted at 2018-01-23 by @gfwilliams I'd be very wary about comparing Espruino running on the ESP8266 and running on something else connected to an ESP8266 by AT commands.
So it'll be very marginally faster if you keep the buffer full all the time. I'd be pretty sure your issue is https://github.com/espruino/Espruino/blob/09412d6a02830abeee16f9582c8c4c7cf0ea4df9/libs/network/esp8266/network_esp8266.c#L763 the chunk size is 268 bytes, so it's only sending that amount each time - which probably results in a whole bunch of small TCP/IP packets being sent, rather than a few large ones. Even so, 5 secs is particularly slow. You could take a look at Posted at 2018-01-23 by Polypod I hope we can raise this upload speed limit I am reaching.
Posted at 2018-01-23 by ClearMemory041063 Try using websockets Webserver method
Posted at 2018-01-24 by Polypod Thanks @ClearMemory041063 for giving attention to my problem, and for the codes to try. You are not testing them on an esp8266, are you ? The last code makes 11,5kb minified, it does not fit, it interrupt the transfer from the IDE. So I had to remove the connecting code and reduce the data to send to 3kb. I cannot send more at once without getting memory errors. Unfortunately, at the end, the result with ws is as slow as with http: 2.5sec for 3kb (see the code below). Maybe because ws server is built on top of an http server.
Posted at 2018-01-24 by Wilberforce Hi - i tried the code above on the linux version of espruino. In the browser - I get an empty page, and only one line of output in the console:
How are you getting mutiple lines output? Posted at 2018-01-24 by Polypod The other lines comes from the interval every 100ms looking and printing how long is the dSnd buffer, and I see it decreasing in my output. Strange that the interval does not seem to start at all in your setup. Posted at 2018-01-25 by ClearMemory041063 Running on Sparkfun Thing -dev ESP8266 boardThe HTML code splits data into lines in this one.
Sending 2 kbytes of data 4 times
8192 bytes sent Posted at 2018-01-25 by Polypod Hmm, that's interesting, it seems hardware related. Your Sparkfun Thing dev-board rocks with espruino !
Posted at 2018-01-25 by ClearMemory041063 Sparkfun ThingFlashing SoftwareI'm using a Windows 10 platform. I'm not sure if it matters but I first installed the latest AT software on the thing. For the Thing I used the code in the "espruino_1v95_esp8266" folder The ExpressIF flashing software uses a file named To configure the download.
For the Thing Espruino download I used
Attachments: Posted at 2018-01-25 by ClearMemory041063 Trying it with a different ESP8266 chip like the ones for a Pico shimFlashed Espruino on to the ESP8266 Running the previous test progrem gives:
8192 / 0.231 = 35,463 bytes per second The tool_config.txt
Crystal freq 26.2 Posted at 2018-01-28 by Polypod So you're a lucky guy, it's working fast for you for 2 different esp8266 based boards..
Posted at 2018-01-28 by ClearMemory041063 Since you are using a NodeMCU the flashing map is described here: I'm testing on a ESP8266-01 with 1MB of flash, I think the NodeMCU has 4MB. Once the page is online you might try from a command prompt ping IP address
Here is some output:
Some variation in the speed
Posted at 2018-01-28 by Wilberforce What is your physical distance to the router? Is it possible the antenna on the board is damaged? Try restarting your router. Have you tried the arduino software recently? Perhaps the hardware is damaged? Posted at 2018-01-28 by @MaBecker What about powesave setting for Wifi, check with Posted at 2018-01-30 by Polypod @wilberforce, I started this conversation after seeing that on arduino my boards were doing ~100kb/sec, while on espruino it was only 1kb/sec, for server a file. In fact, I am using espruino since the v1.91, and all that time I thought it was a hardware limitation of esp8266. But it seems to me that the problem comes from the dSnd buffer sent slowly, every 200m, in small pieces of 256bytes. As it works faster on arduino, the board is not defective. It works the same way on the 2 nodemcu-e12 v2 & v3 that I have. I restarted the router, and tried AP alone (at 20cm from my laptop), station alone and AP+ST, still the same damn result ! @MaBecker, I found it at ps-poll even though it was in AP+ST mode. No change when I set it to "none" using with Wifi.setConfig({powersave:"none"}). @ClearMemory041063, I tried without improvement also flashing the last expressif firmware with the expressif flashing tool like you did, just in case. I used "esp_init_data_default_v05.bin", because there was no "esp_init_data_default.bin". I could not get it working without also changing addresses of this file and of "blank.bin" to 0x3fc000 and 0x37e000. I got a nice :
Then I reflashed espruino, this time with expressif flashing tool. I used the files from this url. I used the addresses from your link and I tried with "blank .bin" at 0x3FE000 like I used to do, both works same way. Still as slow as before. I tried other browsers too, older espruino versions (v1.91, v1.94). The strangest that no one else ever have or noticed this problem before.
Posted at 2018-01-31 by ClearMemory041063 Finaly got a 4MB ESP8266-12 (AI-Thinker ESP8266 MOD) assembled and working.I used 10k pullup resistors on the reset, Enable and GPIO 0 pins and a 10k pulldown resistor on GPIO 15.
Then I did a reset of the chip and ran EraseFlash.bat. Try againPut the batch files into the following folder
661,504 * 8 / 48.125 = 109,964 Bits per second Attachments: Posted at 2018-01-31 by Polypod Thanks, very handy the .bat files. I followed the procedure with no change in speed. I placed the files in the "espruino_1v95_esp8266" directory. I ran info, erase and flash batch files.
I replugged and reflashed the WSdesp8266C. It does not go faster than before.
Attachments: |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Posted at 2018-01-20 by Polypod
Hi all,
I am working on a webserver using espruino and the esp8266, to store sensor data in the flash, and serve it to a web browser. But the network seems very slow, e.g. about 1kb per second, depending on the size of each write, much slower than when using Arduino. In this example, it takes 15second to serve 20kb :
I tried with Arduino, it was 100x faster, and served 1.5Mb in the same 15 seconds. So it cannot be a hardware limitation. Is there a workaround or another way to send data faster with espruino ?
Beta Was this translation helpful? Give feedback.
All reactions