Replies: 37 comments 13 replies
-
|
Code for websocket client: |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Sorry I misunderstood the problem. Here the stack shows it crashes when using the heap metrics. So this is not related to the library. |
Beta Was this translation helpful? Give feedback.
-
|
I didn't explained the String part. In my original code, the From the log of the heap fragmentation, it appears to not fragment, and have > 35k available for the minimal block. Sure, I can't confirm the lib is at fault here. Still the doubt remains because it crashes when doing websocket transfers, and works fine when doing GET/PUT/POST. |
Beta Was this translation helpful? Give feedback.
-
|
Here another stack trace, which I admit has little value too :-( |
Beta Was this translation helpful? Give feedback.
-
|
Another one, with some malloc involved and string concatenation: Could it be a bug in the String library? |
Beta Was this translation helpful? Give feedback.
-
Note that completely removing the heap metrics still makes the esp8266 to crash. |
Beta Was this translation helpful? Give feedback.
-
|
I changed the test case a tiny bit to remove the dependency with string, and just keep malloc: And got a (more) interesting stack trace: Which shows l 112 of AsyncWebSocket.cpp: |
Beta Was this translation helpful? Give feedback.
-
|
Another trace mentioning websocket differently |
Beta Was this translation helpful? Give feedback.
-
No it crashes (first stack trace) when calling There is no tcp transfer involved at the time of the crash. I will have a look at the other traces... |
Beta Was this translation helpful? Give feedback.
-
This one says it crashed during the string concat from main loop. So unrelated to this lib. |
Beta Was this translation helpful? Give feedback.
-
I would suspect your memory instead. The String library is quite (should be) stable |
Beta Was this translation helpful? Give feedback.
-
Here you are not testing the lib, you are just stressing the malloc api. I think you might be able to reproduce the same without the lib... Imagine you have a callback which is executed based on network event or gpio events, doing some free and malloc, and at the same time a loop doing teh same thing without any delay or more cpu work. I just think you are stressing the esp memory allocator... And because you see String or ESpASyncWS traces in the stack, you think it might come from there. |
Beta Was this translation helpful? Give feedback.
-
Well, it is also mentioning Arduino libraries, it is also mentionning String, etc etc. It does not mean the bug is in one of the call stack. It crashes when freeing the memory, several times. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the 2nd pair of eyes :-) I was pointing the lib (websocket in particular) because that is the only remaining thing apart the malloc/free in the main loop ;-) (and I got the first crashes in my project when I added ws connections). All is working fine when only using (and hammering in the same conditions) GET/POST requests. It is definitely something fishy with the memory (as per the various traces and WDT reset sometimes). |
Beta Was this translation helpful? Give feedback.
-
|
Another test: sending a PING ws message each 500 ms: 
Works fine. Also trying each 100 ms: > while true; do echo "PING"; sleep 0.1; done | websocat ws://192.168.4.1/wsAnd it works dine too.
The numbers stay consistent. |
Beta Was this translation helpful? Give feedback.
-
|
Hummm ... Can't reproduce it either with your testcase (with thousands of pings). I just reproduced with "my" original test case (in 37 pings), with the exact same board, and exact same python code. |
Beta Was this translation helpful? Give feedback.
-
|
I made it crash :-)
It /could/ be explained by the code reading some uninitialised RAM, making the behavior different from reset. (but nothing looks like this in the code :-( ) |
Beta Was this translation helpful? Give feedback.
-
Which message queue? |
Beta Was this translation helpful? Give feedback.
-
When you send websocket messages, they go in a queue, which is dequeued depending on the speed of the board / network. This queue can be set to a different value (see the header). |
Beta Was this translation helpful? Give feedback.
-
that's too much for an esp8266. You are just doing a DoS on your board ;-) |
Beta Was this translation helpful? Give feedback.
-
|
I will close this issue. |
Beta Was this translation helpful? Give feedback.
-
I think we misunderstood each other here, I am not DoS'ing the board. I see the queue Moreover, the queue seems guarded by messages like "Too many messages queued: closing connection". I am fine if messages are dropped when the queue is full. But I am not fine with a crash :-( |
Beta Was this translation helpful? Give feedback.
-
|
By the way, it just crashes with 0.5s between WS requests: The float value at the beginning of the line is the time (in s) of the message since the start of program. In this exemple, no |
Beta Was this translation helpful? Give feedback.
-
|
And you run exactly the same sample as the one I uploaded, with your modifications ? Can you test in AP mode ? Right now that's the only difference I see... |
Beta Was this translation helpful? Give feedback.
-
We don't seem to have the same results... For example, if I run with sleep 0.1 (code above) to send a WS message to the esp each 100ms, actually it was slowing down and I was only at 2-3 messages per second. So the reply will never exhaust the queue. It would if we were testing sending messages fast from the loop (which is what I was talking about since ether beging - sorry). But here this is from websocket to the esp (which replies). Sorry! Forget what I said then about that! |
Beta Was this translation helpful? Give feedback.
-
|
This crash was obtained with the exact file from platformio.ini: Using AP mode, my PC connected directly to this AP. |
Beta Was this translation helpful? Give feedback.
-
I have: So same thing.... lol!
same! and what is strange is that you are able to reproduce also with other boards right ? I only have the d1 mini at home... |
Beta Was this translation helpful? Give feedback.
-
|
I reproduced on 2 boards: In both cases a esp8266 anyway. It excludes a board physical issue. I tried a couple of USB cables to eliminate potential power issue, all failing the same. I also tried directly on a power bank, failing the same. |
Beta Was this translation helpful? Give feedback.
-
Hum, that is an interesting difference. In all my experiments, the WS response was always very fast (~1 millisecond), and I never saw a slowdown. |
Beta Was this translation helpful? Give feedback.
-
|
nearly 4 min - no crash
|
Beta Was this translation helpful? Give feedback.
-
|
I experimented with the loop
Using wireshark, packets looks very similar between the 2 clients (the user agent changes, python allows In both cases, there is a single TCP connection, and multiple message transfer. Note: crashes occurs much more often when 2 instances of python client runs in parallel (while it doubles the bandwith, I never observed slow downs.) |
Beta Was this translation helpful? Give feedback.
-
|
What you are doing with the python code is really similar to the bash command, yes. I really don't get it. 2 instances == 2 concurrent connexions so let<S park that one aside for a moment ;-) And this time you have tested with the updated sample in the project, with the rate and the empty loop ^ |
Beta Was this translation helpful? Give feedback.
-
|
All my reports for the last couple of days are with Issue85.ino from 7923d93 (with the "stress" string manipulation in the main loop) I don't have any scientific explanation of my observations (yet :-) ) |
Beta Was this translation helpful? Give feedback.
-
|
Well, your test should be with this version instead: https://github.com/mathieucarbou/ESPAsyncWebServer/blob/main/examples/Issue85/Issue85.ino Like I said at the beginning, if you are stressing the board to fragment the memory while some operations are in progress, whether from async ws or anotehr lib, you'll get an issue at some point. No library on earth will be coded in a way to spend all the free cpu time fragmenting memory... This is considered a bug ;-) |
Beta Was this translation helpful? Give feedback.
-
|
(moving to a discussion until we know what is the cause) |
Beta Was this translation helpful? Give feedback.
-
|
@philfifi : maybe we are looking where we should not.. Didn't you said that you suspected something going on with the string concatenations in the loop() ? for example, if you empty the loop, does it still crash ? |
Beta Was this translation helpful? Give feedback.
-
|
My project used strings (as a class) and string concatenation, and websockets (and other things :-) ) The mininal test case keeps crashing after I removed all the other things. By the way, I got it crashed without the strings, but only with this in the main loop: Without "stress" in the main loop (being string or malloc/free) -> No crash. BTW, I remember having tried standard HTTP requests (instead of WS requests) and I think (not completely sure right now) that HTTP requests didn't exhibit the crash. |
Beta Was this translation helpful? Give feedback.
-
|
oh! so you have answered my question above. the updated example in the project, showing the rates, has an empty loop at the moment. So you say that with an empty loop, you have no crash and can get similar rate output as me then ? But as soon as we introduce some stress in the loop fragmenting the memory, then it crashes ? If you confirm that, then this is not really an issue because no library or code should keep a cpu busy fragmenting memory... This is something everyone should avoid |
Beta Was this translation helpful? Give feedback.
-
|
I confirm, from times where I created the minimal test case: no stress in main loop -> no crash (for minutes/hours)
Agreed. Don't get me wrong, my project was not doing this :-). For the background, I observed my app started to crash (~couple of times a day). After loong nights debugging, I managed to pinpoint to when I started to replace char pointers by string classes in my code. In particular in the ws handler, to (potentially) concatenate data in case the WS request is split in several packets. So each call of the WS handler created a new string. My app originally exchanged 1 WS request per second (with ~20 bytes). So this test case puts all together in order to accelerate the crash to less than a minute (in most cases). Moreover, the fragmentation is measured, and show no significant fragmentation. Biggest block is ~40KB, far enough to fill the dozen of bytes requested by the string or malloc/free. In both cases, objects are destroyed on the next iteration of the loop, so they don't put pressure on the complexity of fitting into the memory. They should just put high demand on alloc/free in term of frequency. |
Beta Was this translation helpful? Give feedback.
-
yes, totally agree! that's what I was thinking about also when saying fragmenting memory (operation of doing frequent malloc and free). I think the frag and maxFreeBlock do not change because this is always the same memory block that is impacted by the same operation at the end. ESP8266 allocates per blocks of 16 bytes each. Maybe it would be different if you were to allocate a random number of bytes, and, above that, randomising the sequence of malloc / free in order to have random sequences like malloc(A), malloc(B), free(B), malloc(C), free(A), malloc(D), malloc(E), etc. It would mimic a little more random memory allocations / free like what can happens in a real app. Also to make the test a bit more realistic in the loop you should do: where x is a number you can play with. Here, the loop would then simulate a quite heavy task, like a mqtt task publishing random content at a high frequency. But whatever you test, I really think that introducing a delay will help! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Description
Bonjour Mathieu,
The ESP8266 is crashing in various ways when using websocket (and some string manipulation).
The crash could be a WDT reset like this one
Or stack trace
The conditions are:
Observations:
Affects:
Board
wemos D1 (esp8266) -> crash
esp32-C3 (super mini) -> works
Stack trace
It varies, here is an exemple:
Additional notes
While it affects all the forks tested, I opened the ticket just here, thanks Mathieu for maintaining this fork :-)
The reproducer is very minimal, plenty of heap is available:
Yes, I should not use ESP8266 anymore :-) Still, it looks like a software bug, triggered by the string manipulation and the websocket together, hence this report. I didn't managed to pinpoint further, lacking debugger connection with ESP8266 :-(
Reproducer: main.cpp
Versions used:
Beta Was this translation helpful? Give feedback.
All reactions