Crash on ESP8266 with rapid reloads

[bdraco]

Platform

ESP8266

IDE / Tooling

Arduino (IDE/CLI)

What happened?

ESPAsyncWebServer 3.9.4 has a regression that causes crashes on ESP8266 when using Server-Sent Events (SSE/AsyncEventSource). The crash occurs during rapid page reloads and is reproducible. The problem is not reproducible on 3.7.8.

LoadStoreError exception during SSE entity listing

Reproducible by quickly reloading a page that uses SSE

Without digging in to deep it seems to be a problem with how AsyncEventSourceResponse::_respond and the connection being destroyed before the the onAck callback fires

Stack Trace

ate 2.00000 % with 1 decimals of accuracy

[13:28:09.480]--------------- CUT HERE FOR EXCEPTION DECODER ---------------

[13:28:09.480]Exception (3):
WARNING Exception type: LoadStoreError: Processor internal physical address or data error during load or store
[13:28:09.481]epc1=0x4000228b epc2=0x00000000 epc3=0x00000000 excvaddr=0x402656df depc=0x00000000

[13:28:09.501]>>>stack>>>
WARNING Found stack trace! Trying to decode it

[13:28:09.501]ctx: cont
[13:28:09.501]sp: 3ffffc60 end: 3fffffd0 offset: 0150
[13:28:09.501]3ffffdb0:  3fff4cfd 3ffffe38 3fff4cfc 40206c3d  
WARNING Decoded 0x40206c3d: ArduinoJson::V742PB22::detail::TextFormatter<ArduinoJson::V742PB22::detail::Writer<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, void> >::writeString(char const*, unsigned int)
[13:28:09.501]3ffffdc0:  3fff494c 3fff000d 3ffffe00 00000000  
[13:28:09.501]3ffffdd0:  0000ffff 3fff494c 3ffffe38 40206dc3  
WARNING Decoded 0x40206dc3: _ZNK11ArduinoJson8V742PB226detail11VariantData6acceptINS1_14JsonSerializerINS1_6WriterINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEvEEEEEENT_11result_typeERSE_PKNS1_15ResourceManagerE$isra$0 at json_util.cpp
[13:28:09.501]3ffffde0:  402656bc 00000005 00000000 00000005  
WARNING Decoded 0x402656bc: etharp_output
[13:28:09.501]3ffffdf0:  00000000 3ffffe60 3ffffe60 3fff00e9  
[13:28:09.501]3ffffe00:  40002514 3fffdd3c 00000000 3fff2564  
[13:28:09.501]3ffffe10:  3ffffeb0 00000050 00000008 00000000  
[13:28:09.501]3ffffe20:  00000000 00000000 00000000 402656df  
WARNING Decoded 0x402656df: etharp_output
[13:28:09.501]3ffffe30:  40264fd8 3ffe8f70 3fff4e0c 3ffffeb0  
WARNING Decoded 0x40264fd8: etharp_output
[13:28:09.501]3ffffe40:  3ffffe60 3ffffe60 00000004 40216590  
WARNING Decoded 0x40216590: AsyncEventSourceClient::_queueMessage(std::shared_ptr<String>&&)
[13:28:09.501]3ffffe50:  3fff3894 00000041 3ffffe80 40224360  
WARNING Decoded 0x40224360: String::copy(__FlashStringHelper const*, unsigned int)
[13:28:09.501]3ffffe60:  3fff00dc 402656df 000000e5 67617373  
WARNING Decoded 0x402656df: etharp_output
[13:28:09.501]3ffffe70:  3fff0128 00000004 3fff3894 4021657a  
WARNING Decoded 0x4021657a: AsyncEventSourceClient::_queueMessage(std::shared_ptr<String>&&)
[13:28:09.501]3ffffe80:  3fff00dc 0041004f 80000000 4021663d  
WARNING Decoded 0x4021663d: std::shared_ptr<String> std::make_shared<String, String>(String&&)
[13:28:09.501]3ffffe90:  3ffffeb0 3fffffff 000d0000 3fff3638  
[13:28:09.501]3ffffea0:  3fff2564 00000000 00000000 402166aa  
WARNING Decoded 0x402166aa: AsyncEventSource::send(char const*, char const*, unsigned int, unsigned int)
[13:28:09.501]3ffffeb0:  3fff2e88 3fff2e7c 00000000 3fff2e88  
[13:28:09.501]3ffffec0:  3fff2e7c 3ffffe60 00000010 4020a620  
WARNING Decoded 0x4020a620: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE12_M_constructIPKcEEvT_S8_St20forward_iterator_tag$isra$0 at mqtt_component.cpp
[13:28:09.501]3ffffed0:  3fff3894 00000005 3ffffe58 00000001  
[13:28:09.680]3ffffee0:  4020f068 3fff1314 3fffff00 00000001  
WARNING Decoded 0x4020f068: esphome::web_server::WebServer::text_sensor_all_json_generator[abi:cxx11](esphome::web_server::WebServer*, void*)
[13:28:09.680]3ffffef0:  4020f068 3fff1314 3fff3604 4020eb64  
WARNING Decoded 0x4020f068: esphome::web_server::WebServer::text_sensor_all_json_generator[abi:cxx11](esphome::web_server::WebServer*, void*)
WARNING Decoded 0x4020eb64: esphome::web_server::DeferredUpdateEventSource::deferrable_send_state(void*, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(esphome::web_server::WebServer*, void*))
[13:28:09.680]3fffff00:  3fff4e0c 0000009f 000000f0 40100e78  
WARNING Decoded 0x40100e78: free
[13:28:09.680]3fffff10:  4025e1f2 3fff0624 000306c4 00000001  
WARNING Decoded 0x4025e1f2: wifi_get_opmode
[13:28:09.680]3fffff20:  3ffe8cb7 000306c4 3ffef508 40214b47  
WARNING Decoded 0x40214b47: esphome::Scheduler::call(unsigned int)
[13:28:09.680]3fffff30:  3fff19bc 40211d58 000306c4 3fff07f0  
WARNING Decoded 0x40211d58: esphome::wifi::WiFiComponent::wifi_sta_connect_status_()
[13:28:09.680]3fffff40:  00000002 3fff0830 3fff3664 4020da00  
WARNING Decoded 0x4020da00: esphome::web_server::ListEntitiesIterator::on_text_sensor(esphome::text_sensor::TextSensor*)
[13:28:09.680]3fffff50:  00000000 3fff0830 3fff3604 40213779  
WARNING Decoded 0x40213779: esphome::ComponentIterator::advance()
[13:28:09.680]3fffff60:  00000002 3fffff80 3fff2684 4020e85c  
WARNING Decoded 0x4020e85c: esphome::web_server::DeferredUpdateEventSourceList::loop()
[13:28:09.680]3fffff70:  00000002 000306c4 3ffef508 40212bbf  
WARNING Decoded 0x40212bbf: esphome::Application::loop()
[13:28:09.680]3fffff80:  000306c4 3fff07f0 00000020 30323a33  
[13:28:09.680]3fffff90:  2035343a 3030312d feef0030 3ffef854  
[13:28:09.680]3fffffa0:  3fffdad0 00000000 3ffef828 3ffef854  
[13:28:09.680]3fffffb0:  3fffdad0 00000000 3ffef828 40225310  
WARNING Decoded 0x40225310: loop_wrapper() at core_esp8266_main.cpp
[13:28:09.680]3fffffc0:  feefeffe feefeffe 3fffdab0 4010048d  
WARNING Decoded 0x4010048d: cont_wrapper
[13:28:09.680]<<<stack<<<

[13:28:09.680]--------------- CUT HERE FOR EXCEPTION DECODER ---------------
[13:28:09.853]\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80\x80[I][logger:028]: Log initialized
[13:28:09.853][C][safe_mode:102]: Unsuccessful boot attempts: 0

Minimal Reproductible Example (MRE)

Reproducible in ESPHome by upgrading to 3.9.4 on esp8266

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of ESPAsyncWebServer (and AsyncTCP for ESP32).

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

⚠️ Possible Duplicate Issue

• ISSUE-324

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[mathieucarbou]

Hi @bdraco,

Let's see if @willmmiles or @vortigont (our network experts) can have a look.

But I would like to stress out that we do not support anymore ESP8266.

We do try to have it compile, but we never test it following changes. These boards work differently regarding the network and memory, will go EOL in a few years and this is generally very hard to maintain a code that pleases both.

The only way we would support it back is by having some company-backed developers helping to test and PR fixes for ESP8266.

So for now the solution is to stick to an older version.

I am though surprised that you needed to go as far to an old 3.7.8 version. Latest network layer changes are mostly in 3.9.x.

Did you try with v3.8.1 ?

[bdraco]

I haven't tried 3.8.1 yet. I very much expect the problem started with 3.9.x from the change in AsyncEventSourceResponse::_respond. We are just preparing for ESPHome 2026.1 and I tried to upgrade it just to not fall so behind. Once we get the release to beta, I'll circle back and give 3.8.1 a try.

[mathieucarbou]

I very much expect the problem started with 3.9.x from the change in AsyncEventSourceResponse::_respond

I don't think this is this change that triggered it but probably much more the new way packets are sent with regard to how lwip works on ESP8266.

For AsyncEventSourceResponse, the code was changed from:

void AsyncEventSourceResponse::_respond(AsyncWebServerRequest *request) {
  String out;
  _assembleHead(out, request->version());
  request->client()->write(out.c_str(), _headLength);
  _state = RESPONSE_WAIT_ACK;
}

size_t AsyncEventSourceResponse::_ack(AsyncWebServerRequest *request, size_t len, uint32_t time __attribute__((unused))) {
  if (len) {
    new AsyncEventSourceClient(request, _server);
  }
  return 0;
}

to

void AsyncEventSourceResponse::_respond(AsyncWebServerRequest *request) {
  String out;
  _assembleHead(out, request->version());
  // unbind client's onAck callback from AsyncWebServerRequest's, we will destroy it on next callback and steal the client,
  // can't do it now 'cause now we are in AsyncWebServerRequest::_onAck 's stack actually
  // here we are loosing time on one RTT delay, but with current design we can't get rid of Req/Resp objects other way
  _request = request;
  request->client()->onAck(
    [](void *r, AsyncClient *c, size_t len, uint32_t time) {
      if (len) {
        static_cast<AsyncEventSourceResponse *>(r)->_switchClient();
      }
    },
    this
  );
  request->client()->write(out.c_str(), _headLength);
  _state = RESPONSE_WAIT_ACK;
}

void AsyncEventSourceResponse::_switchClient() {
  // AsyncEventSourceClient c-tor will take the ownership of AsyncTCP's client connection
  new AsyncEventSourceClient(_request, _server);
  // AsyncEventSourceClient c-tor would also delete _request and *this
};

What you need to know also is that before, a callback was setup client()->onAck(cb) and it was calling _ack.
So this is essentially the same code.
Now, the callback is directly setup on the client object within _respond and this callback still executes the same piece of code.

[mathieucarbou]

@bdraco : following my previous message...

Would you be able to either adapt one of the example in the project, or create a new one, so that we can have a MRE to reproduce ? Right now we have no way to troubleshoot anything without a MRE.

Thank you!