The deferred execution request is not being processed

[butaikis]

I previously used this library from the old repository. I decided to upgrade, and after the upgrade, I started having errors when requesting my API. The fact is that the last request for which the error was delayed, I'll explain:

1. I received a request for ESP32
2. I am reading json, I have sent commands to a third-party UART module
3. I am waiting for a response
4. I am giving a response to the server

As far as I understand, as soon as I exit the handler
WebServer.on("/api", HTTP_POST, [this](AsyncWebServerRequest *request) {}, NULL, [this](AsyncWebServerRequest *request, uint8_t *data, size_t len, size_t index, size_t total){});
it immediately closes the connection, how to resolve this issue?

[mathieucarbou]

not a bug but a usage issue - please open a discussion as explained in the issue template - issues are for real bugs. thanks!

[butaikis]

I need to wait for some time while I wait for third-party data and then call request->send(response);

[me-no-dev]

in reality delay is possible, but should never be done, because it halts the server. I think there was a way for response to be put on hold while data for sending is available, which involved calling back a function to fill data in the response and depending on the return, it will either close the connection or call the function back at later time again. https://github.com/ESP32Async/ESPAsyncWebServer?tab=readme-ov-file#respond-with-content-using-a-callback

[butaikis]

in reality delay is possible, but should never be done, because it halts the server. I think there was a way for response to be put on hold while data for sending is available, which involved calling back a function to fill data in the response and depending on the return, it will either close the connection or call the function back at later time again. https://github.com/ESP32Async/ESPAsyncWebServer?tab=readme-ov-file#respond-with-content-using-a-callback

The fact is that I exit the onBody function into the general loop() loop and wait for data from third-party devices (UART or TCP), and only when I have collected all the data do I call request->send(response)

Do I understand correctly that if I use the new ESP32SAsync/ESPAsyncWebServer library, then I can't do this?

[me-no-dev]

can you please provide a minimal code to reproduce this. I am not sure that we all understand well what you are exactly doing. onBody is for when data is sent to the ESP, not when the ESP will be sending data back

[mathieucarbou]

Do I understand correctly that if I use the new ESP32SAsync/ESPAsyncWebServer library, then I can't do this?

you should not do that from a normal http endpoint request-response. That's not designed to be used for this purpose.
You cannot have the request/response object "exit" the scope of the response handler and use them elsewhere.
Their lifecycle is controlled by the library and the only guarantee you have as a user is that they will be valid under the context of these callbacks. Not before, not after, not outside.

To go async, what you need is sse or ws: you could then have your request normally accept the POST data, return 200 OK, and when your application process is complete, you send the response back with SSE (or WebSocket) to the client.

Have a look in the project in the examples: you have all the use cases inside ready for copy paster ;-)

[butaikis]

@me-no-dev Here is my handler of requests to ESP via /api POST. It works correctly in your version of ESPAsyncWebServer.

WebServer.on(
      "/api", HTTP_POST, [this](AsyncWebServerRequest *request)
      { debug("onRequest"); },
      [this](AsyncWebServerRequest *request, const String &filename, size_t index, uint8_t *data, size_t len, bool final)
      {
        debug("onUpload");
      },
      [this](AsyncWebServerRequest *request, uint8_t *data, size_t len, size_t index, size_t total)
      {
        debug("HTTP Req /api");
        HTTP_SERVER_JSON_BUFFER.concat(data, len);

        debug("len: " + (String)len + " index: " + (String)index + " total: " + (String)total);

        if (index + len == total)
        {
         //set timeout 10s
          request->client()->setRxTimeout(10);
          //save pointer request
          HTTP_Request = request;
          // err Deserializ
          DeserializationError derr;
          if (derr = deserializeJson(INPUT_JSON, HTTP_SERVER_JSON_BUFFER))
          {
            API_Send_Answer("error");
          }
          else
          {
            API_Read_JSON();
           //next, exit onBody
          }
        }
      });

It reads large JSON and then it is processed by functions. They form requests to my UART module and exit onBody. I am waiting for responses from UART devices in loop()
Then, when I collected all the necessary data, I sent the response to the request /api POST as a separate function.:

void Core::WebServer_Response(JsonDocument &OUT_JSON, AsyncWebServerRequest *request)
{
  //create response, set size 61kb 
  AsyncResponseStream  *response = request->beginResponseStream("application/json", 61440);
  serializeJson(OUT_JSON, *response);
  request->send(response);
  debug("HTTP Res send!");
}

[butaikis]

@mathieucarbou I was able to fix the problem, it was necessary to return 0; at the end.
But another problem, serializeJson() does not know how to send in parts, it is impossible to shift to the desired data segment.

[mathieucarbou]

I am in the process of reworking the project examples in PR #38.

I added an example specifically for you ;-)

See: https://github.com/ESP32Async/ESPAsyncWebServer/blob/main/examples/ChunkRetryResponse/ChunkRetryResponse.ino

Tested and it works fine!

[butaikis]

@mathieucarbou Thanks a lot, I'll study it)
Tell me, is it possible to adjust the buffer uint8_t *buffer size in request->beginChunkedResponse() ?

[mathieucarbou]

@mathieucarbou Thanks a lot, I'll study it) Tell me, is it possible to adjust the buffer uint8_t *buffer size in request->beginChunkedResponse() ?

no because that is the maximum len available in the current client pcb space to be sent over the network. This should be named maxLen. Here is an example of code that you can look:

    AsyncWebServerResponse *response = request->beginChunkedResponse("text/html", [](uint8_t *buffer, size_t maxLen, size_t index) -> size_t {
      Serial.printf("%u / %u\n", index, htmlContentLength);

      // finished ?
      if (htmlContentLength <= index) {
        Serial.println("finished");
        return 0;
      }

      // serve a maximum of 256 or maxLen bytes of the remaining content
      // this small number is specifically chosen to demonstrate the chunking
      // DO NOT USE SUCH SMALL NUMBER IN PRODUCTION
      // Reducing the chunk size will increase the response time, thus reducing the server's capacity in processing concurrent requests
      const int chunkSize = min((size_t)256, min(maxLen, htmlContentLength - index));
      Serial.printf("sending: %u\n", chunkSize);

      memcpy(buffer, htmlContent + index, chunkSize);

      return chunkSize;
    });

(from: https://github.com/ESP32Async/ESPAsyncWebServer/blob/main/examples/ChunkResponse/ChunkResponse.ino)

[butaikis]

@mathieucarbou Thank you very much) I hope it will work

[mathieucarbou]

Added a new card in our board to have a look. I my memory is right about how this works, we should be able to support request continuation by providing a request.pause() function to pause a request and then continuation would be done by simply call request.send(...)

ESPAsyncWebServer (view)

[butaikis]

@mathieucarbou thank you very much)

[butaikis]

@mathieucarbou Hello. Will RequestContinuation and ChunkResponse work correctly? Because at the moment I'm getting broken data.

[mathieucarbou]

I have tested them yes, there are 3 examples for those.
Are you pointing to the main branch ?
Would you be able to share your new code ?

[butaikis]

@mathieucarbou

if (auto _request = request->lock())
  {
    AsyncWebServerResponse *response = _request->beginChunkedResponse("application/json", [this](uint8_t *buffer, size_t maxLen, size_t index) -> size_t
                                                                      {
                                                                          if (!index)
                                                                          {
                                                                            CORE.json_remaining_size = serializeJson(OUTPUT_JSON, HTTP_SERVER_JSON_BUFFER);
                                                                            HTTP_SERVER_JSON_BUFFER.clear();
                                                                            HTTP_SERVER_JSON_BUFFER.reserve(CORE.json_remaining_size);
                                                                            debug("json size: " + (String) CORE.json_remaining_size);
                                                                          }
                                                                          if (!CORE.json_remaining_size)
                                                                          {
                                                                            debug("End chunked!");
                                                                            return 0;
                                                                          }
                                                                          size_t s_read = (CORE.json_remaining_size > maxLen) ? maxLen : CORE.json_remaining_size;
                                                                          debug("Chunk send " + (String) s_read);
                                                                          HTTP_SERVER_JSON_BUFFER.getBytes(buffer, s_read + 1, index);
                                                                          CORE.json_remaining_size -= s_read;
                                                                          return s_read;
                                                                      });
    _request->send(response);
  }

[mathieucarbou]

how wait ! either you need to use request continuation, or a "slow" chunk callback waiting for the data.

request continuation will work fine with example you gave at start, here: #34 (comment)

where you keep the request and reuse it after.

You can have a look at the example here: https://github.com/ESP32Async/ESPAsyncWebServer/blob/main/examples/RequestContinuation/RequestContinuation.ino