The server doesn't complete my file uploads that are larger than 2MB

[Shimoe-Koharu16]

Platform

ESP32

IDE / Tooling

Arduino (IDE/CLI)

What happened?

Whenever I try to upload a file that's larger than 2MB, it doesn't continue and leaves an incomplete file data (corrupt file in other words). To be specific, I was trying to upload a file that's 28MB+ on size (ZIP file), but when I'm about to extract the file on my PC or phone (after downloading the file/ZIP folder), it doesn't open with any file explorers I use (Nautilus and Files). The Serial output says that the upload was a success but it doesn't actually finished writing on the SD card. I already added .flush() to rule out RAM limitations (but the average RAM use while upload is ~210kB based on the dashboard on Blynk that I set up on my ESP32 as an RTOS task). But, it still does produce the same bug or problem

Stack Trace

------------------------------------------
  Compile Date/Time : Oct  7 2025 18:40:59
  Compile Host OS   : linux
  ESP-IDF Version   : v5.5.1-1-g129cd0d247
  Arduino Version   : 3.3.0
------------------------------------------
Board Info:
------------------------------------------
  Arduino Board     : ESP32_DEV
  Arduino Variant   : esp32
  Arduino FQBN      : esp32:esp32:esp32:UploadSpeed=921600,CPUFreq=240,FlashFreq=80,FlashMode=qio,FlashSize=4M,PartitionScheme=default,DebugLevel=debug,PSRAM=disabled,LoopCore=1,EventsCore=1,EraseFlash=none,JTAGAdapter=default,ZigbeeMode=default
============ Before Setup End ============
Server Initialized
=========== After Setup Start ============
[  7369][D][NetworkManager.cpp:83] hostByName(): Clearing DNS cache
INTERNAL Memory Info:
------------------------------------------
  Total Size        :   317784 B ( 310.3 KB)
  Free Bytes        :   155816 B ( 152.2 KB)
  Allocated Bytes   :   150304 B ( 146.8 KB)
  Minimum Free Bytes:   155772 B ( 152.1 KB)
  Largest Free Block:    94196 B (  92.0 KB)
------------------------------------------
GPIO Info:
------------------------------------------
  GPIO : BUS_TYPE[bus/unit][chan]
  --------------------------------------  
     1 : UART_TX[0]
     2 : GPIO
     3 : UART_RX[0]
     5 : SD_SS
    17 : GPIO
    18 : SPI_MASTER_SCK[3]
    19 : SPI_MASTER_MISO[3]
    23 : SPI_MASTER_MOSI[3]
============ After Setup End =============
[  7486][D][NetworkManager.cpp:127] hostByName(): DNS found IPv4 128.199.144.129
Upload Start: てるくず.zip
Upload Success: てるくず.zip

Minimal Reproductible Example (MRE)

Here's the upload endpoint for my website to send the files to
  server.on(
    "/upload", HTTP_POST, [](AsyncWebServerRequest *request) {
      request->send(204);
    },
    handleUpload);

And here's my upload handler 

`void handleUpload(AsyncWebServerRequest *request, const String &filename,
                  size_t index, uint8_t *data, size_t len, bool final) {

  if (index == 0) {
    if (!enable_upload) {
      events.send("Upload disabled from the the dashboard", "toast", millis());
      return;
    }
    Serial.printf("Upload Start: %s\n", filename.c_str());
    fn = filename;
    String path = "/uploads/" + filename;
    uploadFile = SD.open(path, FILE_WRITE);
    if (!uploadFile) {
      Serial.println("Failed to open file for writing");
      return;
    }
  }
  if (!enable_upload) {
    events.send("Upload disabled from the the dashboard", "toast", millis());
    return;
  }
  if (uploadFile) {
    if (len > 0)
      uploadFile.write(data, len);
      uploadFile.flush();
      digitalWrite(2, HIGH);
    if (final) {
      uploadFile.close();
      Serial.printf("Upload Success: %s\n", filename.c_str());
    }
    digitalWrite(2, LOW);
  }
}`

I confirm that:

• I have read the documentation.
• I have searched for similar discussions.
• I have searched for similar issues.
• I have looked at the examples.
• I have upgraded to the lasted version of ESPAsyncWebServer (and AsyncTCP for ESP32).

[mathieucarbou]

Hi,
Please open a support ticket first next time in order for the team to have the chance to help you before asserting there is an issue in the library.

[mathieucarbou]

@Shimoe-Koharu16 : here are some comments:

1. is there a reason you use FILE_WRITE instead of FILE_APPEND (currently your writes restarts each time from 0) ?
2. no error checking about written bytes: size_t written = uploadFile.write(data, len); => what if written < len ?
3. how the enable_upload is controlled ? If set to false during an upload operation, it will leave a corrupt file. There is no cleanup phase

[Shimoe-Koharu16]

I tried applying what you asked my on why I used FILE_WRITE instead of FILE_APPEND with this code

void handleUpload(AsyncWebServerRequest *request, const String &filename,
                  size_t index, uint8_t *data, size_t len, bool final) {

  if (index == 0) {
    if (!enable_upload) {
      events.send("Upload disabled from the dashboard", "toast", millis());
      return;
    }
    Serial.printf("Upload Start: %s\n", filename.c_str());
    fn = filename;
    String path = "/uploads/" + filename;

    uploadFile = SD.open(path, FILE_APPEND);
    if (!uploadFile) {
      Serial.println("Failed to open file for writing");
      return;
    }
  }

  if (!enable_upload) {
    events.send("Upload disabled from the dashboard", "toast", millis());
    return;
  }

  if (uploadFile && len > 0) {
    size_t written = uploadFile.write(data, len);
    uploadFile.flush();
    digitalWrite(2, HIGH);

    if (written < len) {
      Serial.printf("Warning: written %u of %u bytes\n", written, len);
    }
  }

  if (final) {
    if (uploadFile) {
      uploadFile.close();
      Serial.printf("Upload Success: %s\n", filename.c_str());
    }
    digitalWrite(2, LOW);
  }
}

But the problem is still there, incomplete file uploads. In comparison:

This is it's original file size

and here's the file size that's from the ESP32 server

[Shimoe-Koharu16]

I could make a repository with my program files so you could help me rule out what's causing this error

[mathieucarbou]

For example, look at the Upload example. There is an upload/file endpoint.

Generate a random text file of 200K and run:

curl -v -F "[email protected]" http://192.168.4.1/upload/file

you should see it being uploaded, then being read to be served back.

Try to use this example, comment out the Littlefs part to avoid serving at the moment to just verify that you have your 20MB uploaded. Then add the SD part, but keep the logs.

Also in the request callback, read the file size.

For example, if I generate a 20MB file and use this endpoint (without saving the file because I do not have an SD) I can see all the callback calls reaching the end:

curl -v -F "[email protected]" http://192.168.4.1/upload/file

ESP32 console shows a lot of calls until... the final one:

Upload[20MB.bin]: start=19990398, len=1436, final=0
Upload[20MB.bin]: start=19991834, len=1436, final=0
Upload[20MB.bin]: start=19993270, len=1436, final=0
Upload[20MB.bin]: start=19994706, len=1436, final=0
Upload[20MB.bin]: start=19996142, len=1436, final=0
Upload[20MB.bin]: start=19997578, len=1436, final=0
Upload[20MB.bin]: start=19999014, len=986, final=1

Also, the maximum buffer size (LWIP) is 1436 bytes, so the callback is called as many times as necessary, in my case nearly 14000 times. So for a 20MB file, this means 14000 writes... That's a lot of a SDCard, which is known to be very slow on ESP32.

With my little esp32dev, I can upload a 20,000,000 bytes file (20MB) in about 50648 ms.
This is about 395 KB / second... So very slow... And your SD Card writing might be even slower....

I think fo that you work you would need to get an ESP32 with a lot of PSRM (16MB) and in the callback, allocate 1MB buffer, that you fill, then, when full, you put the buffer in the queue, and allocate a new one one PSRAM. Then have an async task that is just dequeuing the buffers and write them to disk. If the queue is full (12 elements for example), then the async callback needs to wait before putting the buffer in the queue. This will slow down the upload to let the time for the SD Card to write.

You should probably see that the SDCard operation is slower than the speed of upload.

I am not there to tell you everything: this is up to you to troubleshoot the cause. Not this project's team.

The webserver has its limitations, the file system too.

[mathieucarbou]

Also, this is highly possible that your SDCard write operation are so slow that when executed within the context of the async_tcp callback (in the handler) you are slowing down so much the callback that the watchdog triggers and crashes your ESP and it reboots.

Try to remove the watchdog with: -D CONFIG_ASYNC_TCP_USE_WDT=0

Note: do that just for testing, because it does not remove the cause of error, and might just cause your ESP to deadlock instead

[Shimoe-Koharu16]

Ok, thanks for helping me, I'll try to rebuild my program from scratch with this scenario in mind, because I initially designed it to handle small files (~2MB) to help with my former school's (rural area) IT system for file sharing.

Now I'm trying to expand this for personal use, that lead me to this issue.

So yeah, thank you so much for the guidance :)