Serving large files

[diten]

I need to serve a couple of big files (materialize.min.css and materialize.min.js) which are about 150kb each.
Which, I know, is A LOT for a uC (even a ESP32)
I'm using LittleFS,

I thought that somewhat reading chunks and sending them back to the request would work. but I didn't manage to do it right.

Any suggestions ?
Of course, not using css/js or using something really small is my next step.

Thanks in advance. chers, Diten.

[mhaberler]

maybe consider this? https://github.com/BCsabaEngine/svelteesp32

ready to go for this webserver and PsychicHTTP

maybe also look into https://github.com/theelims/ESP32-sveltekit?tab=readme-ov-file

[diten]

Hi Michael, thanks for your response, I'll have to look into it if I can't solve it, and for the next proyect.
Thanks.

[mathieucarbou]

@diten : there is an exemple in the project already:

https://github.com/mathieucarbou/ESPAsyncWebServer/tree/main/examples/StreamFiles

It shows how to use streams to stream one or several files concatenated through ESPAsyncWebServer without using heap or stack.

Note that streams are used to concatenate: otherwise ESPAsyncWebServer is already able to serve Files without using the stack or heap. Read the docs:

https://github.com/mathieucarbou/ESPAsyncWebServer?tab=readme-ov-file#documentation

[diten]

Hi Mathieu, many thanks for your response.
Yes, I have tried a modified solution based on the StreamFiles,
And also something like this (I already trashed this option)

server.on("/someLargeFile.min.js", HTTP_GET, [](AsyncWebServerRequest *request) {
File tmp = LittleFS.open("/someLargeFile.min.js", "r");
request->send(tmp, "text/html", tmp.available());
tmp.close();
}

This only send 4kb back to the browser, and then it restarts the ESP32.
I don't think that splitting a 150kb file to 4kb chunks (for reading each file and then sending it back or StreamConcat them) is a good idea.

Also, I didn't know how (and if is mandatory) to create a StreamString with the File opened.

[diten]

What is somewhat working, I'm not sure if is the best solution.

Reading your links @mathieucarbou , I saw a interesting function:

server.serveStatic("/", LittleFS, "/").setDefaultFile("index.html");;

This, worked perfectly for my needs.

There is (I hope) a small thing, that what it should be registered with server callbacks, they are First looked into flash, not found, and then they are found on the server callback list and processed.
I think that there should be a way to prioritize first the registered handlers, they should be called first, and in the case there is no registered handler, a look in the flash should be done. Cause, I'm not sure, but I think that a lookup in the handlers would be faster. but, I didn't dig deep and I'm not savvy enough in C/C++ (if LittleFS has a file list cached in ram, it's probably the same).

[mathieucarbou]

Yes this is the way to go for static files and what I was about to suggest but you found it faster ;-)
Note that you should not use serveStatic on virtual endpoints: serveStatic should be used with real files behind because it will add a Etag / Since cache header to the response so that the browser does not always request the file again.

server.serveStatic("/someLargeFile.js", LittleFS, "/someLargeFile.min.js");
server.serveStatic("/index.html", LittleFS, "/index.html");

The beauty of serveStatic is that it will automatically first try to find a someLargeFile.min.js.gz to serve, and if not available, will fallback to someLargeFile.min.js.

I encourage you to pre-gzip the files and server them gzipped to reduce the length.

[diten]

Ahh, wow, great, I didn't know that I could point directly to every file on serveStatic, I saw a general like "/" (root) and it serves all.
I have a small amount of files to serve, so this will be even better.

Yes, I kind of thought that serveStatic would work like an simple apache server that serves real files, anyway, thanks for the advice.
Also, very good advice on the .gziping the files.

Again, many thanks.

[vortigont]

this is how I use to serve static files from LittleFS

    server.serveStatic("/", LittleFS, "/")
        .setDefaultFile("index.html")
        .setCacheControl("max-age=180, must-revalidate");  // 180 second for caching, then revalidate based on etag/IMS headers

With this construct your browser will only fetch static files once. Then it will only recheck if file has changed.
You can play with max-age depending on how often you update and flash your files.

[proddy]

That's interesting. I've played around with different strategies for caching and still not 100% satisfied for my use case, which is serving HTML code from flash memory (not the file system). But in principle it's the same approach.

I went from something in the beginning like a simple addHeader("Cache-Control", "public, immutable, max-age=31536000") to now adding a "last_modified" date when the flash files were generated and including a hash. I'm not sure though if I'm overcooking it?

if (request->header("If-Modified-Since").indexOf(last_modified) > 0) {
  return request->send(304);
} else if (request->header("If-None-Match").equals(hash)) {
  return request->send(304);
}

AsyncWebServerResponse * response = request->beginResponse_P(200, contentType, content, len);

response->addHeader("Content-Encoding", "gzip");
response->addHeader("Cache-Control", "must-revalidate"); // ensure that a client will check the server for a change
response->addHeader("Last-Modified", last_modified);
response->addHeader("ETag", hash);

request->send(response);

[vortigont]

if you serve from RO data, like const char*, then you do not have a proper date and the content won't change. You should rely on E-TAG only.

Cache-Control: max-age=86400, must-revalidate
ETag: "12345"

Using Last-Modified makes sense when you serve files from FS that could be changed during operation. I.e. I use this to serve .json files that could be updated in-place by a project run-time code.

[proddy]

I do regular updates of the firmware (with embedded HTML) and have to keep reminding users to F5/Refresh the browser to see the latest Web updates. Adding a date which is set a build time fixes that.