Sending large dynamic HTML site

[minou65]

Hi everyone,

I'm dynamically generating a large HTML page with a size of around 65 kB. My current approach looks like this:

this->_responseStream = request->beginResponseStream("text/html", 1024);
_responseStream->print(_content);
...
_responseStream->print(_content);
_request->send(_responseStream);

However, when I increase the buffer to 70 kB, I encounter the following errors:

[E][cbuf.cpp:104] resize(): failed to allocate temporary buffer
[E][webresponses.cpp:848] write(): Failed to allocate

These errors repeat multiple times. Interestingly, the same page works (albeit slowly) when I use chunked responses via the web server.

I’m aware of the chunked response method:

request->beginChunkedResponse("text/html", [_content](uint8_t* buffer, size_t maxLen, size_t index) -> size_t
But this still requires me to build the entire HTML string first and then split it inside the lambda function.

Maybe I’m overcomplicating things, but I’m looking for a way to stream each _responseStream->print(_content); directly to the client in chunks—without needing to preassemble the full HTML string.

Any ideas or suggestions?

[EmileSpecialProducts]

you can see what I have done in this post
get lage sensor data from ESP ( Json)
But I still have the issue that this will only work for one request at a time,
As the CurrentSensorData will be restarted when the second request is started and the older request is not finned yet

[minou65]

I found a solution that works for me. I'm not sure if it's the best one, but it does the job. I write the content for the webpage into a file first. Once the file is complete, I send it to the client.

With this approach, the full content is no longer held in RAM, which helps avoid memory allocation issues when dealing with large pages.

			File file = LittleFS.open("/myhtml.html", "r");
			if (file) {
				AsyncWebServerResponse* response = _request->beginChunkedResponse(
					"text/html",
					[file](uint8_t* buffer, size_t maxLen, size_t index) mutable -> size_t {
						size_t bytesRead = file.read(buffer, maxLen);
						if (bytesRead == 0) {
							file.close();
							LittleFS.remove("/myhtml.html");
						}
						return bytesRead;
					}
				);

				for (const auto& header : _headers) {
					response->addHeader(header.first, header.second);
				}
				_request->send(response);
				_headers.clear();
			}

[robertlipe]

This may not be a concern for your application, but notice that this approach probably isn't safe for concurrency.

Assuming that this code is reasonably lives up to the name and you have two requests for that file a zillisecond apart. Consider you're halfway through building this file (in the code you don't show) and an interrupt happens or another request gets started, or for whatever reason, there's a "task" switch (maybe not a literal thread, but another web request) for this same handler. Now THAT code dutifully opens myhtml.html for writing and starts scribbling on it. Maybe it finishes, and maybe it doesn't before you're called back to the code reading and writing this one. Nothing good will come of that.

Maybe this isn't an issue for your site where you're the only user and you know this file is only served to one well-behaved client and it's only read once every hour on the hour or whatever. A solution like the above is probably fine. Now if this is serving multiple users, at any time multiple people might try to access this; you're going to get grumpy users that are being served doodoo on their request.

You've found something that works well enough for you, and that's super. I'm just sticking a post-it note here for any other intrepid readers that might try to do this: this isn't a great solution if you don't control the readers and know there is exactly one of them at any moment.

Are there security concerns if your SD card gets stolen and the contents of myhtml.html are readable?

Back to the question, though, it is unfortunate that chunking requires the file size to be known in advance so it can tell the receiver how much stuff to expect. If you have an ESP32 without PSRAM, consider if throwing another $2 at the problem so the code doesn't stress about multiple coopies. You might see if the built-in template processor can help, but that's just going to build a second copy behind your back anyway. Double check that you don't have more memory than you hae "turned on" (-DBOARD_HAS_PSRAM or something and maybe a PSRAM_init - both are from memory, so double check), though. If you have 8MB of RAM and your you're tripping for something that's only a few KB, it's worth checking.

Finally, isn't the hunk above basically the librarys's own servStatic() ?

[EmileSpecialProducts]

I have started beginChunkedResponse missing request object discussions
As in basic it is the same issue,
For the upload of a file I can use request->_tempObject to do this way but not for download as I am missing the request object
you can find the portable-Async-disk-driveEx at this Github page.
This is a ESP SD-Card web server that supports FatEx >32Gb

It has the code for download and this works, It relies on the index to determine the requested file. While not completely foolproof, this method is effective in the vast majority of cases.
And is does not use the file size for chunking requires the file size is not known in advance and still works.

The Upload code:

server.on("/edit", MY_HTTP_POST, 
        [](AsyncWebServerRequest *request)
          {
            DBG_OUTPUT_PORT.println("File upload completed " + request->url());
            //request->send(200, "text/plain; chartset=\"UTF-8\"", "File upload completed");
            //request->send(200);
            request->redirect("/"); 
          }, 
        [](AsyncWebServerRequest *request, String filename, size_t index, uint8_t *data, size_t len, bool final)
          {
            typedef struct Uploadf {int count;file_t uploadFile;} Uploadf;
             if (!index) {
              request->_tempObject = malloc(sizeof(Uploadf));
              ((Uploadf *)(request->_tempObject))->count=1 ;
              ((Uploadf *)(request->_tempObject))->uploadFile =sd.open("/"+ filename, O_RDWR  | O_CREAT);
            }
            if (len) {
              ((Uploadf *)(request->_tempObject))->count++;
              //DBG_OUTPUT_PORT.printf("%x Count\n",((Uploadf *)(request->_tempObject))->count);
              ((Uploadf *)(request->_tempObject))->uploadFile.write(data, len); 
            }
            if (final) {
              ((Uploadf *)(request->_tempObject))->uploadFile.close();
              //DBG_OUTPUT_PORT.printf("final Uploaded %s in %d blocks ",filename.c_str(),((Uploadf *)(request->_tempObject))->count);
              free(request->_tempObject); // this free is also done in the AsyncWebServerRequest destructor
              request->_tempObject=NULL;
             }
          }
    );