socket / HTTP Server: How to answer 1 request with multiple packets (to save RAM)

[luxlux]

I have build a specialized ESP32 Datalogger with an http server to get the data out of the logger.
I use

import network
import socket
import uasyncio 

for the server and start it with
uasyncio.create_task(uasyncio.start_server(serve_client, "0.0.0.0", 8080))
When a http request comes in I send my data with something like

    responseContent = "some html code .... this is my response with some data .... "
    headers = 'Content-type: text/html\r\n\r\n'
    writer.write('HTTP/1.0 200 OK\r\n')
    writer.write(headers)
    writer.write(responseContent)
    await writer.drain()
    await writer.wait_closed()

This works so far.

I log the data into a ringbuffer for which i use a bytearray.
If i request the data via HTTP, and a lot of data was accumulated, i run into a shortage of ram, before my response is complete and was send.

This is how i want to solve it:
Usually one HTTP request can be answered with a sequence of multiple packets.
Like so:
Produce only the first chunk of data for the response -> send the packet -> garbage collect the memory ->
produce the next chunk -> send it out ...
and so on. Doing it this way, i would not need a lot of ram to answer the whole request, instead i need only the ram for one of multiple packets.
But how can i send only a chunk of data, so the HTTP request will not finshed and i'm able to send the next chunk to the still open request?

I know, there would be also other solutions possible, like answer the request only with a small page, with some data and wait for a new request to get the next page of data. but this is not what my question is about!

[DavesCodeMusings]

I had a similar out of memory situation when reading large files as HTTP replies. Here's how I solved it...

First, create a function that returns the file contents in small chunks rather than all at once.

def read_file_chunk(file):
    """
    Given a file handle, read the file in small chunks to avoid large buffer requirements.

    Args:
        file (object): the file handle returned by open()

    Returns:
        bytes: a chunk of the file until the file ends, then nothing
    """
    while True:
        chunk = file.read(512)  # small chunks to avoid out of memory errors
        if chunk:
            yield chunk
        else:  # empty chunk means end of the file
            return

Then, call that function repeatedly, draining between each write.

with open(file_path + '.gzip', 'rb') as file:
    for chunk in read_file_chunk(file):
        writer.write(chunk)
        await writer.drain()

The HTTP client connection stays open until the entire file is sent. By having the "chunk" in a separate function, the memory allocated to the chunk variable gets deleted each time and flagged for garbage collection. This keeps RAM usage low.

You can probably adapt this code to send out your buffer contents instead of a file.

This is part of a larger project. For the whole thing, see: https://github.com/DavesCodeMusings/thimble/blob/main/thimble.py

[luxlux]

Many thanks, this helps a lot!

Before i could use only a buffer of about 3.000 measurement data sets, now a buffer of 12.000 sets is possible.

I have done it like this

# generator function for chunks of a bytearray 
def bytearrayChunk(bytearr,size):
    count = 0
    while True:
        start = count * size
        chunk = bytearr[start:start+size]  # small chunks to avoid out of memory errors
        count = count + 1
        if chunk:
            yield chunk
        else:  # empty chunk means end of the file
            return

the response part:

        startline = 'HTTP/1.0 200 OK\r\n'
        headers = 'Content-type: application/octet-stream\r\n\r\n'
        firstPacket = True
        for chunk in bytearrayChunk(ringbuffer,1028): 
            if firstPack: writer.write(startline + headers)
            firstPacket = False
            writer.write(chunk)
            await writer.drain()
        await writer.wait_closed()

With tests i found out, only the first packet should contain the HTTP startline and HTTP headers. the following packets are pure data. like defined in RFC 2616 a chunked answer to a request will be finished (and the started download ends) if a zero length chunk is received. unfortunately i don't know how to send a zero length chunk. I have checked implementation of writer.drain(). it will only send something if len(buf) > 0 ...
Anyhow with closing the socket await writer.wait_closed() the request will also be satisfied and the download ends. :)

[DavesCodeMusings]

Glad to hear it's helping. Have you thought about a microSD to store your data? I've recently started experiment with them on the ESP32. It's a pretty easy SPI wire-up and a few lines of code in boot.py to get it mounted.

[luxlux]

yes, something like this would be an other way. but ONLY storing data somewere for analyzing later is not what i needed in my case. i still have here some bought temperature & humidity datalogger USB sticks. But i wanted a solution, to see the logged data live to be able to understand how events, which are not logged, will influence the temperature. so i can add comments to the logged data. Only in correlation with the comments interpreting the data is easy.

In fact I have build this logger to debug my new fridge. ;-)

I have bought a highly energy efficient new fridge & freezer combination from Haier. Unfortunately something is wrong with it, the temperature inside, especially in the upper area is too high. now i have build a logger with 5 onewire ds18x20 sensors, which measures the temperature in the different areas in the fridge. if i open the door and put something on the 2. compartment, i can make a note together with the logs. and i see live, how the temperature is rising in this compartment in the next minutes. when the temp is high enough, the fridge steps in and will bring cool air into this area, until the temp falls again. but after a cooling cycle, the temp rises again, on this compartment more the in a regular cooling cycle. This is because the new item have still stored some heat. i can also see that other areas will not change that much. after some cooling cycles and possibly an half hour later, the added item in the fridge, has reached the temp of the compartment, and will not cause a raise of temp in the compartment anymore. This is something you can only find out with a lot of different data logging, or much more easy with live data ... :)

i can add notes of the energy consumption at a moment. (no, i have not build an energy logger up to now.)

if i, on the other side, check the data on my UBS datalogger sticks 2 days later, when taking them out of the fright, i don't really remember such details. so the benefit of data which is stored some were for later is not the same ...

after logging now for some days, i guess i found the source of the problem. the heater, which should defrost the invisible cooling element regularly, seems to be defective. so a lot of humidity is condensing and freezing on it. if it gets more and more, the fan wich is blowing air through this cooling grid, will not be successful. so cool air will not arrive, were it is needed. so the temperature in the fridge is getting higher and higher, while the energy consumption also gets higher and higher, because it always tries to cool more. But that makes the problem even bigger ...