Creating a bytearray in Flash memory

[Rotaclick]

Hi, I am currently having problems with creating a bytearray and "unable to allocate memory error"

I am using an ESP32 Lolin lite with 4Mb of Flash memory - I have a 27Kb of HTML stored in the external flash memomry at 0x250000 which I can read back using the esp.flash_read() function without any problems and view the webpage on another device when esp32 is set up as an access point.

My problem comes when I incorporate this in my main script - I then run into problems with not being able to allocate the 27Kb due to not enough memory - although I have enough in the external flash, but guessing when I create a bytearray it doesn't get created in the external flash?

Is there a way of creating the bytearray in the flash - for example at address 0x300000 which I know is available and not being used?

'def web_page():
data = bytearray(27440)
page = esp.flash_read(0x250000, data)
page = data.decode('utf-8')
#print(page)
return page'

Many thanks

[dlech]

Flash memory is not the same as RAM. Since the web page is already in flash memory, it doesn't seem like putting another copy there would be very useful.

If web_page() is being called more than once, you could cache the result so you don't have to allocate more memory each time it is called.

_cached_web_page = None

def web_page():
    global _cached_web_page

    if _cached_web_page is None:
        data = bytearray(27440)
        esp.flash_read(0x250000, data)
        _cached_web_page = data.decode('utf-8')

    return _cached_web_page

Or, if you don't actually need the entire web page in RAM at one time, e.g. you are writing it to a network stream, you could read from flash in smaller chunks.

[Rotaclick]

Hi, thanks for your reply.

I agree that it seems a waste generating the html data again as it is already on the flash. The issue I have is that when I need to send the html data as a response to a socket connection I believe the only was I can do it is to generate a byte array with the html in and send that… unfortunately when trying to generate the byte array Micropython can’t allocate enough memory, that’s why I was hoping to create the byte area in the external flash where I have room.

[dlech]

If the socket response accepts bytes as data, perhaps you could just pass a memory view of the flash memory instead of copying it to RAM (in a bytearray)? e.g. using uctypes.bytes_at() if it is available on your board (assuming the flash memory is memory mapped and not external flash).

[Rotaclick]

Hi, thanks for your response. I’ve tried using uctypes.bytes_at() but unfortunately when trying to read the flash it causes ESP to reboot… so I’ve given up with reading from the flash and will mpy-cross compile my main script in a bid to memory and hope I can get it to run that way - again thanks for your help

[bixb922]

As @dlech said, read the 27kb in smaller segments of, for example 1kb, and send those 1kb segments. That way you will need at most 1kb of RAM and it will at most be a bit slower.

[Rotaclick]

Hi thanks for reply… I do like the idea of breaking it up into 1Kb chunks and sending it that way but I couldn’t find anything on the internet about sending chunks of html via the urequest module.

[bixb922]

If I understand correctly, the ESP32 Lolin is sending the page. In that case, you could use a web server such as microdot or tinyweb, both have the capability to send data in chunks to avoid allocating a large buffer. Are you also receiving the html page on another ESP32?

[Rotaclick]

Hi, yes esp is sending webpage to a mobile device which could be iPhone or android etc… I will have a look at some of your suggestions for the web servers; thanks for your input.

[bixb922]

https://github.com/belyalov/tinyweb

https://github.com/miguelgrinberg/microdot