streampy/microdot --- MemoryError: memory allocation failed for file more than 5KB

[rarefeen]

Before I start, I have enough space left in the ESP32. here is a mem_info()

stack: 704 out of 15360
GC: total: 108032, used: 34176, free: 73856
 No. of 1-blocks: 536, 2-blocks: 119, max blk sz: 72, max free sz: 328

Anyone faced issue where loading file more than 4/5KB shows "MemoryError: memory allocation failed"?
What I am doing is, set up a webserver using microdot. then trying to read data. data is coming as form-data as a data field and file. currently
using postman to test.
As it seems, I can read data as big as 4-5KB but more than that it shows me memory error. I already incresed content length, body length in Request class too.
Here is a snippet of my code and error:

from microdot_asyncio import Microdot, Request
Request.max_content_length = 1 * 500 * 1024  # 1MB requests allowed
Request.max_body_length = 1 * 250 * 1024
app = Microdot()
@app.route('/')
async def hello(request):
    print("checked")
    return 'Hello, world!'
@app.post('/test')
async def test(request):
    print(request.headers)
    print(request.body)
    return {'hello': 'world'}
def start_server():
    try:
        print("connecting")
        app.run(port=80,debug=True)
        print("Connected")
    except Exception as e:
        app.shutdown()
        print(e)
        pass

The error I receive is:

Traceback (most recent call last):
  File "microdot_asyncio.py", line 331, in handle_request
  File "microdot_asyncio.py", line 95, in create
  File "uasyncio/stream.py", line 1, in readexactly
MemoryError: memory allocation failed, allocating 9866 bytes

any advice would be appreciate to get some insight on this.

[sophiedegran]

That look nice..
Request.max_content_length = 1024 * 1024 # 1MB
@app.get ("/") not route

I suggest you to use return send_file ("x.html") you can deal with very big data

and there's a discussion group on Miguel microdot git

[rarefeen]

@sophiedegran I am not trying to return file, instead I am accepting files. I want to receive 100-500KB files and save it in microSD.
@app.get ("/") not route I was using default program from microdot @app.route("/") to test the connection in this scenario. :)

[bixb922]

I understand that you are receiving a file. Memory may be too fragmented. Doing gc.collect() more frequently may alleviate that. On the other hand, have you looked at the max_body_length parameter of the Request class? Setting that to 0 lets receive the file as a Stream object, which is much less memory intensive. You process the file as it is received.

[rarefeen]

@bixb922 i was using Request.max_body_length = 1 * 250 * 1024 for max body length. I didn't think about making it zero. let me try it.
Also thank you for the advice on doing more gc.collect. I will definitely test it. :)

[sophiedegran]

Using uploads_async.py example work perfectly with big files (5Mb), sending the upload to sd card... if not for you try with small ftp server

[rarefeen]

@sophiedegran do you have any example?
I mean using the script I am using or similar? on the webserver I am sending data to ('/test) route from postman, using form-data.

[sophiedegran]

Don't understand what you want to do.. if you want to upload file with microdot on your server (board), look at the example uploads_async.py for the server and the index.html for the client..

[rarefeen]

@sophiedegran , sorry if i was not clear. I am not trying to upload file using microdot. as I said in main post, I made webserver using microdot. and from my client, I am sending files to server which is esp32 using microdot. and server should receive the file/data, and read it.
for example think that my client is sending txt file which is 500KB. but server [esp32 microdot server] cannot read the file, it shows memory error. so esp32 is receiving the file.
how the file is coming? using POST method and body is form-data. I used POSTMAN as example as it is easier to understand

[karfas]

@rarefeen: You find the example for uploading larger files to microdot in the microdot repository uploads_async.py.

[sophiedegran]

Ok try Miguel upload example as i tell you and look if you've got same error. Reduce the file size still it work ... after that to have more memory it's very simple declare microdot files in manifest.py and compil micropython..

[rarefeen]

@sophiedegran but I am not uploading any files from esp32. So it doesn't make sense to do upload example, or maybe I am not understanding what you meant.
instead you could say I am downloading or reading or the file in esp32. in simpler word, I am sending 500KB of file to ESP32 from client, and trying to read it or parse the data then save it in SD card, for simplicity sake, let's just say I want to read 500kb of txt file on ESP32 which is coming from client to ESP32 microdot server. [reading it just an example, the file could be image/audio/or any type].

[bixb922]

I hope you have already solved your problem. If not, just to clarify; “upload a file“ normally means exactly what you are describing, that is, to send a file from the client (for example a PC) to a server (in this case the ESP32), so the upload_asyncio.py example mentioned by @sophiedegran is relevant. It shows how the received file is split in smaller chunks of 1024 bytes to lessen the need for large memory allocations.

[rarefeen]

@bixb922 I have not resolved the issue. I was away for couple of days. I see, in this case let me check it. I got confused with "upload file with microdot ". I thought @sophiedegran was talking about sending file using microdot, which I didn't wanted.
It shows how the received file is split in smaller chunks of 1024 bytes to lessen the need for large memory allocations. it makes more sense, thank you. I will try it.