ESP32 memory allocation error - strange difference between chips

[eljots]

Hi all!

I have a program running on the ESP-WROOM-32 module and I am trying to run it on the next, identical module but when I try to open the same text file I get an error:

Traceback (most recent call last):
File "boot.py", line 77, in
File "astro.py", line 543, in
File "astro.py", line 188, in selection
MemoryError: memory allocation failed, allocating 35656 bytes

I'm trying to find the differences between modules by checking the amount of memory:
183 .............
184 print(gc.mem_free())
185 file=open(fil,"r")
186 print(gc.mem_free())
187 whileTrue:
188 line = file.readline()
189 ..........

The allocation error also appears if I try to open the file manually in console and try to read a line from the file with the command
line = file.readline()

The memory measurement results from lines 184 and 186 for both modules show insignificant differences: 184 -> old 80512, new 80384; 186 -> old 80256, new 80128.

Both modules have the same upython image, I tried with the other two available recently but with no improvement. The set of executable files and text files for reading is identical, I even have a few more test files on the working module. The opened file (fil) is text and has a size of approximately 120k. When I significantly reduce the number of lines from 3000 to 500, the program starts working.

Do you have any suggestions where the problem might be?
Maybe the modules should be properly configured in addition to the standard micropython image upload?

eljot

[glenn20]

I cant explain why you are only seeing these memory issues on only one of your devices, but I can explain why you are getting a lot of memory fragmentation. Each call to the readline() method will allocate a new str object to hold the return value. This will lead to high memory fragmentation for large files.

Instead, you can allocate a single bytearray buffer (wrapped in a memoryview object) and reuse that for reading data from the file (with the readinto() method). This is more inconvenient as you need to manage extracting lines from the buffer yourself and you have to deal with bytes rather than str data. However, it can be used to read very large files without memory fragmentation.

[eljots]

glenn20 thank you for the inspiring suggestion but in this case the solution is probably different.

As I mentioned, even trying to manually access the file using the cli commands: file = open ('file.txt', 'r') and line = file.readline() resulted in a memory allocation error for the first operation of reading from the file.

I also noticed earlier, on another occasion, that text files created by .py scripts are later read correctly by other programs, and these files, even a few lines long, that I created externally and sent to the module were not read correctly.
This suggested to me that the likely cause was in the way the file was transferred to the module, and I think I was right.

So far, I have used the native option in upyCraft to transfer files, but I noticed that after re-reading the file from the module, it contains additional lines at the beginning and/or end of the file. This is a bit strange but doesn't matter (I guess) for executable files (.py). Unfortunately, files with the txt extension sent in this way are so deformed that the scripts cannot read them, which ultimately results in memory allocation errors.

As soon as I used cli + ampy, the problem disappeared!

Before closing the topic, I will try to confirm this rule on other modules that I should receive for testing today.

eljot

[eljots]

Hi Everyone!

Finally, I confirmed on several ESP32 copies that the system's behavior towards text files is not influenced by the specific processor, the version of the upython image, or the method of its installation, because I used both esptool and uPyCraft tools for this purpose.
Unless I see a negative impact on how executable files (scripts) are transferred on their performance, to avoid problems I will use ampy for text files.
I have been working with the program that showed a problem with opening .txt files for several months on the same ESP32 and I probably uploaded the text files using ampy at the beginning and forgot about it. Later, for convenience, I used to send .py scripts using uPyCraft, and only when the program was ready to be tested on another chip, it turned out that text files were deformed when sent using uPyCraft.

eljot