Error when trying to open csv file in Microptyhon Raspberry Pi Pico version

[clarsbyte]

So this is my code; I have both my python file and csv file in my raspberry pi pico already as you can see in the image1. However, when I run it, it returns an error.

    #from machine import Pin, I2c
    #from ssd1306 import SSD1306_I2C
    #import framebuff
    import time
    from sys import stdin
    
    w = 128
    h= 32
    
    #oled = I2C(0,scl=Pin(1),)
    
    df = []
    delim = ','
    with open('colordata.csv','r') as file:
        for line in file:
            df.append(line.rstrip('\n').rstrip('\r').split(delim))
    
    def color_recognition(r,g,b):
        color = 'Unknown'
        names =[]
        index = []
        threshold = 50
        for i in range(len(df)):
            R = int(df[i][4])
            G = int(df[i][5])
            B = int(df[i][6])
            diff = abs(r - R) + abs(g -G) + abs(b-B)
            if diff <= threshold:
                index.append(diff)
                names.append(df[i][2])
        
        
        if len(index)>0:
            result = sorted(index)[0]
            for j in range(len(index)):
                if index[j]==result:
                    color = names[j]
        else:
            pass
            
    
        return color

The error:

    Traceback (most recent call last):
      File "<stdin>", line 16, in <module>
    MemoryError: memory allocation failed, allocating 4096 bytes
    PROBLEM IN THONNY'S BACK-END: Exception while handling 'Run' (thonny.plugins.micropython.mp_back.ManagementError: Command output was not empty).
    See Thonny's backend.log for more info.
    You may need to press "Stop/Restart" or hard-reset your MicroPython device and try again.
    
    
    Process ended with exit code 1.

What can I do to fix this calamity?

[sosi-deadeye]

Try to use a generator to save memory. I guess the optimization is better and in this case the problem is shifted to the function min. The function min consumes the generator, and it seems to use lesser memory, then appending new items to a list. I do not know the exact implementation, but I guess there are always kept two values in memory for comparison. The lower value is kept.

The sort function also allocates memory with the same size as the list itself. Maybe a little more.

import time
import micropython
import gc

DEBUG = True


def timeit(function):
    def inner(*args, **kwargs):
        start = time.ticks_ms()
        result = function(*args, **kwargs)
        print(f"{function.__name__} took {time.ticks_diff(time.ticks_ms(), start):.1} ms\n")
        return result
    return inner


@timeit
def colorname(color, *, file="colors.csv"):
    """
    Source of colors.csv: https://github.com/codebrainz/color-names/blob/master/output/colors.csv
    """
    @micropython.native
    def reader():
        delimiter = ","
        with open(file) as fd:
            for line in fd:
                yield line.split(delimiter)

    @micropython.native
    def parser():
        """
        Assigning the first field to name and the last 3 fields to R, G, B
        Calculating the threshohld.
        Yield the result, if threshold is lesser or equal than 50 as the calculated absolute difference
        """
        # https://peps.python.org/pep-3132/
        for name, *_, R, G, B in reader():
            # https://peps.python.org/pep-0572/
            if (threshold := sum(abs(C - c) for C, c in zip(map(int, (R,G,B)), color))) <= 50:
                yield threshold, name

    try:
        if DEBUG:
            gc.collect()
            before = gc.mem_free()
        
        result = min(parser())
        
        if DEBUG:
            after = gc.mem_free()
            print("Mem usage:", before - after)
        return result[1]

    except ValueError:
        return "Unknown"


def min_function_test():
    def gen(start, end):
        while True:
            if start >= end:
                break
            yield start
            start += 1
    
    @timeit
    def min_func():
        print("min with generator")
        min(gen(0, items))
        
    @timeit
    def max_func():
        print("max with generator")
        min(gen(0, items))

    @timeit
    def min_max_1():
        print("Min-Max with generator")
        min_val = 0
        max_val = 0
        for value in gen(0, items):
            min_val, max_val = min(value, min_val), max(value, max_val)
        print(f"Min: {min_val} | Max: {max_val}")
        print()

    @timeit
    def min_max_2():
        print("Min-Max with range")
        min_val = 0
        max_val = 0
        for value in range(0, items):
            min_val, max_val = min(value, min_val), max(value, max_val)
        print(f"Min: {min_val} | Max: {max_val}")
        print()
        
    items = gc.mem_free()
    print("From 0 to", items, end="\n\n")
    for func in (min_func, max_func, min_max_1, min_max_2):
        func()


def bad():
    gc.collect()
    names = []
    colors = []
    with open("colors.csv") as fd:
        for line in fd:
            row = line.split(",")
            names.append(row[0])
            # it fails here
            colors.append((int(row[-3]), int(row[-2]), int(row[-1])))


print("Running min_function_test")
try:
    min_function_test()
except MemoryError:
    print("My assumption was wrong\n")
else:
    print("min_function_test has enough memory\n")


colors = [
    (255, 0, 0),
    (0, 255, 0),
    (0, 0, 255),
    (255, 255, 0),
    (0, 255, 255),
    (255, 0, 255),
    (210, 90, 38),
]

print("Running optimized function")
for color in colors:
    print(colorname(color), color)
    print()


print("Running bad function")
bad()

Output:

MPY: soft reboot
Running min_function_test
From 0 to 173792

min with generator
min_func took 2188 ms

max with generator
max_func took 2188 ms

Min-Max with generator
Min: 0 | Max: 173791

min_max_1 took 6930 ms

Min-Max with range
Min: 0 | Max: 173791

min_max_2 took 3765 ms

min_function_test has enough memory

Running optimized function
Mem usage: 29856
colorname took 788 ms

red (255, 0, 0)

Mem usage: 29616
colorname took 791 ms

electric_green (0, 255, 0)

Mem usage: 29392
colorname took 791 ms

blue (0, 0, 255)

Mem usage: 29648
colorname took 792 ms

electric_yellow (255, 255, 0)

Mem usage: 29600
colorname took 791 ms

aqua (0, 255, 255)

Mem usage: 29616
colorname took 792 ms

fuchsia (255, 0, 255)

Mem usage: 29904
colorname took 792 ms

flame (210, 90, 38)

Running bad function
Traceback (most recent call last):
  File "main.py", line 140, in <module>
  File "main.py", line 109, in bad
MemoryError: memory allocation failed, allocating 4096 bytes
MicroPython v1.22.0-preview.156.g7d7b9037c on 2023-11-13; Raspberry Pi Pico W with RP2040
Type "help()" for more information.