A better way to split()?

[jmytyk]

Hi All,

I'm attempting to use the ESP8266 to interface with a LED driver, the driver communicatation needs a PWM signal. The ESP board is connected to a PWM breakout via i2c to set the PWM duty cycle so the drivers react properly. Everything is up and running, I can modify the duty cycles, pull values from a look-up table to set duty cycles as needed. I can talk to 10 different drivers...

I would like to extend this project so that I can create an arbitrary intensity shape 'offline' to be used in my controller. (Intensity and duty cycle are the same here...) By offline I mean, using something like scipy.interpolate -> CubicSpline to visualize the curve i'm trying to create on my laptop. This all works, but i've run into a problem loading shape into memory on the ESP8266. I'm exporting the cubic spline values as a text file, and it's a giant single line string (30k columns-ish) (4096 numbers x-xxxx, split by a space.) This is not likely an ideal approach, and this whole thing is a new problem for me... anyone have any ideas on how better to move a List created in one spot into a List to be used on the 8266??

Where this is falling over is when i call Split() it runs out of memory. Doing a bit of digging, this isn't a new problem as there are up to 3 internal copies of the string being chopped up. Is there a more memory efficient version of split? I can trade time, a few 10's of seconds to chug through is okay... I've seen a few suggestions, but nothing that made sense to me.

Thanks in advance!

[dlech]

• write an equation to approximate the curve and generate the values on the fly instead of a file with lots of values
• use binary data instead of text, read X bytes (one row) at a time out of the file and convert to ints using int.from_bytes() or struct.unpack().

[jmytyk]

thanks -- i came across the unpack() approach, i didn't see the int.from_bytes() -> i'll take a look at both.

cheers!

[jmytyk]

here's what worked, with some extra debug print() for those also stumbling through...

import struct

#--------------------------on laptop side
# Write integers to a file
with open("numbers.bin", "wb") as f:
    for number in data_out:
        data = struct.pack("i", number)
        f.write(data)


#--------------------------on micro side

def parseBin():
    
    with open("numbers.bin", "rb") as f:
   
        numbers = []
  
        while True:
            data = f.read(struct.calcsize("i"))
            print("attempting", data)
            if not data:
                break
            number = struct.unpack("i", data)[0]
            numbers.append(number)
        
    print("length of loaded numbers file", len(numbers))
    
    return numbers