Better way than json data?

[jbmfg]

I am using a D1 mini with micropython to display on a 16x16 led matrix. To address the individual leds i am loading a json file that contains a list of 256 rgb values. Looks something like this:

[ { "name": "mario", "frames": 2, "width": 16, "height": 16, "rgb_array": [ [136, 112, 0],[255, 255, 255],[136, 156, 0],[112, 0, 0]....[255,255,255] ], [ [136, 112, 0],[255, 255, 255],[136, 156, 0],[112, 0, 0]....[255,255,255] ] } ]

So i take that json data, parse out the rgb_array part and send that to the LED matrix. However, I run out of memory when i load in the json file. Is there a better way to do this? I would like to have a list of maybe 10 items in the json file to loop through and send to the matrix but anything more than a single frame causes a memory error. I've also tried to create one json file per animation (like the three frames of Mario running) but as soon as i try to load the next animation (or oftentimes the next frame) i run out of memory. I am open to other solutions beside json but nothing i've found fits the bill.

[sosi-deadeye]

You could use struct to pack 3 * 256 bytes.

import array
import struct

# client side
def send(name, frames, width, height, rgb_array):
    return struct.pack("<10s3H768B", name.encode(), frames, width, height, *rgb_array)


data = send("mario", 2, 16, 16, [255, 255, 0] * 256)

# micro controller
view = memoryview(data)  # should prevent copies


def receive(view):
    name, frames, width, height = struct.unpack("<10s3H", view[:16])
    rgb_array = struct.unpack_from("<768B", view, offset=16)
    return (name.decode().replace("\x00", ""), frames, width, height, rgb_array)


name, frames, width, height, rgb_array = receive(view)

The total size of this format is 784 bytes.
Code is not tested with micropython.

[jimmo]

The best thing you can do is have the file data in exactly the format suitable for sending to your LED matrix (i.e. if the LED expects BGR pixels, then encode it in BGR bytes. Neopixels are GRB from memory). The approach suggested by @sosi-deadeye using struct.unpack is a good example, but you might go one step further and not even unpack it, just send it straight out.

If you're loading the data off the filesystem, in order to minimise the memory used, you want to just have a single frame per file. i.e. the ideal is to open the frame file, read the whole thing, and send it directly to the strip without any processing.

If you want to use JSON, then something like:

{
  ... metdata...
  frames: [
    "<hex or base64 encoded data>",
    "<hex or base64 encoded data>",
  ]
}

Then you can load the JSON, and to send a frame you display.write(bytes.fromhex(json["frames"][index])).

[dmazzella]

you can use ucbor for binary data

[peterhinch]

See this summary of MicroPython serialisation libraries.

[jbmfg]

Excellent stuff everyone, thank you! As i said, i was originally storing the gif data as sets of 256 rgb integer triplets (ie (255, 255, 255)). The suggestions of sosi-deadeye and jimmo gave me the idea of converting that to bytes instead. Much better! So what I ended up doing is forgetting about the metadata as i dont really need it. Then i just took the frames' rgb values and moved them to one big list rather than a list of lists. So for 1 frame i have a list of 768 integers. I then called bytearray on that big list and saved that to a file. Then on the esp8266, i load that file and pull out every 3bytes and send that to the neopixel. Works like a charm and no memory issues! Thanks everybody!