Reading/Writing .mpy Files with MQTT

[AngeloFatica]

I have a project I am working on where I am able to send python files through mqtt to my pico and have it update those files inside its own file system. It is currently working with python files as plain text, but I cannot figure out how to get mpy to write to a file properly. I have it to the point where it matches exactly what you see from opening the mpy file, except it reads as plain text instead of binary/hex. I am using 'wb' to write to the file currently, but I have had no luck getting a compatible mpy file to work. Is this possible with mpy files?

[jimmo]

This should definitely be possible. I'm not sure what you mean by "except it reads as plain text instead of binary/hex" but sounds like something is going wrong with the encoding/decoding or the file mode (e.g. is it being encoded to base64 or hex for the mqtt message, but not decoded before writing?). You will need to post your code so we can provide more specific advice.

[AngeloFatica]

elif topic.decode() == "filedata":
        if "4d" in str(msg.decode()).split()[0]:
            filedata = msg
            isBinary = True
            f = open(filename, 'wb')
            f.write(filedata)
            f.close()

So this is the receipt of bytes from mqtt, when running type on it its read as bytes.
When I was talking about plain text vs hex I was talking about when I open it in a text editor to compare the hex, the text editor tells me the file is plain text where as it will tell me the actual mpy compiled file is binary. The actual text of both files is line for line and space for space the same.

[AngeloFatica]

Ive tried many renditions of this code before writing to the file, including casting to bytes again, string then bytes and a bunch of other random stuff to no avail.

[peterhinch]

There is something odd going on here. I wrote the following and cross-compiled it to rats26.mpy:

for x in range(10):
    print(x)

I then did the following

>>> with open("rats26.mpy", "rb") as f:
...     d = f.read()
... 
>>> len(d)
68
>>> with open("rats27.mpy", "wb") as f1:
...     f1.write(d)
... 
68
>>> import rats27
0
1
2
3
4
5
6
7
8
9
>>> 

So it's possible to read an .mpy file as binary data, write it out to a new binary file, and run it. If this is wrecked by putting a communication channel between the reading of d and its subsequent writing, then the channel must be corrupting the data.

[robert-hh]

Why do you consider that odd?

[peterhinch]

I think it's odd that @AngeloFatica reports that the data is unchanged. As far as I can see it must have been corrupted.

[AngeloFatica]

It sounds to me as its an encoding difference. Im sending it as a raw from mqtt and it is showing up as bytes in my msg. What encoding would work for mpy files?

[AngeloFatica]

4d06 001f 4523 1661 6475 636d 3335 352e
7079 000f 0874 696d 6500 0e6d 6163 6869
6e65 0008 6a73 6f6e 0014 6765 745f 7374

This is an example of the format I have when I decode it which seems like a normal mpy to me except its in utf-8 here. Just getting it to the proper encoding from this point is where I think I am having trouble.

[AngeloFatica]

@peterhinch @jimmo I solved this issue by concatenating all of the bytes into one long byte string then unhexlifying it. The issue was that I was copying the text version of the file which was bytes spaced out and having many lines, so I needed to get them into one line and in a byte array (like this: b'M\x06\x00\x1fE#\x16aducm355.py\x00\x0f\x08time\x00\x0emachine)