MPY board file upload/download automation

[liudr]

I have been using thonny to manually upload files to my ESP32 MP dev boards for years but I need to automate some of these tasks such as to write a python program to connect to an MPY board, determine the files on the board and perform file upload and updates to the board, all without the user's interaction. The goal is to mass provision and update boards with automated scripts. I've read about adafruit-ampy, rshell, and mpfshell. It seems that mpfshell is for manual interaction with an MPY board, including entering commands to manipulate the file system. I need a programmatic way to interact with an MPY board. ampy seems to fit the description with plenty of command-line options to do these. rshell has a process commands from a file feature. I wonder if there exists a tool that is a Python module so I can call its methods and get say a list of the folder content in returned values instead of a printout in STDOUT. Maybe ampy is my best option? Thank you for any thoughts and inputs!

[liudr]

After testing ampy for a bit, I felt that I could use it, except, I just couldn't invoke it within Python script with os.popen(). I am not using a virtual environment. I installed ampy with admin terminal under win 11. I was able call it from within a terminal window with ampy or python -m ampy but just not when I used os.popen(). I tested pip and esptool. Both could be invoked with os.popen() so maybe it's just how ampy was written that caused the issue?

My python snippet:
ampout=os.popen("python -m ampy -p COM3 ls")
Response printed out:
C:\Program Files\Python312\python.exe: No module named ampy.main; 'ampy' is a package and cannot be directly executed

I also tried mpy_cross, which only runs when I used python -m but not directly. It runs with os.popen(). I'm stumped.

[bixb922]

Mpremote is callable from Python, although AFAIK there is no documentation about the call interface. See for example https://github.com/micropython/micropython/blob/master/tools/mpremote/mpremote/transport_serial.py

This is another approach where you may get some code and ideas, it also exposes an api: https://github.com/bixb922/freezeFS
This can generate self-extractable compressed files (like zip self-extractable archives). Run the resulting .py file and the microcontroller gets updated. This is what I developed to several microcontrollers

[glenn20]

I have built a python module, mpremote-path, which uses and wraps around functionality in mpremote. It aims to provide a python pathlib compatible API to access and manipulate files on a serial-attached micropython board from the host computer. It may be useful for you.

One caveat is that I have not been able to test it on Windows yet, and some peculiarities with the pathlib implementation might cause some unexpected problems.

[glenn20]

You may also be interested in the micropython-esp32-ota and mp-image-tool-esp32 tools.

[liudr]

Thanks @glenn20 I'll check out your module and the ota/image tools. Does mpremote have any way to increase serial baud rate to say 1.5MBaud (I have CP2102N on my boards)? I'm not familiar with the underlying mechanism (maybe RAW REPL?) the module relies on so I don't know if this baud rate can be changed. I tested ampy (doesn't seem to have any recent updates since 2021) and I can only use the default baud rate of 115200. All other values will result in lock-ups. So file upload with the default 115200 is very slow, about 3.5 minutes. My OTA routine can do that within 20 seconds. Apparently WiFi is faster than this baud rate :)

FYI, I made my own OTA functionality by downloading files from a web server that hosts my firmware to the esp32 board. This is not the typical OTA where an image of a partition is downloaded and flashed to OTA0 or OTA1, whichever is not the current OTA partition. It instead has OTA0 and OTA1 folders in the file system and a root-folder OTA.json contains information of which OTA folder to enter and execute code.

[glenn20]

@liudr : AH yes, my OTA tool is aimed at over-the-air updates of the micropython firmware itself, rather than python and other files on the filesystem.

As far as I know, mpremote had no functionality to chamge the baud rate. Micropython has only recently enabled support for changing the baud rate of the repl uart on esp32. I am running some experiments to dynamically set the baud rate for file transfers with mpremote-path. I'm not sure if I can make that reliable yet though.

[liudr]

Thanks again @glenn20
I am using a customized MP 1.20 that I compiled to enable W5500 ethernet and host name changing under ethernet connection. It's been detached from the main development for a while. I don't know if it is recent enough to have a baud rate change feature in it. I've settled with this scenario:
Run a PC python script that uses ampy to retrieve root folder content and then send a script for execution. The script includes the OTA features I have, which is using file system folders instead of partitions to keep 2 copies of firmware plus a factory version. Then do some basic set up for OTA, call the OTA features included in the script ampy sends to ESP32. I've already tested my OTA features via http requests so all I have to do is to make sure to conform to that convention I set up in OTA by http requests. With the amount of files I have, a minimal of 20 seconds is needed to complete the OTA, vs over 3 minutes when transferring files via ampy at 115200. I bet that each file transferred takes a delay after setting esp32 to raw repl so dozens of files mean a lot of delays (I set 2 second delays since less times mean inconsistent level of success). I tried thonny, which takes 40-plus seconds (serial port). Maybe thonny transfers all files together with no resetting repl and reenterring raw repl.