How to make a .mpy file for v 1.23 upython?

[krismc]

For the last few months I've been making .mpy files for v1.23 upython by typing

>pipx run  mpy-cross==1.23.0 -v code.py

into the windows command prompt
It's been working OK till today

Fatal error from pip prevented installation. Full pip output in file:
    C:\Users\Admin\pipx\logs\cmd_2024-12-24_16.19.45_pip_errors.log

Some possibly relevant errors from pip install:
    ERROR: Could not find a version that satisfies the requirement mpy-cross==1.23.0 (from versions: none)
    ERROR: No matching distribution found for mpy-cross==1.23.0
Error installing mpy-cross from spec 'mpy-cross==1.23.0'.

I tried

mpy-cross -v code.py

which produces a .mpy file that gives

ValueError: incompatible .mpy file

when I try to run it in v1.23 upython on an esp32. Presumably because it's v1.24?

Anybody know a way to produce v1.23 compatible .mpy files from the windows command prompt?

[bixb922]

See here https://pypi.org/project/mpy-cross/

This version of mpy-cross allows

mpy-cross --compat 1.23 my_app.py

[krismc]

I tried installing this version.

C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\mpy_cross>pip install mpy-cross
Collecting mpy-cross
  Downloading mpy_cross-1.24.1.post2-py2.py3-none-win_amd64.whl.metadata (3.6 kB)
Downloading mpy_cross-1.24.1.post2-py2.py3-none-win_amd64.whl (1.2 MB)
   ---------------------------------------- 1.2/1.2 MB 1.8 MB/s eta 0:00:00
Installing collected packages: mpy-cross
Successfully installed mpy-cross-1.24.1.post2

Sadly, it shows no inclination to actually make .mpy files. Or if it is making them, I can't find where it puts them.

C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\mpy_cross>mpy-cross --compat 1.23 code.py
usage: mpy-cross [<opts>] [-X <implopt>] <input filename>
Options:
--version : show version information
-o : output file for compiled bytecode (defaults to input with .mpy extension)
-s : source filename to embed in the compiled bytecode (defaults to input file)
-v : verbose (trace various operations); can be multiple
-O[N] : apply bytecode optimizations of level N

Target specific options:
-msmall-int-bits=number : set the maximum bits used to encode a small-int
-mno-unicode : don't support unicode in compiled strings
-mcache-lookup-bc : cache map lookups in the bytecode
-march=<arch> : set architecture for native emitter; x86, x64, armv6, armv7m, armv7em, armv7emsp, armv7emdp, xtensa, xtensawin

Implementation specific options:
  emit={bytecode,native,viper} -- set the default code emitter
  heapsize=<n> -- set the heap size for the GC (default 2097152)

C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\mpy_cross>

[bixb922]

I am sorry to cause a confusion. I checked again and what I am using is this:

https://pypi.org/project/mpy-cross-multi/

And the command is:

mpy-cross-multi --micropython 1.23 myprog.py

[Josverl]

In the vast majority of mpy-cross errors, I personally have found the error to be in the source.
I find the mpy-cross returns very little to no information if it can't compile for some reason, and I have needed to trigger native compilation mpremote run foobar.py or mpremote exec 'import foobar'on the MCU or in the unix port to get meaningful error reporting. ( Painful on esp8266 that simply does not have the memory to finish compilation)

[peterhinch]

Another option is to run a linter against the file. Weeds out a lot of errors before they get near the compiler.

[krismc]

Isn't it reasonable to expect though that a program that compiles for v1.24 (and used to compile for v1.23) should continue to compile for v1.23 after v1.24 supplanted v1.23 as the latest version?

[Josverl]

Isn't it reasonable to expect though that a program that compiles for v1.24 (and used to compile for v1.23) should continue to compile for v1.23 after v1.24 supplanted v1.23 as the latest version?

Semver minor changes, such 1.23.x --> 1.24.x , can add features that are not available in the older version,

So if there is no change at all to your code since the previous time you (cross)-compiled it, yes it should compile again.
If there were changes, it is likely, but not guaranteed.

Is your code public so we can try to help find a solution?

[krismc]

So typically what happens is a .mpy that compiles & runs OK in v1.24 seems to compile in v1.23 but won't run once it gets over ~75 lines. This failure of v1.23 files to run seems to have happened about the time that v1.24 became the default firmware.

I've inflated a generic test file with some unused functions to put it over the size limit.

The file compiles with

 mpy-cross -v wifibig.py

and runs on a v1.24 esp32 s2

>>> import os;os.uname()
(sysname='esp32', nodename='esp32', release='1.24.1', version='v1.24.1 on 2024-11-29', machine='Generic ESP32S2 module with ESP32S2')
>>> import wifibig

pwrup cause reset
01  connecting to WiFi -20 -19 -18 
07  ('192.168.0.67', '255.255.255.0', '192.168.0.1', '192.168.0.1') ss -73

but won't run on a v1.23 esp32 when compiled with

mpy-cross-multi --micropython 1.23 wifibig.py

or any of the other v1.23 compile cmds suggested on this thread.

>>> import os;os.uname()
(sysname='esp32', nodename='esp32', release='1.23.0', version='v1.23.0 on 2024-06-02', machine='Generic ESP32S2 module with ESP32S2')
>>> import wifibig  

# installing esp32-ulp on the esp32 (for programs that use the ulp counter)
import sys; from time import sleep, time, ticks_ms as ms; t0=ms()
from machine import deepsleep, reset_cause

def _t(): t=(ms()-t0+1000)/1000; elapsed_secs='%02d '%int(t); return elapsed_secs

def _getwlanscan():                                                                                                                                   # averages 5.7s
  from network import WLAN, STA_IF, AP_IF
  ap=WLAN(AP_IF); ap.active(0); sta=WLAN(STA_IF); sta.active(1)
  print(_t(), 'scanning for', AP); s=sta.scan()                                                                                                                                        # (b'WiFi', b'\x1c;\xf3a\x0c\x8c', 10, -73, 3,        False)
  for i in s:                                                                                                                                         #   ssid,           bssid,         ch,  SS, authmode, hidden
    print(_t(), i)
    if AP.encode() in i: print(_t(), AP, 'on channel', i[2]); break
  else: print(_t(), 'AP not found'); return
  for i in range(2):
    if not sta.isconnected(): print(_t(), 'connecting', end=' '); sta.connect(AP, pwd)
    for i in range(20):
      if sta.isconnected():
        ss=sta.status('rssi'); ss=ss if ss<0 else '?'
        print(); print(_t(), sta.ifconfig(), 'ss', ss); return 1                                                                                      # sta.status only available after connect without sta.scan
      sleep(2); print(i-20, end=' ')
    print('timeout')
  print(_t(), 'no wifi'); return                                                                                                                      # _schedo will _timestamp a 'no wifi' msg via _ifdiff is this is new

def _getwlan():                                                                                                                                       # averages 3.2s
  from network import WLAN, STA_IF, AP_IF
  ap=WLAN(AP_IF); ap.active(0); sta=WLAN(STA_IF); sta.active(1)
  for i in range(2):                                                                                                                                  # OSError: Wifi Internal Error on 2nd pass if no connection on 1st
    if not sta.isconnected(): print(_t(), 'connecting to '+AP, end=' '); sta.connect(AP, pwd)
    for i in range(20):
      if sta.isconnected():
        ss=sta.status('rssi'); ss=ss if ss<0 else '?'                                                                                                 # sta.status()=1010
        print(); print(_t(), sta.ifconfig(), 'ss', ss); return 1
      sleep(2); print(i-20, end=' ')                                                                                                                  # shorter sleeps (eg 0.5s) often timeout on waveshare s2
    print('timeout')
  print(_t(), 'no wifi'); return

def _getwlanfixedip():                                                                                                                                # averages 2.7s
  from network import WLAN, STA_IF, AP_IF
  ap=WLAN(AP_IF); ap.active(0); sta=WLAN(STA_IF); sta.active(1); maca=b'\xa2\xb5<<\xf5\xd9'
  sta.ifconfig(('192.168.0.220', '255.255.255.0', '192.168.0.1', '192.168.0.1'))
  for i in range(2):
    if not sta.isconnected(): print(_t(), 'fixed ip, connecting to '+AP, end=' '); sta.connect(AP, pwd, bssid=maca)
    for i in range(20):
      if sta.isconnected():
        ss=sta.status('rssi'); ss=ss if ss<0 else '?'                                                                                                 # sta.status()=1010
        print(); print(_t(), sta.ifconfig(), 'ss', ss); return 1
      sleep(2); print(i-20, end=' ')
    print('timeout')
  print(_t(), 'no wifi'); return

def _getwlanscan1():                                                                                                                                   # averages 5.7s
  from network import WLAN, STA_IF, AP_IF
  ap=WLAN(AP_IF); ap.active(0); sta=WLAN(STA_IF); sta.active(1)
  print(_t(), 'scanning for', AP); s=sta.scan()                                                                                                                                        # (b'WiFi', b'\x1c;\xf3a\x0c\x8c', 10, -73, 3,        False)
  for i in s:                                                                                                                                         #   ssid,           bssid,         ch,  SS, authmode, hidden
    print(_t(), i)
    if AP.encode() in i: print(_t(), AP, 'on channel', i[2]); break
  else: print(_t(), 'AP not found'); return
  for i in range(2):
    if not sta.isconnected(): print(_t(), 'connecting', end=' '); sta.connect(AP, pwd)
    for i in range(20):
      if sta.isconnected():
        ss=sta.status('rssi'); ss=ss if ss<0 else '?'
        print(); print(_t(), sta.ifconfig(), 'ss', ss); return 1                                                                                      # sta.status only available after connect without sta.scan
      sleep(2); print(i-20, end=' ')
    print('timeout')
  print(_t(), 'no wifi'); return                                                                                                                      # _schedo will _timestamp a 'no wifi' msg via _ifdiff is this is new

def _getwlan1():                                                                                                                                       # averages 3.2s
  from network import WLAN, STA_IF, AP_IF
  ap=WLAN(AP_IF); ap.active(0); sta=WLAN(STA_IF); sta.active(1)
  for i in range(2):                                                                                                                                  # OSError: Wifi Internal Error on 2nd pass if no connection on 1st
    if not sta.isconnected(): print(_t(), 'connecting to '+AP, end=' '); sta.connect(AP, pwd)
    for i in range(20):
      if sta.isconnected():
        ss=sta.status('rssi'); ss=ss if ss<0 else '?'                                                                                                 # sta.status()=1010
        print(); print(_t(), sta.ifconfig(), 'ss', ss); return 1
      sleep(2); print(i-20, end=' ')                                                                                                                  # shorter sleeps (eg 0.5s) often timeout on waveshare s2
    print('timeout')
  print(_t(), 'no wifi'); return

def _getwlanfixedip1():                                                                                                                                # averages 2.7s
  from network import WLAN, STA_IF, AP_IF
  ap=WLAN(AP_IF); ap.active(0); sta=WLAN(STA_IF); sta.active(1); maca=b'\xa2\xb5<<\xf5\xd9'
  sta.ifconfig(('192.168.0.220', '255.255.255.0', '192.168.0.1', '192.168.0.1'))
  for i in range(2):
    if not sta.isconnected(): print(_t(), 'fixed ip, connecting to '+AP, end=' '); sta.connect(AP, pwd, bssid=maca)
    for i in range(20):
      if sta.isconnected():
        ss=sta.status('rssi'); ss=ss if ss<0 else '?'                                                                                                 # sta.status()=1010
        print(); print(_t(), sta.ifconfig(), 'ss', ss); return 1
      sleep(2); print(i-20, end=' ')
    print('timeout')
  print(_t(), 'no wifi'); return

AP='my_ssid'; pwd='my_pwd'
causes={2:'reset', 1:'pwrup', 3:'wdt', 4:'wakeup', 5:'hardreset'}; cause=causes[reset_cause()]; print(); print('pwrup cause', cause)                  # won't report wakeup in repl mode on thonny, reports pwrup as hardreset
_getwlan()

#import mip; mip.install('github:micropython/micropython-esp32-ulp')

[Josverl]

@krismc,
Python 3.11.9
from reactions elsewhere I gather that you have not been making progress.
I have tried to reproduce your issue based on the file above on Windows 11 on Python 3.11.9 , and I cannot get it to fail.
a few notes though :

1) you mentioned using two different tools : mypy-cross and mpy-cross-multi

I have verified correct operation with :
- mpy-cross v1.23.0
- mpy-cross v1.24.1.post2
- mpy-cross-multi --version MicroPython a61c446 on 2024-06-01; mpy-cross emitting mpy v6.3
- --micropython 1.22.0
- --micropython 1.23.0
- --micropython 1.24.1 # Error: targeting MicroPython v1.24.1 is not supported

In my test all tools produces binary identical results, which is to be expected as the targeted MicroPython versions all use the same .mpy version ( v6.3).
but note that even a v1.20.0 .mpy file was still binary identical. ( apparently the .mpy bytecode needed none of v6.1+ features)

So both tools work very well - mpy-cross-multi most likely needs an update though. (@dlech where should this be reported? )

To install a specific version I recommend to use version specifiers like ~=1.24.1 as there are sometimes post releases such as mpy_cross-1.24.1.post2

> pip install mpy-cross~=1.23.0
Collecting mpy-cross~=1.23.0
  Using cached mpy_cross-1.23.0.post2-py2.py3-none-win_amd64.whl.metadata (3.6 kB)
Using cached mpy_cross-1.23.0.post2-py2.py3-none-win_amd64.whl (999 kB)
Installing collected packages: mpy-cross
Successfully installed mpy-cross-1.23.0.post2

> pip install mpy-cross~=1.24.1
Collecting mpy-cross~=1.24.1
  Using cached mpy_cross-1.24.1.post2-py2.py3-none-win_amd64.whl.metadata (3.6 kB)
Using cached mpy_cross-1.24.1.post2-py2.py3-none-win_amd64.whl (1.2 MB)
Installing collected packages: mpy-cross
Successfully installed mpy-cross-1.23.0.post2

Hardware tests

I tested the generated .mpy modules on the following combination of hardware and firmware

Board	V1.23.0	V1.24.1	V1.25.0-preview-365
ESP32	✅	✅	✅
ESP32 S3	✅	✅	✅
ESP32 C6	✅	✅	✅
ESP32 S2	...	...	...

I have no ESP32-S2 so unable to test.

A few other things.

I noticed that you tend to compress the python code on a single line.
Cross compiling a well formatted source file, again produces the exact same micropython bytecode, not a single byte longer.
so there is no runtime size advantage , and the code is , IMO, a lot harder to read.

I actually think that there is a logic hidden in _getwlanscan where two overlapping for loops both use i as iterator, that may be better to understand if that function is formatted.

With the default options , the .mpy file is actually a bit larger as it needs to store additional 'source lines' for an error's traceback to be able to refer to a line in your code. if you can do without , add -O3 to mpy-cross and you'll save those bytes , while keeping your core readable.
(P.S. Im not sure if or when these line number are ever loaded into RAM, or weather they remain in Flash / ROMFS )

[krismc]

Thnx for all your effort squire but I cannot make this work

C:\Users\Admin>pip install mpy-cross~=1.23.0
Collecting mpy-cross~=1.23.0
  Downloading mpy_cross-1.23.0.post2-py2.py3-none-win_amd64.whl.metadata (3.6 kB)
Downloading mpy_cross-1.23.0.post2-py2.py3-none-win_amd64.whl (999 kB)
   ---------------------------------------- 999.2/999.2 kB 2.2 MB/s eta 0:00:00
Installing collected packages: mpy-cross
  Attempting uninstall: mpy-cross
    Found existing installation: mpy_cross 1.24.1.post2
    Uninstalling mpy_cross-1.24.1.post2:
      Successfully uninstalled mpy_cross-1.24.1.post2
Successfully installed mpy-cross-1.23.0.post2

C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\mpy_cross>mpy-cross -v code.py

but when I put it on an esp32s2 (version='v1.23.0 on 2024-06-02') & try to run it

>>> import code.mpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: incompatible .mpy file

Is there some test I can run to find out where my 1.23.0.post2 mpy differs from yours?
Why does mip seem to install something slightly different to what was requested? In your examples you asked for 1.23.0 but it installed 1.23.0.post2 and when your requested 1.24.1 it installed 1.23.0.post2

[Josverl]

Why does mip seem to install something slightly different to what was requested? In your examples you asked for 1.23.0 but it installed 1.23.0.post2

The ~= is the Compatible release version specifier. I prefer it over the == .
Read more on Python packaging Version specifiers

Why does mip seem to install something slightly different to what was requested? In your examples you asked for 1.23.0 but it installed 1.23.0.post2 and when your requested 1.24.1 it installed 1.23.0.post2

My bad, That was a C&P error :

pip uninstall mpy-cross
WARNING: Skipping mpy-cross as it is not installed.
PS D:\mypython\!-stubtestprojects\issue-checks> pip install -I mpy-cross~=1.24.1
Collecting mpy-cross~=1.24.1
  Using cached mpy_cross-1.24.1.post2-py2.py3-none-win_amd64.whl.metadata (3.6 kB)
Using cached mpy_cross-1.24.1.post2-py2.py3-none-win_amd64.whl (1.2 MB)
Installing collected packages: mpy-cross
Successfully installed mpy-cross-1.24.1.post2

[notice] A new release of pip is available: 24.0 -> 25.0.1

[dlech]

@dlech where should this be reported?

https://github.com/pybricks/python-mpy-cross-multi

[andrewleech]

The https://pypi.org/project/mpy-cross/ package can export older versions too, though that's handled by the "mostly transparent" wrapper script, not when executing the mpy-cross binary directly as in https://github.com/orgs/micropython/discussions/16477#discussioncomment-11665175

If you use a tool like uv to install it'll handle the path for you, so you can uv tool install mpy-cross and then just run mpy-cross --compat 1.23 my_app.py as per the "Older Version Support" section in the readme https://pypi.org/project/mpy-cross/

Behind the scenes that package bindles older mpy-cross binaries as well which are auto-selected by the wrapper script.

On a related note though, 1.23 is still the same bytecode version as the current version (1.24) so you shouldn't need to use an older version mpy-cross at this point, the current version should still work fine.

[krismc]

The variable results with different .MPYs I produce (sometimes I get an 'incompatible mpy' sometimes the code just hangs) makes me think I might be falling foul of the Windows directory structure on my PC. I'm working from C:\Users\Admin\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\mpy_cross> and indeed the mpy-cross.exe file in there changes when I try different pip installs & the .mpy produced by mpy-cross appears here also, but

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: C:\Users\Admin\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe -m pip install --upgrade pip

makes me think these packages maybe expect a different path?

[Josverl]

It appears that you have installed (different) versions of mpy-cross in the root python environment , and perhaps also different versions in different virtual environments.
Sanity check: run where.exe mpy-cross for the different terminals / command promts you frequent.

I would recommend first to remove all copies of mpy-cross and mpy-criss-something :

• running pip unstinstall mpy-cross multiple times in the same prompts as above untill it canot be found
• removing the pipx copy : pipx uninstall mpy-cross
• removing the uv tool copy : uv tool uninstall mpy-cross

while pipx run and uvx allow for a quick one-off install&run, you get the "version-of-the-day" , I prefer stable and predictable.

I recommend to pick one tool to manage all your system wide python tools ( uv tool , or pipx ) both of them work very well.

• install the most recent version of mpy-cross for your machine using that tool
  • uv tool install mpy-cross ,
  • pipx install mpy-cross

both of the tools you need to make sure that the tools are added to your path , this is a one-time action
either uv tool update-shell or pipx ensurepath

after that you can use mpy-cross in any prompt,
if you need to update , you update it once

• https://docs.astral.sh/uv/guides/tools/
• https://pipx.pypa.io/latest/getting-started/

[Josverl]

@andrewleech

If you use a tool like uv to install it'll handle the path for you, so you can uv tool install mpy-cross and then just run mpy-cross --compat 1.23 my_app.py as per the "Older Version Support" section in the readme https://pypi.org/project/mpy-cross/

If I try this on Windows 11 , python 3.11 , uv 0.5.26 :

uv tool install  mpy-cross
Ignoring existing environment for `mpy-cross`: the requested Python interpreter does not match the environment interpreter
Resolved 1 package in 119ms
Installed 1 package in 77ms
 + mpy-cross==1.24.1.post2
Installed 1 executable: mpy-cross.exe

mpy-cross --compat 1.23 .\basic.py
Error: No mpy-cross binary found in: C:\Users\josverl\AppData\Roaming\uv\tools\mpy-cross\Lib\site-packages\mpy_cross\archive\v1.5.1

>  tree C:\Users\josverl\AppData\Roaming\uv\tools\mpy-cross\Lib\site-packages\mpy_cross
C:\USERS\JOSVERL\APPDATA\ROAMING\UV\TOOLS\MPY-CROSS\LIB\SITE-PACKAGES\MPY_CROSS
├───archive
│   ├───v1.11
│   ├───v1.12
│   ├───v1.19
│   ├───v1.20.0
│   ├───v1.22.0
│   ├───v1.23.0
│   ├───v1.9
│   └───v1.9.3
└───__pycache__

Ill report that as a separate issue

[krismc]

Did the mpy-cross purge Jos ('where.exe mpy-cross' is such a great cmd for finding trouble lurking in the shadows) then tried mpy_cross-1.23.0.post2 and mpy_cross-1.24.0.post2 & verified the MPYs from both will run on a 1.23.0 esp32. Emboldened I then made & successfully ran a 1.17.0 mpy. So nice to finally stumble out of the minotaur's labyrinth into the sunshine, thanks for the hand holding.

I'm gonna stick with the pip uninstall mpy-cross / pip install mpy_cross~=1.17.0 way of doing things. Those flavour-of-the-month multi wrappers that supposedly generate any version .mpy are OK in theory but I can't make 'em work in practice.

The grief has left with me a new found interest in .MPYs. Can you suggest any simple tools for exploring .mpy structure & function? In particular it would be great to be able to compare 2 .MPYs. I'm wondering how you were able to determine they were binary identical?

[Josverl]

Glad you found a way out of the maze.
I think you also helped uncover a problem with --compat that I myself also failed to get to work for some time.
Reporting in this thread helped locate that problem, and allowed @andrewleech to fix that.

To compare I wrote a few lines of python, and ran that from a Jupiter notebook.

I can share that later.

[krismc]

I was keen to try mpy-tool.py https://docs.micropython.org/en/latest/reference/mpyfiles.html but from the instructions & usage example it feels like I'd need upython on a linux box.

[andrewleech]

Yes thanks for the discussions and thanks @Josverl for the review/testing of the --compat argument, it was indeed broken in my mpy-cross package.
While I believe the bug is fixed it's not yet in a released version, I'm still trying to fix/improve the release process to allow a re-release to happen!