PYPI mpy-cross fails to install on macos-latest (macos-14/arm64) and MacOS testing

[Josverl]

I notice that some of my GHA tests that depend on mpy-cross have started to fail on macOS-latest with :

pipx run mpy-cross --version
ERROR: No matching distribution found for mpy-cross.
Error installing mpy-cross

macos-latest latest appears to have been recently bumped from 12.x to 14.x

Looking into some more detail I notice that the micropython-ci test only runs on : macOS-11 + Python 3.8

The macos-11 label has been deprecated and will no longer be available after 6/28/2024.
ref: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

So a few questions:

1. Can mpy-cross run on macos-latest (macos-14) or macos-13 ?
   I have no macos to test on - but in a Q&D test in actions
   the macos test fails on macos-14 with: 3 tests failed: ffi_callback float_parse float_parse_doubleprec
   

2. Does the mpy-cross package need an update to allow it to run on macos-14 ?

@andrewleech - are you still building the PyPi package for mpy-cross ?

On the ports_unix:macos test
4. What is the rationale for the mac-os test using macos-11.0 rather than macos-13 or -12 ?
I know that there have been a few MacOS / Python combinations that have caused issues with black in my own test , but with Python 3.11 + and the switch to ruff that should not be an issue anymore

Failing: macos-14 on arm64

Current runner version: '2.316.0'
Operating System
  macOS
  14.4.1
  [2](https://github.com/Josverl/micropython-stubber/actions/runs/8874892059/job/24363187691#step:1:2)3E224
Runner Image
  Image: macos-14-arm64
  Version: 20240422.[3](https://github.com/Josverl/micropython-stubber/actions/runs/8874892059/job/24363187691#step:1:3)
  Included Software: https://github.com/actions/runner-images/blob/macos-1[4](https://github.com/Josverl/micropython-stubber/actions/runs/8874892059/job/24363187691#step:1:4)-arm64/20240422.3/images/macos/macos-14-arm64-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macos-14-arm64%2F20240422.3
Runner Image Provisioner
  5.0.369.1+[5](https://github.com/Josverl/micropython-stubber/actions/runs/8874892059/job/24363187691#step:1:5)5cddbb57f254a3[6](https://github.com/Josverl/micropython-stubber/actions/runs/8874892059/job/24363187691#step:1:7)9b6e7b3b508cebdbee5d24be

_______________________ test_mpy_cross_bytecode_version ________________________

    def test_mpy_cross_bytecode_version():
        "Check if mpy-cross can be installed and run"
        cmd = ["pipx", "run", "mpy-cross", "--version"]
        result = subprocess.run(cmd, capture_output=True)
>       assert "mpy-cross emitting mpy" in result.stdout.decode("utf-8")
E       AssertionError: assert 'mpy-cross emitting mpy' in ''
E        +  where '' = <built-in method decode of bytes object at 0x10509b1b8>('utf-8')
E        +    where <built-in method decode of bytes object at 0x10509b1b8> = b''.decode
E        +      where b'' = CompletedProcess(args=['pipx', 'run', 'mpy-cross', '--version'], returncode=1, stdout=b'', stderr=b'\xe2\x9a\xa0\xef\xb8\x8f Found a space in the home path. We heavily discourage this, due to multiple\n    incompatibilities. Please check our docs for more information on this, as\n    well as some pointers on how to migrate to a different home path.\ncreating virtual environment...\ninstalling mpy-cross...\nFatal error from pip prevented installation. Full pip output in file:\n    /Users/runner/Library/Application Support/pipx/logs/cmd_2024-04-29_07.33.27_pip_errors.log\n\nSome possibly relevant errors from pip install:\n    ERROR: Could not find a version that satisfies the requirement mpy-cross (from versions: none)\n    ERROR: No matching distribution found for mpy-cross\nError installing mpy-cross.\n').stdout

tests/dependencies/test_dependencies.py:39: AssertionError

Success:

Current runner version: '2.315.0'
Operating System
  macOS
  1[2](https://github.com/Josverl/micropython-stubber/actions/runs/8791449281/job/24125704819#step:1:2).7.4
  21H1123
Runner Image
  Image: macos-12
  Version: 20240412.2
  Included Software: https://github.com/actions/runner-images/blob/macOS-12/20240412.2/images/macos/macos-12-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20240412.2
Runner Image Provisioner
  2.0.[3](https://github.com/Josverl/micropython-stubber/actions/runs/8791449281/job/24125704819#step:1:3)61.1+33a726b5[4](https://github.com/Josverl/micropython-stubber/actions/runs/8791449281/job/24125704819#step:1:4)233caea1331[5](https://github.com/Josverl/micropython-stubber/actions/runs/8791449281/job/24125704819#step:1:5)dba3887b0e7c5a1d482

Full details

Q&D test of a unix build & test on MacOS-14

  macos-14:
    runs-on: macos-14
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-python@v5
      with:
        python-version: '3.8'
    - name: Build
      run: source tools/ci.sh && ci_unix_macos_build
    - name: Run tests
      run: source tools/ci.sh && ci_unix_macos_run_tests
    - name: Print failures
      if: failure()
      run: tests/run-tests.py --print-failures    

Results in :1

3 tests failed: ffi_callback float_parse float_parse_doubleprec

Run tests/run-tests.py --print-failures
  tests/run-tests.py --print-failures
  shell: /bin/bash -e {0}
  env:
    pythonLocation: /Users/runner/hostedtoolcache/Python/3.8.10/arm64
    PKG_CONFIG_PATH: /Users/runner/hostedtoolcache/Python/3.8.10/arm64/lib/pkgconfig
    Python_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.8.10/arm64
    Python[2](https://github.com/Josverl/micropython/actions/runs/8875532785/job/24365171243#step:6:2)_ROOT_DIR: /Users/runner/hostedtoolcache/Python/[3](https://github.com/Josverl/micropython/actions/runs/8875532785/job/24365171243#step:6:3).8.10/arm64
    Python3_ROOT_DIR: /Users/runner/hostedtoolcache/Python/3.8.10/arm6[4](https://github.com/Josverl/micropython/actions/runs/8875532785/job/24365171243#step:6:4)
--- /Users/runner/work/micropython/micropython/tests/results/ports_unix_ffi_callback.py.exp	2024-04-29 08:27:18
+++ /Users/runner/work/micropython/micropython/tests/results/ports_unix_ffi_callback.py.out	2024-04-29 08:27:18
@@ -1,2 +1 @@
-org string: bytearray(b'foobar')
-qsort'ed: bytearray(b'abfoor')
+CRASH
\ No newline at end of file
--- /Users/runner/work/micropython/micropython/tests/results/float_float_parse_doubleprec.py.exp	2024-04-29 08:27:1[6](https://github.com/Josverl/micropython/actions/runs/8875532785/job/24365171243#step:6:6)
+++ /Users/runner/work/micropython/micropython/tests/results/float_float_parse_doubleprec.py.out	2024-04-29 08:2[7](https://github.com/Josverl/micropython/actions/runs/8875532785/job/24365171243#step:6:7):16
@@ -1,9 +1,9 @@
-1e+300
-1e+200
-1.0
-1.0
-1e+100
-1e-100
+1.000000000000003e+300
+1.000000000000003e+200
+1.000000000000003
+1.000000000000003
+1.000000000000003e+100
+1.000000000000003e-100
 9.00000000000000e-301
 9.00000000000000e-201
 9.00000000000000e-01
--- /Users/runner/work/micropython/micropython/tests/results/float_float_parse.py.exp	2024-04-29 0[8](https://github.com/Josverl/micropython/actions/runs/8875532785/job/24365171243#step:6:8):27:16
+++ /Users/runner/work/micropython/micropython/tests/results/float_float_parse.py.out	2024-04-2[9](https://github.com/Josverl/micropython/actions/runs/8875532785/job/24365171243#step:6:9) 08:27:16
@@ -2,12 +2,12 @@
 0.0
 1.0000e+00
 1.0000e+20
-1.0
-1e+20
+1.000000000000001
+1.000000000000001e+20
+False
 True
 True
 True
-True
 1e-37
 1e-37
 1e-37

FAILURE /Users/runner/work/micropython/micropython/tests/results/ports_unix_ffi_callback.py

FAILURE /Users/runner/work/micropython/micropython/tests/results/float_float_parse_doubleprec.py

FAILURE /Users/runner/work/micropython/micropython/tests/results/float_float_parse.py

[andrewleech]

FYI the mpy_cross package on pypi is published from my auto-builder on Gitlab:

• https://pypi.org/project/mpy-cross/#data
• https://gitlab.com/alelec/mpy_cross

This little CI project is something I set up ~6 years ago and have barely touched since, it diligently updates its own micropython submodule including tags, building and publishing mpy_cross packages as needed without human intervention.

There's a loose plan to replace this with one built here, but no schedule or timeline for that.

Being hands-off I really haven't looked into updating the compatibility of the packages unless someone points out one is broken... like now.

Edit: the build image currently used to cross compile for macos hasn't been updated in a few years, but there's now actual Mac runners available on Gitlab I could possibly use instead: https://docs.gitlab.com/ee/ci/runners/hosted_runners/macos.html

[Josverl]

I think the crux is in the arm64 that is now the new default for macos-14/-latest - and somehow that was overlooked 6 years ago :-)

it would be great if you could get a mpy-cross package that can run on both macos arm64 and x64
for now I've pinned the tests in my repo to macos-13

For the MicroPython repo and build that will need to update from macos-11 as the time is running out on that one.
Pinning to macos-13 (x64) will punt the x64/arm64 problem a few years into the future, but may not be the best solution.

[dlech]

In the meantime, I've published https://pypi.org/project/mpy-cross-multi which pulls in multiple versions of mpy-cross at the same time and has builds for arm64 macOS.

[andrewleech]

Cheers, actually just yesterday I got new builds working for my mpycross package for OSX, both Intel and Apple silicon.

I'll get a release of it out soon, and look at how to rebuild old versions as appropriate

[Josverl]

In the meantime, I've published https://pypi.org/project/mpy-cross-multi which pulls in multiple versions of mpy-cross at the same time and has builds for arm64 macOS.

If you can add in a --mpy option, that would be useful to avoid needing to lookup the mpy / version page, and still get the best/latest cross-compiler

[dlech]

The --micropython option automatically chooses the correct compiler (e.g. MPY ABI version, e.g. 6.2) based on the MicroPython version. I figure most people would know the MicroPython version they were running rather than the MPY ABI version.

And since the packages it pulls in have commands of their own, if you know the MPY version you want, you can run mpy-cross-$MPY instead of mpy-cross-multi --micropython $MP.

Either way, you shouldn't have to look up the table to translate one version to the other.

But if that doesn't solve all use cases already, happy to consider a different way of doing it.