Can I package a MicroPython project for PyPI using pyproject.toml?

[DavesCodeMusings]

I have a project that I'd like to make available as a module. I started by reading the tutorial suggested by the instructions at pypi.org. I successfully created the package and uploaded to test.pypi.org using a pyproject.toml file per the instructions in their tutorial.

Now, after looking at the MicroPython package docs I see there are subtle differences with the gzip compression and I suspect the package I created using CPython and the PyPI tutorial will not work with MicroPython.

The MicroPython doc references setup.py and the setuptools module. But, I got the feeling from PyPI's documentation that pyproject.toml is the new way of doing things and I'm left scratching my head, wondering how to proceed.

Is it possible to use pyproject.toml with MicroPython? If others have uploaded MicroPython modules to PyPI, I'm hoping you'll be willing to share some wisdom.

[dhalbert]

We put essentially all the CircuitPython libraries on PyPi; perhaps that could be helpful. We recently switched to pyproject.toml. Take a look at our repos: e.g. https://github.com/adafruit/Adafruit_CircuitPython_SHT31D for a single-file library, https://github.com/adafruit/Adafruit_CircuitPython_HID as an example of a multifile library. The libraries are checked and put on PyPi using GitHub Actions.

[jimmo]

For some extra context for @DavesCodeMusings (sorry if you already know this) -- one of the really awesome things that Adafruit do is make all their CircuitPython drivers work on CPython too.

So for example, if you have a Raspberry Pi (e.g. a Pi 4, running Linux), you can install a CircuitPython library (using pip), and their "Blinka" shim library will make it look as if it's running on CircuitPython (even though it's talking to the Linux kernel GPIO/SPI/I2C drivers etc).

So pip-installability (i.e. PyPi) is a really fundamental part of that workflow -- no microcontrollers involved.

For installation on microcontrollers (i.e. actual CircuitPython), from Adafruit's tooling at least, my understanding is that PyPI is not involved? @dhalbert ? (i.e. circup which is kind of your equivalent of mpremote mip as far as I understand).

Worth mentioning also, separate to this, there's also Thonny, which uses a tool called pipkin (by the same author), which does use PyPI as the source. However pipkin works with "regular" PyPI packages (no custom sdist or compression or filesystem layout), and instead patches the package on the fly during installation. (So this is another reason why CircuitPython publish all their packages to PyPI -- so they are installable via Thonny).

[dhalbert]

Thanks @jimmo. Right, exactly, we were already publishing most libraries to PyPi for Blinka, and Thonny's use of PyPi cemented the idea of publishing everything to PyPi, even those libraries that can't possibly run under Blinka.

The original, original motivation for Blinka was actually to run CircuitPython libraries on top of MicroPython. That still exists, but it is not as mature as the Linux shim.

Yes, circup is the recommended tool for easy library installation on actual boards.

[jimmo]

If you want to package your project for use in CPython, then yes, PyPI and pyproject.toml (or any other other ways) are the way to go.

For packaging for MicroPython we take a much more light-weight approach with the mip tool (available both on-device and via mpremote). See the latest version of the document you linked too https://docs.micropython.org/en/latest/reference/packages.html

(This is all new since v1.19.1 -- mip will be installed by-default on network-capable boards starting with v1.20. You can use mip with mpremote as of the version that was released a few weeks ago, v0.4.0. You can also install mip on your board manually from micropython-lib.).

Some historical perspective: we used to have a upip tool which supported installing "special" micropython packages from PyPI (which required custom packaging and other details, exactly what you're referring to). There were a number of problems with this approach. The one that I found the most frustrating was that finding suitable packages on PyPI was basically impossible because it was too much of a burden on package maintainers to do it correctly, as well as being able to tell the difference between micro- and non-micro packages (not to mention packages for incompatible MicroPython-based distributions). It was also very difficult to support things like compiling to .mpy automatically etc. (Not impossible mind you, but more complexity than necessary).

We also wanted to have all this work in the same way as freezing packages into the firmware.

So there are two main approaches for packaging MicroPython:

• Publish your entire package to micropython-lib (i.e. to https://github.com/micropython/micropython-lib). This is particularly helpful for the community for packages like standard-library packages or "python ecosystem" packages (e.g. micro-implementations of commonly-used packages) because then it's a single community-maintained repository of useful things. Also for drivers.
• Publish your package to your own github repo and write a package.json file (see https://github.com/jimmo/micropython-mlx90640 for an example). Then users can install your library directly using e.g. mip.install('github:jimmo/micropython-mlx90640') or mpremote mip install github:jimmo/micropython-mlx90640. This is useful for projects where you want to maintain ownership/control. See mip-installable version of microdot miguelgrinberg/microdot#67 for some discussion relating to microdot.

We plan to bridge the gap between these two and make it possible to register your github repos with the micropython-lib repo -- you will get all the ownership benefits of hosting in your own repo, but all the deployment (e.g. compile to .mpy, install by name) of micropython-lib.

[DavesCodeMusings]

I like the idea of adding a package.json to my existing repo. It seems like it would have the least amount of barriers. I already have the repo, the GitHub account, etc.

The reason I latched onto PyPI was from using Thonny as an editor and its Tools > Manage Packages... capability. It seemed like it was going straight to PyPI.

[jimmo]

I like the idea of adding a package.json to my existing repo. It seems like it would have the least amount of barriers. I already have the repo, the GitHub account, etc.

Great! This is exactly the idea and the motivation. You already have all this stuff set up on GitHub, why not host it from there too. It's also inherently namespaced and easy to trace. (Worth noting that mip still supports dependencies, both third-party and micropython-lib, as well as versioning via git tag/branch).

The reason I latched onto PyPI was from using Thonny as an editor and its Tools > Manage Packages... capability. It seemed like it was going straight to PyPI.

I hope Thonny & pipkin can add support for our scheme too. We publish a package index here https://micropython.org/pi/v2/index.json that tools can query. Alternatively we may investigate some way of automatically pushing micropython-lib packages to PyPI (but this is not high on the priority list right now).

[DavesCodeMusings]

I realize this post is getting a little old, but I wanted to share my recent success using package.json in a GitHub repository. The format is pretty simple. It looks like this:

{
  "urls": [
    ["repl_buddy.py", "github:DavesCodeMusings/repl-buddy/repl_buddy.py"]
  ],
  "version": "1.0"
}

Even when there's only one file like the example above, I still like the package.json approach, because it makes the mip command short and simple, requiring only the username/repo instead of a full URL.

Windows:

py -m mpremote mip install github:DavesCodeMusings/repl-buddy

Linux:

mpremote mip install github:DavesCodeMusings/repl-buddy

Very slick! Whoever thought this up and built it has my sincere appreciation. And, I'm deleting my PyPI account now. ;-)

[jimmo]

Woohoo! Glad it worked for you. repl-buddy looks very neat too!

[DavesCodeMusings]

Hey, thanks! REPL Buddy came about once I realized all I have to do is open PuTTY on a serial port and I get a REPL prompt. Serendipity! For some reason, I thought it should be more work than that. (Too much time spent with Arduino and C++, I guess.)

MicroPython amazes me more and more each day.