Add `mpremote mip install -r requirements-mip.txt and 'pyproject.toml'

[Josverl]

I was thinking to add a -r/--requirement option to mpremote mip install command, to allow installing packages from a requirements file.

This is a common feature in Python package managers, and it would be useful for users who want to install multiple packages at once. urrently the only option is to chreate a shell script ( .ps1/.sh/zsh/....) that calls mpremote mip install for each package.

The primary goal would be is to add a -r/--requirement option to mpremote mip install command, to allow installing packages from a requirements file.

I want to focus on the mpremote mip package manager, and not attempt to implement all capabilities in the mip module available in the MicroPython runtime, as not all features available in mpremote mip package manager make sense in mip module .
Some changes, such as changes to the dependency specifiers, should be reflected in the mip module as well

• A requirements file: requirements.txt - A text file listing the dependencies required for the project, commonly used in Python projects.
• A micropython package.json file , with additional features
• pyproject.toml - A configuration file for Python projects that can specify dependencies and build system requirements.
• A lock file: requirements-mip.lock - A file that records the exact versions of dependencies to ensure reproducible installations.

I want to initially focus on the 'package.json' and pyproject.toml file, as these are the most common formats used in Python projects.

I have not spend efforts on the lock file yet , as that would also mean implementing a resolver for the dependencies, which is not trivial.
Hoever even if the lock file is not part of the initial PR, I think it would be a good idea to consider the structure / format so that it would be possible to install based on a lock file in the future in order to achieve reproducible installations.
@aivarannamaa , do you have suggestions on this ?

Related conversations:

• https://github.com/orgs/micropython/discussions/16683
• https://github.com/orgs/micropython/discussions/16250

using the micropython package.json format from a local file [Added]

In a merged but not yet released version of mpremote the command
mpremote mip install <package_name>.json will install the defined content of that package.

• @glenn20

• package.json can have:
  • multiple local files or remote ( key = "url")
  • multiple dependencies listed ( key = "deps")

https://docs.micropython.org/en/latest/reference/packages.html#writing-publishing-packages

Note: All files and modules will be deployed to the same directory.
while it is possible to use a mpremote --target option to specify a target directory,
it is not possible to use the --target option to specify a target directory for each file or module in the package.json file.

To deploy a micropython solution to a device using dependencies will, in its current form, require multiple package files. One to install the dependencies, and one for the main application.
I do not think that they can be combined in a single file, but that would be a nice feature to have. Note that the same will need to be handled for the pyproject.toml file as well.

Any integration with other python tooling tooling, such as for the installation or configuration of formnatters , linters and static type checking will need to be added, or handled in other configuration files such as pyproject.toml.
Whilhile it is possible for additional nodes to be addedd to the package.json file, some additional guidance, and possible a official schema would be useful to help users understand how to use the package.json file for their projects.

I think the documentation on the use of a local package.json file for deployment to a device should be improved, as I could not find a clear description of how to use this, and what the limitations are.

using the pyproject.toml file format.

Similar to the requirements file, it should be possible to use a pyproject.toml file to specify the dependencies for a micropython project. This is a common file format used in Python projects, and it would be useful for users who want to specify their dependencies in a standard way, and allow for a single configuration file for the project.

This would allow users to use the same file format for mpremote mip as they do for other Python projects, making it easier to share micropython projects, including their dependencies.
The pyproject.toml file should be a standard toml file, with a [project] section for the project metadata, and a [tool.mpremote] section for the mip dependencies.

mpremote mip install -r pyproject.toml
File: pyproject.toml:

[project]
name = "micro-parrot"
version = "0.1.0"
classifiers = [
  "Programming Language :: Python :: MicroPython",
  "License :: OSI Approved :: MIT License"
]
mip = [
    "aiorepl",
    "aioble==0.5.0",
    "github:peterhinch/micropython-mqtt",
    "github:scruss/micropython-SYN6988/syn6988.py",
]

The use of the [tool.mpremote] table is linked to the name of the mpremote package on PiPI

the name mip seems a logical location , leaving room for other configuration, but there are other options as well.
I'm open to alternatives here to cover additional scenarios, as after reading it is simply a dict with a list of packages.
packages = toml_data['tool']['mpremote']['mip']
We could extend to handle different ports , or groups similar to [extras]

using the requirements.txt format

There is little or no use for a requirements.txt format as with 1.25.0 mpremote will be able to use package.json files ( which I initially overlooked)

requirements.txt is no longer considered

It should be possible to use a requirements file format that is compatible with the pip package manager.
This would allow users to the simple requirements file format for mpremote mip, making it easier to share MicroPython projects, including their dependencies.

mpremote mip install -r requirements-mip.txt
will install the requirements from requirements-mip.txt file:

File: requirements-mip.txt

# micropython-lib
umqtt.simple
aiorepl
# version specific 
aioble==0.5.0
# legacy version specification
binascii==2.4.1
# Github packages
github:josverl/micropython-stubs/mip/typing.mpy

The requirements file format is a simple text file with one package per line, or a comment line starting with #.
Note: The https://pip.pypa.io/en/stable/reference/requirements-file-format/ requirement file format has (many) additional features that are not supported by mpremote mip install, such as:

• -e for editable installs
• -r for including other requirements files
• -c for constraints files

PEP 508 Dependency specifiers ( all formats)

mip does not currently recognize the PEP 508 dependency specifiers. Instead it uses the @ sign to denote a version specifier.

$ mpremote mip install [email protected]

However thr @ sign is not a standard way to specify dependencies in Python packaging. The PEP 508 standard uses the == operator for exact version matching.
Also the @ sign is also used in the mip module to denote a branch or tag in a git repository.

$ mpremote mip install github:org/repo@branch-or-tag

I think it would be better to use the PEP 508 standard for version specifiers in mip, and use the == operator for exact version matching. This would make it more consistent with other Python packaging tools and avoid confusion with the @ sign used for git repositories.
To allow for backwards compatibility, mip should still recognize the @ version specifiers, but apart from GitHub and GitLab sources, I propose it should be deprecated in favor of the == per PEP 508 standard.

https://packaging.python.org/en/latest/specifications/dependency-specifiers/#dependency-specifiers
Exerpt from PEP 508:

version_cmp   = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '==='

Currently mip only recognizes the == operator for exact version matching, or specifies the most recent version available.

While I think that there is benefit to adding support for the remaining version specifiers in PEP 508, I am not sure if there is suffient of a use case for this.
While there are multiple packages that have multiple published versions , I am not sure if there currently is a need end users to specify a version range for a package as there are cureenltly only a few packages that have more than 2 published versions.

I think that the most common use case for mip are

• micropython-lib : install the most recent version or a specific version from micropython-lib
• http or git : install a package from a specific the branch or tag of the package.

Proof of concept

A limited PoC is located in my fork of micropython.

~/micropython/scratch$ mpremote mip install -r pyproject.toml 
Install github:peterhinch/micropython-mqtt
Installing github:peterhinch/micropython-mqtt/package.json to /sd/lib
Installing: /sd/lib/mqtt_as/__init__.py
Installing: /sd/lib/mqtt_as/mqtt_v5_properties.py
Installing: /sd/lib/mqtt_as/range.py    
Installing: /sd/lib/mqtt_as/range_ex.py 
Installing: /sd/lib/mqtt_as/clean.py    
Installing: /sd/lib/mqtt_local_example.py
Install aiorepl
Installing aiorepl (latest) from https://micropython.org/pi/v2 to /sd/lib
Installing: /sd/lib/aiorepl.mpy
Install aioble                          
Installing aioble (0.5.0) from https://micropython.org/pi/v2 to /sd/lib
Installing: /sd/lib/aioble/__init__.mpy
Installing: /sd/lib/aioble/core.mpy
Installing: /sd/lib/aioble/device.mpy
Installing: /sd/lib/aioble/peripheral.mpy
Installing: /sd/lib/aioble/server.mpy
Installing: /sd/lib/aioble/central.mpy  
Installing: /sd/lib/aioble/client.mpy   
Installing: /sd/lib/aioble/l2cap.mpy    
Installing: /sd/lib/aioble/security.mpy
Done

👷try it: uv pip install git+https://github.com/Josverl/micropython.git@mpr/mip_requirements_file#subdirectory=tools/mpremote
( slow even with uv due to the size of the micropython repo, but it works)

[scruss]

MicroPython doesn't have a TOML parser in micropython-lib, so using device-side mip might have to rely upon a requirements.txt-type file.

This seems to mostly duplicate the function of package.json, which is already in mip and documented too.

[Josverl]

Thanks for the feedback.

I only ever looked at package.json as a way to publish a solution, not to manage a solution.
And that may because that is the way that the docs describe it.

But you have a point that it may be possible to use it during development as well now that it is possible to refer to a local file as the source.
Before that the need to start a webserver to refer to a local file was just impractical.

[scruss]

TD?

[Josverl]

Fat fingers, small keyboard

[aivarannamaa]

• I like the idea that mpremote mip could be more powerful than themip module. It would be a pity if the existence of mip module, because of its constrained environment, held back some improvements in MicroPython packaging. (TBH, I never understood the need for this module. Is it for avoiding transferring files over slow serial connection? Or for avoiding local development altogether?)
• I miss a good format for declaring not only project dependencies but also the files that need to end up on the device (think setuptools' py-modules, packages, MANIFEST.in etc). package.json comes close, but the urls block feels rather clumsy with references to specifc repository
  • wow, I now re-checked and discovered 2992e34. This is really nice! Together with 11c9656, package.json now looks like a decent option for MicroPython project specification.
  • package.json is designed for specifying libraries, but with some tweaks in (mpremote) mip it could be useful for apps as well.
    • (mpremote) mip install could install the main files (the urls part) to / whenever main.py is present in the package. The deps would still go to /lib.
    • mpremote mip install should be able to install to a local target. One could then install deps into project dev directory, with something like mpremote mip install --only-deps --local-target ./typings ./package.json, for purpose of static analysis / code completion. This doesn't have to be manual -- an IDE could do this transparently.
      • yes, there is micropython -m mip, but this comes with the mip-lite, which doesn't support package.json (right?)
• If I had to choose between package.json and requirements.txt, I'd rather vote for further development of package.json support, as it covers more aspects of project specification.
  • I'd love to be able to specify all these aspects in pyproject.toml, though. Perhaps in the future we have the choice between package.json and pyproject.toml, but I think right now it's more important to improve the logical structure and semantics of MicroPython project specification. And if some improvements would mean ditching the mip MicroPython module, I wouldn't object.

I also miss mip list and mip uninstall, but these are connected to project specification only indirectly (possibly via lock-files), so it's probably better to leave these for another thread, like @Josverl seemed to suggest.

[peterhinch]

I think the main application of mip is for WebREPL based maintenance of networked devices. It may be impractical to attach a USB lead to devices in the field.

I'd rather vote for further development of package.json support

Concur.

[Josverl]

I completely overlooked the (upcoming) use of a local package.json file.
I think the feature is under documented, and missing a few features, but its better to expand on that , than the requirements.txt format, therefore I have removed that from the initial post.

I do think that mpremote mip install is missing features that I would very much like to have:

• ability to describe the complete deployment ( source & data files, dependencies)
  It makes sense to extend the package.json with features such as Aivar mentioned above
  • allow to specify target locations outside of '/lib' or --target
  • better version specification
• integration with other python tooling( linters, formatting )
  for this we could use the [tools.mpremote][mip] table to refer to one (or more) package.json files,
  rather than duplicate the info in the package.json.

[glenn20]

@Josverl I'm more than happy to improve or update the docs for the recent package.json changes if you have any suggestions. The existing chages were focussed on updating the existing documentation (which was focussed on usage examples, rather than a full file format specification).

[Josverl]

I think that some additional docs may be in order as I failed to see how it could be used for local deployment, however try to outline the different use cases and solutions first.

[glenn20]

Well, the initial motivating use cases for PR #12476 and PR #12477 were:

• mip install from git repo on local machine, eg:

  git clone https://github.com/glenn20/micropython-esp32-ota.git
  mpremote mip install micropython-esp32-ota/package.json

  (Don't do this - this package is not setup for a local package.json file yet).

  This is also useful:

  • while developing your own mip packages. You can mpremote mip install them for testing during development without having to commit every change and push to origin.
  • Installing packages to devices from local clones when you are without internet access (eg. updating devices in the field).

• Support for relative URLs in the urls section is also useful for installing from mirrors or forks of package repos (the urls do not need to be rewritten in the fork). If the use of relative URLs was required, it would also reduce the surface for malicious or mistaken urls pointing to somewhere unexpected.

• Furthermore, I envisioned during development of my own packages, I could use a tool to download clones of my dependencies (mip packages) so that my DE would find them (eg. in the local .venv - like I do with your stubs packages). Then mpremote could install from the clones in the local venv. (That is the tool I am musing about below).

• Write your own collection of local package.json files for github packages which have not (yet) been mipified.

You can see some additional commentary on use cases in the discussions of the PRs.

[glenn20]

As suggested by @aivarannamaa above, the package.json does seem a good candidate for specifying the dependencies and project configuration of a micropython project.

Another approach might be to specify the dependencies in the pyproject.toml and regard the package.json as a build artifact (like the uv.lock file). The more that we can express in the pyproject.toml the less we need to do to provide specialised micropython support for the various IDEs and tools.

Main uses for the pyproject.toml file (in the cpython world):

1. Project metadata (for publication and building purposes) (not currently used in micropython package publishing)
2. Configuration for package building (eg build-system) (no build-step currently for micropython packages)
3. Configuration for virtual environments (dependencies (inc versions))
4. Tooling configuration (eg. tool.mypy, tool.ruff, tool.tox, ...)
5. Source of information for development environments (DEs):
   • Dependencies (including package type stubs) (via virtual envs)
   • Tooling configs (mypy, ruff, ...)

DEs generally get tooling configs from the DE configs (eg. .vscode/settings.json) and the tools pick up additional config from the pyproject.toml file (my own preference is to put all tool configs in the pyproject.toml so results are the same in the DE and in the cli and CI).

The new-ish [dependency-groups] table may be an alternative approach for specifying micropython-specific dependencies in the pyproject.toml file (rather than tool.mpremote) - but I haven't really thought that through.

For development in the cpython world, I use uv sync to build my virtual environment and resolve and install the dependencies (also builds a uv.lock file). These are then found by my DE for type checking, etc. This is also used to update the venv.

For development in micropython it would be useful to have a command which performs similar actions:

• download and install micropython type stubs (according to config in pyproject.toml)
• download and install dependencies into a local cache for use by the DE in resolving types, etc.
  • Optionally compile to mpy
• (possibly) generate the package.json file

When deploying the package during development, the package and dependencies could be installed from the local micropython "venv". Then I have an all-local environment I can preserve and rebuild for reproducible deployments.

If the package.json file is generated, then we could feasibly do things like update the version number and publishing metadata as part of CI and pre-commit tooling. It could also be possible to specify if the package.json file should use permissive or strict versioning (so it can be used as a lock file).

[Josverl]

pyproject.toml has my personal preference as well, but it should not be the only way , hence my suggestion to combine it with the existing package.json.

specifying stubs
I do that currently using project.optional-dependencies.
that is not understood by all front-ends , but uv-pip does grok it.

[project.optional-dependencies]
# install to folder typings
# uv pip install -r pyproject.toml --extra stubs --target typings 
stubs = [
  "micropython-esp32-stubs"
]

[dependency-groups]
my first idea was to use project-dependencies , but I assumed that would be quite complex and would require developing a complete packaging back-end.

with dependency groups this does require a more complex syntax, as essentially for each module it would need to specify that that module should only be deployed on micropython , likely using an environment marker

Something like requests == 2.8.* ; implementation_name ~= "micropython"

there is a reference imlementation to parse from .toml , so it would be simple to use for an experiment.

an important thing to not is that neither mpremote mip , not mini mip currently support the normal dependency specifiers.
This is something that I think should be improved , and omissions clearly documented.

[aivarannamaa]

If I'm not mistaken, all pyproject.toml standard locations for package names (including dependency groups) are assumed to reference PyPI namespace. This means MicroPython deps from micropython-lib or GitHub need to go under a [tool.something].

[byzantic]

I think that the new ROMFS facility adds something to the mix ..

I can envisage some tool taking a set of package descriptions plus (local) files, deploying them to a directory, then generating a ROMFS image of that directory. This could then be deployed to a device using mpremote.

The goal here would be some way to automatically build a deployable application using CI or on a build server.

As a further refinement, if there were a way to combine a .romfs image with an existing .uf2 or .bin file to produce a .uf2 file, then it would be straightforward to build an app that is easily deployable, or to provide board specific support.