Make mip record package info

[aivarannamaa]

I propose introducing a metadata file for recording information about packages installed by mip (at least by the mpremote variant).

The file would sit in the target folder (e.g. /lib) and cover information about all packages installed into this folder.

For example, mip install github:andyrids/micropython-networkutils/networkutils would create following file:

[
  {
    "package": "github:andyrids/micropython-networkutils/networkutils",
    "requested_version": "latest",
    "resolved_version": "1879d6ec38d4392178dcd3bc8d46f54120d3d1d4",
    "metadata": {
      "urls": [
        [
          "networkutils/core.py",
          "https://gitlab.com/micropython-iot-projects/libraries/micropython-networkutils/-/raw/HEAD/networkutils/networkutils/core.py"
        ]
      ],
      "deps": [
        [
          "logging",
          "latest"
        ],
        [
          "github:josverl/micropython-stubs/mip/typing.mpy",
          "main"
        ],
        [
          "github:josverl/micropython-stubs/mip/typing_extensions.mpy",
          "main"
        ]
      ],
      "version": "0.0.1"
    },
    "files": [
      {
        "path": "networkutils/core.py",
        "sha256": "0cb5ddeb0cf4d25d3d597780d0054479041208bfcf08c18b3c67c046c9276601"
      }
    ]
  },
  {
    "package": "logging",
    "requested_version": "latest",
    "index": "https://micropython.org/pi/v2",
    "resolved_version": "0.6.2",
    "metadata": {
      "version": "0.6.2",
      "hashes": [
        [
          "logging.py",
          "ba7c1bf2"
        ]
      ],
      "v": 1
    },
    "files": [
      {
        "path": "logging.py",
        "sha256": "ba7c1bf28a494d22680556cf7565fc15cef30c74fdf5077140e39732dc3b42a8"
      }
    ]
  },
  {
    "package": "github:josverl/micropython-stubs/mip/typing.mpy",
    "requested_version": "main",
    "resolved_version": "96d06f782a17419189ed9c164ebd36a977cdf025",
    "files": [
      {
        "path": "typing.mpy",
        "sha256": "c64c2de9d05d3ae432c121f453cd9a17712fed874c8c03b810c89315e4c6e6c0"
      }
    ]
  },
  {
    "package": "github:josverl/micropython-stubs/mip/typing_extensions.mpy",
    "requested_version": "main",
    "resolved_version": "96d06f782a17419189ed9c164ebd36a977cdf025",
    "files": [
      {
        "path": "typing_extensions.mpy",
        "sha256": "286bb50dd7e072415338d917c01adc16a56d98eb79aa02f5bac13c15ed65de40"
      }
    ]
  }
]

Here is how one could benefit from this:

• This allows implementing mip list and mip uninstall. You could easily look up the version of your installed dependency.
• This allows creating tools for synchronising /lib folder with package.json. If each "deps" entry is represented in this metadata file, the tool can tell you quickly that your device is up to date.
• Some people may want to add this file into their project's git repo (maybe together with the py-files of all dependencies). This would effectively lock the dependencies.

I have already completed an implementation of this idea, but I haven't created the pull request yet.

The implementation may have bugs, but the big features are done:

• list and uninstall work. After uninstall removes the last remaining package, the metadata file (currently named as packages-info.json) gets removed as well.
• All installation sources currently supported by mip (github, gitlab, index, local json, py or mpy url) are supported and properly recorded.
• When given a github or gitlab package, install looks up and records the hash of the latest commit in required branch (or HEAD).

Looking forward to hear your comments!

[Josverl]

Looks interesting and it would be good to have reproducible installs.
A few questions if I may

• What is the logic/contract behind versions and hashes?
• how does it record installs from a local/private source?

[aivarannamaa]

What is the logic/contract behind versions and hashes?

I assume your question is about this part:

    "package": "github:andyrids/micropython-networkutils/networkutils",
    "requested_version": "latest",
    "resolved_version": "1879d6ec38d4392178dcd3bc8d46f54120d3d1d4",

When installing github:andyrids/micropython-networkutils/networkutils@<branch-name>, mip looks up the latest commit in this branch and uses its hash as "resolved_version". In this example, one can later install github:andyrids/micropython-networkutils/networkutils@1879d6ec38d4392178dcd3bc8d46f54120d3d1d4 and get the same version even if the repository has moved on.

It may make sense to record short (7-symbol) hashes instead -- this would make the metadata file a bit smaller and mip list output a bit cleaner. That's something to discuss.

When the version is omitted from the install request or explicitly given as "latest", mip looks up the latest commit in the default branch (equals to using HEAD as the branch name).

When a tag is given as version, it will be resolved as is.

Note that there is not much to do with the "version" given in package.json of the installed package. It may or may not match with the tag name of the commit. mip only accepts commit hash, tag or branch name as version specifier for installing github/gitlab packages.

how does it record installs from a local/private source?

Good question! I was considering different options and finally chose to record the absolute path to the local source.

[Josverl]

Thanks,

How does your proposal compare to PEP 751 lock files?

Not saying that we should blindly follow that, but I would like to understand why not, or if there could be a Micro version of that, before establishing a new standard.

[aivarannamaa]

A PEP 751 lock file does not record which package installed which files. This could be solved by extending the format, though. From the other side -- pylock.toml is designed for more complex packaging system so that considerable part of the features would remain unused in MicroPython context.

Another reason is that MicroPython already uses a JSON file (package.json) and it also has JSON parser in stdlib, so a TOML file would feel a bit alien in this context.

It would be worthwhile to mimic CPython's lockfile if the rest of MicroPython packaging tried to mimick CPython's approach, but in current situation it might be even confusing if MicroPython adapted pylock.toml.

Of course, beside formatting, pylock.toml (or uv.lock or some other lockfile from CPython world) may have some features or principles, which are worth taking over. Do you have something in mind?

[peterhinch]

Consider the case where multiple packages share the same dependency. How does uninstalling one of the set deal with the dependency?

[aivarannamaa]

Only the packages directly given in unistall command get removed. The dependecies remain installed. pip also works this way.

[aivarannamaa]

I now changed the top level of the metadata format a bit (added extra dict with "packages" key for future extension and turned list of specs into dict of specs):

{
  "packages": {
    "github:andyrids/micropython-networkutils/networkutils": {
      "requested_version": "latest",
      "resolved_version": "c68d57aa5ceef22289a72b316b5e26eb56715786",
      "metadata": {
        "urls": [
          [
            "networkutils/core.py",
            "https://gitlab.com/micropython-iot-projects/libraries/micropython-networkutils/-/raw/HEAD/networkutils/networkutils/core.py"
          ]
        ],
        "deps": [
          [
            "logging",
            "latest"
          ],
          [
            "github:josverl/micropython-stubs/mip/typing.mpy",
            "main"
          ],
          [
            "github:josverl/micropython-stubs/mip/typing_extensions.mpy",
            "main"
          ]
        ],
        "version": "0.0.1"
      },
      "files": [
        {
          "path": "networkutils/core.py",
          "sha256": "8029c9073ed34c9bcab5e810ce032ea631ad952db112be8456fe861f449b8016"
        }
      ]
    },
    "logging": {
      "requested_version": "latest",
      "index": "https://micropython.org/pi/v2",
      "resolved_version": "0.6.2",
      "metadata": {
        "version": "0.6.2",
        "hashes": [
          [
            "logging.py",
            "ba7c1bf2"
          ]
        ],
        "v": 1
      },
      "files": [
        {
          "path": "logging.py",
          "sha256": "ba7c1bf28a494d22680556cf7565fc15cef30c74fdf5077140e39732dc3b42a8"
        }
      ]
    },
    "github:josverl/micropython-stubs/mip/typing.mpy": {
      "requested_version": "main",
      "resolved_version": "30b34ee3d54b0a349b7ef87fd7daf863cc14e0e4",
      "files": [
        {
          "path": "typing.mpy",
          "sha256": "c64c2de9d05d3ae432c121f453cd9a17712fed874c8c03b810c89315e4c6e6c0"
        }
      ]
    },
    "github:josverl/micropython-stubs/mip/typing_extensions.mpy": {
      "requested_version": "main",
      "resolved_version": "30b34ee3d54b0a349b7ef87fd7daf863cc14e0e4",
      "files": [
        {
          "path": "typing_extensions.mpy",
          "sha256": "286bb50dd7e072415338d917c01adc16a56d98eb79aa02f5bac13c15ed65de40"
        }
      ]
    }
  }
}

Also, I changed the file name to "mip-packages.json" (instead of "packages-info.json" proposed earlier). The more specific name is better, when other tools (e.g. circup) also start adding their metadata files.

Also, I changed the policy of uninstalling old versions when installing a package which has been already installed. Now the old version is uninstalled only if the user explicitly wants to install latest version or when the requested and installed versions are different. This leaves room for introducing version ranges for package specifications in the future -- it is not hard to turn this approach into a simple non-tracking version resolution procedure.

After these changes I created the pull request: #17646