Publishing a package for mip install. Has anyone succeeded?

[peterhinch]

I have attempted to follow the instructions here producing package.json reproduced here:

{
  "urls": [
    ["gateway.py", "github:peterhinch/micropython-mqtt/blob/master/mqtt_as/esp32_gateway/gateway.py"],
    ["gwconfig.py", "github:peterhinch/micropython-mqtt/blob/master/mqtt_as/esp32_gateway/gwconfig.py"],
    ["mqtt_as.py", "github:peterhinch/micropython-mqtt/blob/master/mqtt_as/mqtt_as.py"],
    ["mqtt_local.py", "github:peterhinch/micropython-mqtt/blob/master/mqtt_as/mqtt_local.py"],
    ["primitives/__init__.py", "github:peterhinch/micropython-mqtt/blob/master/mqtt_as/esp32_gateway/primitives/__init__.py"],
    ["primitives/ringbuf_queue.py", "github:peterhinch/micropython-mqtt/blob/master/mqtt_as/esp32_gateway/primitives/ringbuf_queue.py"]
  ],
  "deps": [
    ["collections-defaultdict", "latest"],
    ["aioespnow", "latest"]
  ],
  "version": "0.1"
}

Attempts to install produce this outcome:

>>> import mip
>>> mip.install("github:peterhinch/micropython-mqtt/blob/master/mqtt_as/esp32_gateway/package.json")
Installing github:peterhinch/micropython-mqtt/blob/master/mqtt_as/esp32_gateway/package.json to /lib
Package not found: github:peterhinch/micropython-mqtt/blob/master/mqtt_as/esp32_gateway/package.json
Package may be partially installed
>>> 

What am I doing wrong?

[mcauser]

Looking at the code in https://github.com/micropython/micropython/blob/master/tools/mpremote/mpremote/mip.py#L53

It's trying to rewrite the path so it can be accessed via raw.githubusercontent.com
github:peterhinch/micropython-mqtt/blob/master/mqtt_as/esp32_gateway/package.json -> https://raw.githubusercontent.com/peterhinch/micropython-mqtt/master/mqtt_as/esp32_gateway/package.json

Looks like it wants you to go:
mpremote mip install github:peterhinch/micropython-mqtt/master/mqtt_as/esp32_gateway/package.json

Try dropping the /blob from the urls in your package.json

[mcauser]

Mine works: https://github.com/mcauser/micropython-tm1637/blob/master/package.json - but it's very simple and doesn't use branches or paths, just a package.json in the root.

[peterhinch]

Thanks for that, I now have something that installs. This could do with being documented as it seems to assume some knowledge of GitHub's directory structure.

It's made a mess of my directory structure by installing to /lib rather than / but I'm sure I can fix that.

[mcauser]

I think putting a leading slash in the first url gives it an absolute path, otherwise it defaults to installing into /lib.

Edit: nope, to install a file in the root dir, you have to go ../file.py
ie. /lib/../file.py

[scruss]

tbh, I'd be surprised/disappointed if a package installed through mip didn't end up in /lib. It's where libraries live.

[peterhinch]

Agreed - but it's questionable whether it's the right place for applications.

[peterhinch]

@mcauser

Edit: nope, to install a file in the root dir, you have to go ../file.py
ie. /lib/../file.py

That's not pretty :) I've changed the app to use relative imports. Running it is unchanged whether it's in / or /lib; my practical problem is now fixed :)

It strikes me we're left with two questions:

1. Should there be a more elegant means of installing applications to /?
2. How to document the necessary URL-wrangling - removing /blob/master in my case.

[Josverl]

Have you tried mpremote mip install --target / to specify where the package should be installed.

That works for installing multi file packages not located in the root of a repo

See https://github.com/Josverl/micropython-stubber/tree/board_stubber/mip
for my notes on getting this to work

Note that all files, must be from the same branch. Current mip does not allow mixing branches, including dependencies... There is a PR for that, but waiting for more votes on whether that would be needed.

Also be aware that not all ports include /lib in sys.Path. Can't remember which port, but it had me scratching my beard for some time

[scruss]

1. Should there be a more elegant means of installing applications to `/`?

Aside from the --install suggestion, I think this issue hinges on the difference between packages (= mip's intended installation type) and applications. For me, the former lives in /lib (or some non-root expression of sys.Path) and the latter is something I could replace /main.py with.

[jimmo]

hi @peterhinch

1. Yes I agree that the github: scheme is a bit subtle in that it's rewriting it into the https URL with the blob/branch etc. What you were doing would work if you used the full https URL. I will update the docs.

2. The target option is the intended way to install to somewhere other than /lib. (What it actually does is find the first entry in sys.path that ends with /lib, because you might have your lib dir mounted somewhere else).

@scruss already mentioned you can use mpremote mip install --target / ... but this also works with the on-device version >>> mip.install(..., target="/")

Do you think it's worth adding some metadata to a package to say what it's default target directory should be? I hadn't really considered mip being used to install the main "application", but it seems like a useful enough thing to be able to do?

4. As @Josverl I also plan to add the support for mixing branches. I think @mcauser raised this yesterday too. We need to make it so you can specify the branch for any individual target in the urls section (like you can with deps).

[peterhinch]

Point 1. definitely needs documenting as it's quite confusing.

I think a great mip use case is installing applications, especially where these involve multiple files and subdirectories. I also think that the natural home for applications is /. I'm agnostic on how such an installation is achieved so long as it's documented: mip.install(..., target="/") is fine by me.

mip is a great tool once you've been taught how to use it :)

[DavesCodeMusings]

Do you think it's worth adding some metadata to a package to say what it's default target directory should be?

I can see this as a useful feature. Could it be as simple as:

• Any file in package.json with an absolute path gets installed to that path
• Any file with a relative path gets installed to /lib
• Using target='/dir' prepends /dir to every path

For example:

• /myprogram.py gets installed to /myprogram.py
• /data/myfile.yml gets installed to /data/myfile.yml
• mymodule.py gets installed to /lib/mymodule.py
• /lib/mymodule.py would also install to /lib/mymodule.py if you wanted to be explicit about it.

With a target=foo, it would be:

• /myprogram.py gets installed to /foo/myprogram.py
• /data/myfile.yml gets installed to /foo/data/myfile.yml
• mymodule.py gets installed to /foo/lib/mymodule.py

[peterhinch]

@jimmo Using target="/" works: the application installs and runs.

However the directory structure is not ideal as dependencies are also installed to /. I think deps libraries from micropython-lib should be installed to /lib. I wonder if there's ever a need to override this?

[jimmo]

Thanks @peterhinch -- I will have a think about how to solve this.

@DavesCodeMusings I think your suggestion about using the path is probably a good way to do this. One note is that on different boards the default location is different. e.g. /flash on stm32, vs / on esp32, but I'm sure we can address this.

[Josverl]

Or a boot.py-mounted sd card

[aivarannamaa]

Perhaps the default target could be dynamically changed to "/" (or "/flash") if "main.py" or "boot.py" is present in the package? Also, compute the default target per package, such that the app-package ends up in "/" and dependencies (lib-packages) in "/lib"?

[jkorte-dev]

When trying to set up a mip package for my project I encountered a few problems:

1. I wanted to install all dependencies under /lib and my package (files) under / . The only solution I found was to create 2 package.json files: one for the package and one for the dependencies and install them separately. Solutions to this problem have already been proposed in this discussion. I vote for DaveCodeMusings idea.

2. One of my dependencies is not available via micropython-lib (a nrf24l01 driver with auto acknowledge and dynamic payloads support). I could define it as part of my package via url, but I want to install it as an mpy file depending on the micropython version used. This is only possible if I set up my your own repository. However, it is not possible to specify a path to a custom repository index for dependencies. Therefore I had to outsource the dependency and install it separately with mpremote mip install --index ....

The ability to specify the source index for dependencies would have solved my problem 2.

e.g as an optional additional field after the version

{
  "urls": [ ],
  "deps": [
    [ "nrf24", "latest", "https://raw.githubusercontent.com/jkorte-dev/mpy-dtu/master/nrf24" ]
}

[FoxyHunter7]

https://docs.micropython.org/en/latest/reference/packages.html

From what I can see in the documentation, it should be possible, given that repo is setup to be like a MIP package of course.

[jonnor]

Regarding the "application" versus "library" distinction and install directories, that also came up when @vshymanskyy was working on ViperIDE support for installing example code and also data files for "projects". Ref vshymanskyy/ViperIDE#42