Add support for constraints, install wheels before pyodide packages by bollwyvl · Pull Request #181 · pyodide/micropip

bollwyvl · 2025-01-27T22:29:10Z

references

fixes support constraints for transient dependencies? #175
split off from Add support for constraints, install in topological order #177

changes

mostly as on Add support for constraints, install in topological order #177
roll back the last few commits of Add support for constraints, install in topological order #177
- transaction.py
  - if constraints are present, keep gathering dependencies from Pyodide packages, but don't capture the entire graph
- install.py
  - install all external wheels before Pyodide packages
  - avoids loadPackage magically fulfilling the dependencies which may be in constraints

breaking changes

this changes many fewer signatures than Add support for constraints, install in topological order #177
there might be cases in the wild where the previous behavior was somehow working

for more information, see https://pre-commit.ci

…yodidegh-175-constraints

…de/wheel install order

bollwyvl · 2025-01-27T22:42:36Z

@ryanking13 @hoodmane this should Do The Thing without breaking as much!

ryanking13 · 2025-01-28T08:01:18Z

CHANGELOG.md

+- `micropip.install` now installs wheels from PyPI or custom indexes before built-in
+  Pyodide packages, reversing the previous behavior.
+  [#181](https://github.com/pyodide/micropip/pull/181)


Is there a user-facing change related to this? I think the order of installation is an implementation detail that is not visible to the user. What a user sees whether the whole set of packages (with dependencies) are installed or not.

Yeah, I don't know: seems like In interactive package management, which can be modified by about five different flags now, I think "reversing what we used to do," warrants at least a note. I am not sure if it's breaking, but things folk did before might not work now, and they might not need workarounds (like the double-install thing).

I see. Makes sense. Let me check the code more carefully and review.

I think the current wording could be a bit misleading to users though. When the same package exists in both the pyodide lock file and PyPI, the lockfile currently takes precedence (in terms of which package to choose, not the order of installation). This PR is not related to it, but maybe the user will think that this PR is changing that behavior.

Right: today, in the transaction phase once a dependency chain hits a pyodide package, further dependency resolution was skipped because the lockfile would take care of the rest. This was a performance win, for sure. In the install stage, they were all mixed together in wheel_tasks, and there was really no way to make any claims about what would happen.

With this change, the transaction will keep going all the way down for every novel dependency. In the install phase, the PyPI wheels get installed (in parallel) before calling loadPackage, which won't re-install (and re-download) a pyodide wheel.

This does mean there will be an extra add_dependency_inner for each depends item, but I couldn't figure out a way to do it that would be remotely correct.

... but that's a lot to say in a CHANGELOG, maybe there's a terser way?

tried some more things on 92baa16

With this change, the transaction will keep going all the way down for every novel dependency. In the install phase, the PyPI wheels get installed (in parallel) before calling loadPackage, which won't re-install (and re-download) a pyodide wheel.

Hmm, I still don't quite get what which won't re-install means. By default, during the dependency resolution step, micropip will check if the package is available in the lockfile, and select the one from the lockfile instead of the one from the PyPI.

So, in the installation step, we expect no overlap between transaction.pyodide_packages and transaction.wheels. Does this PR change this behavior?

Note the download of two pytest wheels:

After this PR, since the wheel would already be installed, it would look more like this:

Got it. I didn't consider putting multiple packages in install(...). Thanks.

bollwyvl · 2025-01-29T13:09:02Z

Will work on merge conflicts.

…raints-wheels-first

bollwyvl · 2025-01-30T14:58:10Z

Merging main, looks like something is broken, but can't reproduce locally:

    @pytest.mark.skip_refcount_check
    @run_in_pyodide(packages=["micropip"])
    async def test_load_binary_wheel2(selenium_standalone_micropip):
        from pyodide_js._api import repodata_packages
    
        import micropip
    
>       await micropip.install(repodata_packages.regex.file_name)

tests/test_install.py:329: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/lib/python3.12/site-packages/micropip/package_manager.py:146: in install
    ???
/lib/python3.12/site-packages/micropip/install.py:52: in install
    ???
/lib/python3.12/site-packages/micropip/transaction.py:71: in gather_requirements
    ???
/lib/python3.12/site-packages/micropip/transaction.py:84: in add_requirement
    ???
/lib/python3.12/site-packages/micropip/transaction.py:180: in add_requirement_inner
    ???
/lib/python3.12/site-packages/micropip/transaction.py:227: in _add_requirement_from_package_index
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   ValueError: Can't fetch metadata for 'regex-2024-9-11-cp312-cp312-pyodide-2024-0-wasm32-whl'. Please make sure you have entered a correct package name and correctly specified index_urls (if you changed them).

bollwyvl · 2025-01-30T15:10:48Z

Ah, it looks like the .whl is being canonicalized as a package name into -whl

-wasm32-whl

... but also no idea why it's not failing locally...

bollwyvl · 2025-01-30T18:56:56Z

Back to ✔️

ryanking13

Looks good to me overall. Thanks for your work!

ryanking13 · 2025-02-01T02:18:08Z

micropip/transaction.py

+        if self.verbose and messages:
+            for constraint, msg in messages.items():
+                logger.info("Transaction: constraint %s discarded: %s", constraint, msg)


In the long run, it seems better to change it so that the installation fails when constraints resolution fails (just like pip's behavior). However, micropip's current dependency resolution is incomplete on its own, so let's leave it for now.

bollwyvl · 2025-02-01T15:52:55Z

Thanks all!

agriyakhetarpal · 2025-02-01T16:03:33Z

Released in https://github.com/pyodide/micropip/releases/tag/v0.9.0, thank you!

bollwyvl and others added 17 commits January 23, 2025 09:44

strawman api/docs changes for pyodide#175

48295ce

update PR number

64bbbe8

apply constraints in transaction

e50f58c

add minimal test

57de68c

test set_constraints

7e4dde5

add set_constraints

dab4b42

[pre-commit.ci] auto fixes from pre-commit.com hooks

5b5b1c8

for more information, see https://pre-commit.ci

more tests

d5d2694

Merge remote-tracking branch 'origin/pyodidegh-175-constraints' into p…

03f6b0b

…yodidegh-175-constraints

handle more emfs cases

c61f4ad

address review comments

48340ed

update changelog

6402421

update docs

a6106d5

handle env markers and additive constraints

98d1faa

continue gathering requreiments from pyodide packages, reverse pydodi…

c59c754

…de/wheel install order

fix pr number in changelog

c47ee41

fix spelling

32d3836

bollwyvl closed this Jan 27, 2025

bollwyvl reopened this Jan 27, 2025

bollwyvl marked this pull request as ready for review January 27, 2025 22:40

bollwyvl mentioned this pull request Jan 27, 2025

Add support for constraints, install in topological order #177

Closed

8 tasks

tighten up utils, docstrings

a1ff4e2

ryanking13 reviewed Jan 28, 2025

View reviewed changes

ryanking13 mentioned this pull request Jan 29, 2025

New 0.9 release #184

Closed

agriyakhetarpal added this to the 0.9.0 milestone Jan 29, 2025

bollwyvl added 2 commits January 29, 2025 07:23

merge upstream

7f88152

skip an await if not needed, rework changelog entry

92baa16

bollwyvl requested a review from ryanking13 January 29, 2025 14:36

bollwyvl mentioned this pull request Jan 30, 2025

Allow to install extra packages from extra indexes from jupyter-lite.json jupyterlite/pyodide-kernel#158

Closed

Merge remote-tracking branch 'upstream/main' into pyodidegh-175-const…

d45f62d

…raints-wheels-first

try more url checking

00e6e7a

ryanking13 approved these changes Feb 1, 2025

View reviewed changes

ryanking13 merged commit c9221a0 into pyodide:main Feb 1, 2025

bollwyvl deleted the gh-175-constraints-wheels-first branch February 1, 2025 15:52

bollwyvl mentioned this pull request Feb 1, 2025

Support constraints in piplite jupyterlite/pyodide-kernel#165

Open

5 tasks

ryanking13 mentioned this pull request Feb 8, 2025

micropip claims it's installing typing-extensions 4.12.2, but it's not #201

Closed

Uh oh!

Conversation

bollwyvl commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

references

changes

breaking changes

Uh oh!

bollwyvl commented Jan 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanking13 Jan 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bollwyvl commented Jan 29, 2025

Uh oh!

bollwyvl commented Jan 30, 2025

Uh oh!

bollwyvl commented Jan 30, 2025

Uh oh!

bollwyvl commented Jan 30, 2025

Uh oh!

ryanking13 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bollwyvl commented Feb 1, 2025

Uh oh!

agriyakhetarpal commented Feb 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bollwyvl commented Jan 27, 2025 •

edited

Loading

ryanking13 Jan 29, 2025 •

edited

Loading