Add repodata support for filenames in the wheel format

[danyeaw]

What is the idea?

Add support for solvers getting filenames in the {distribution}-{version}-py3-none-any.whl format that pure-Python wheels come in and successfully get the package name and version.

Why is this needed?

Currently rattler and libmamba-solver support conda filenames in the format of {name}-{version}-{build_string}.tar.bz2. As we add wheel filenames to repodata so that users can conda install <wheel>, they won't be in the format that the solver expects and the solver will fail to parse them.

What should happen?

Update rattler_repodata_gateway to accept {distribution}-{version}-py3-none-any.whl filenames and properly get the name and version information. In order to standardize this, a CEP will also probably be required.

Additional Context

A future issue will need to track implementing this in libmamba-solver as well.

[travishathaway]

@danyeaw,

Some good news on this front, I have verified that this is actually possible and this experimental pull request proves it:

• experimental: add support for wheel packages to repodata parsing rattler#1918

The other good news is that I didn't have to change anything with the solver. All the work for this was done in the rattler_repodata_gateway crate. Thankfully, the solver doesn't actually parse the file names. It instead relies on a much better PackageRecord abstraction 😌.

Here's how you can test it out yourself (assuming you have a cloned copy of rattler already):

# Checkout my experimental branch
gh pr checkout 1918 && cd py-rattler

# Build a local copy
pixi run build

# Launch a test shell
pixi shell -e test

# Start a python interpreter to run the code in the next section...
python

Make sure to first download the two attached repodata files (conda_repodata.json and whl_repodata.json)

import asyncio
from pprint import pprint
from rattler import (
    MatchSpec,
    Channel,
    PackageName,
    Version,
    SparseRepoData,
    GenericVirtualPackage,
    solve_with_sparse_repodata
)

# This can be anything because we don't actually load from here
chn = Channel("http://localhost")

# Define our sparse repodata using mock data
sparse_whl = SparseRepoData(chn, "noarch", "whl_repodata.json")
sparse_conda = SparseRepoData(chn, "osx-arm64", "conda_repodata.json")

# Define mock virtual package records
virtpkg = GenericVirtualPackage(PackageName("__osx"), Version("15"), "0")
virtpkg_unix = GenericVirtualPackage(PackageName("__unix"), Version("15"), "0")

# This works 🎉 
solution = asyncio.run(solve_with_sparse_repodata(["idna"], [sparse_whl, sparse_conda], virtual_packages=[virtpkg, virtpkg_unix]))

pprint(solution)

You can try with other packages in the whl_repodata.json but the solves fail for most because of missing packages, but you do get a meaningful solve error, and that means the solve is working correctly and taking the wheel package into account.

I did run into one edge already though. The name of the typing-inspection package differs. In the wheel file, it uses an underscore, but in the package record itself, it uses a dash (e.g. typing_inspection vs typing-inspection. That just seems a like a bug we would iron out in the final implementation though.

I'm hoping this work will be useful as we move towards finding the best solution here.

conda_repodata.json
whl_repodata.json

[danyeaw]

Hey Travis, amazing work creating a prototype for this! I ran through the example and it worked great for me as well.

We do need to figure out the name mapping for typing_inspection. The context for anyone coming across this is that there are name differences between PyPI and conda for the same packages. There is also a few name differences between conda-forge and defaults channels. @ryanskeith did an analysis of this which is posted here: https://conda-incubator.github.io/conda-pypi/reference/conda-channels-naming-analysis/

Right now the wheels repodata is not being generated by conda-index (we do need to get to updating conda-index next quarter). @aovasylenko implemented a static name mapping similar to the one in this repo at conda_pypi/grayskull_pypi_mapping.json.

When I try to install typing_inspection or typing-inspection, I get a failure:

>>> solution = asyncio.run(solve_with_sparse_repodata(["typing_inspection"], [sparse_whl, sparse_conda], virtual_packages=[virtpkg, virtpkg_unix]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dyeaw/Projects/rattler/py-rattler/.pixi/envs/test/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/Users/dyeaw/Projects/rattler/py-rattler/.pixi/envs/test/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/Users/dyeaw/Projects/rattler/py-rattler/rattler/solver/solver.py", line 189, in solve_with_sparse_repodata
    for solved_package in await py_solve_with_sparse_repodata(
exceptions.SolverException: Cannot solve the request because of: No candidates were found for typing_inspection *.

I think we need to work with the PREX team to determine if the mapping is behaving like we expect and if something else is needed in rattler to manage this when mapped name is provided in the name field.

[dholth]

I did run into one edge already though. The name of the typing-inspection package differs. In the wheel file, it uses an underscore, but in the package record itself, it uses a dash (e.g. typing_inspection vs typing-inspection. That just seems a like a bug we would iron out in the final implementation though.

In Python packaging, names (in the Python metadata) are normalized to have only hyphen -: https://packaging.python.org/en/latest/specifications/name-normalization/#name-format, but in the wheel filename names are normalized to have only underscore _. This allows the wheel filename to be parsed by splitting on -.

As far as we can tell the typing-extensions, typing-inspection, and confusingly typing_extensions package names have a correct mapping between conda and Python.

[danyeaw]

Duplicate of conda/rattler#1967.