Proposal: New backend for combined build of Conda (.conda) + Python (.whl, tar.gz) artifacts

[millsks]

Summary:

I’d like to propose a new build backend that produces both Conda packages (using Rattler) and Python wheels/sdists (using Hatchling).

Motivation:

This would streamline packaging for projects that need to distribute both Conda and PyPI artifacts from a single backend. This would also be a stepping point to also include build steps for projects also include front-end (node.js) that would also need to be built.

Proposed approach:

• Use the dependencies from the pixi manifest as a single source of truth to build both the python files and the conda files. The use of the compressed mapping from the parselmouth repository would be required?
• Use Hatchling for wheel/sdist builds
• Use Rattler for Conda builds. Possibly use the path to the wheel or tarball to build from instead of the path to the source. This could help when it comes to building the recipe if feedstock was needed to include in conda-forge?
• Output all artifacts to the build directory

Questions:

• Is this in scope for pixi-build-backends?
• Any design considerations or pitfalls I should be aware of?
• If the combined build capabilities do not align with the overall vision of pixi-build-backends, then does there exist the opportunity to run the pixi build sequentially allowing a user to matrix multiple build backends on a CI pipeline?

[millsks]

@rxm7706 had opened issue prefix-dev/pixi-build-backends#254 also asking about Node.js build backend for Python projects. Linking it here for discussion purposes.

[lucascolley]

Use Hatchling for wheel/sdist builds

We should be able to generalise over all possible pyproject.toml backends, using https://build.pypa.io/en/stable/.

---

I think the main question here is whether it makes sense to produce something other than a conda package with pixi build, or whether it should require a flag or other (sub)command.

If we did go for something other than pixi build being needed, there is the question of whether this should be rolled into pixi-build-python, or something else that can be used in combination with or as an extension to pixi-build-python.

[millsks]

It might make more sense to keep it as an extension for the python backend. Give the end users the extra capabilities that they can configure in the build section. I like this approach.

[lucascolley]

Use the dependencies from the pixi manifest as a single source of truth to build both the python files and the conda files. The use of the compressed mapping from the parselmouth repository would be required?

I'm not sure where parselmouth comes into this? If I understand correctly, we could build wheels/sdists in environments entirely made up of conda dependencies.

[lucascolley]

can you clarify whether you are thinking about different conda/pypi names for:

1. the dependencies of our package, or
2. our package itself?

For case (1), I think we can get all dependencies as conda packages, even when we want to build wheels/sdists. For case (2), we could just have a field like package.wheel.name or similar — unless you are thinking this would be better in one central location rather than localised to each package?

[millsks]

The dependencies of the project/package. For example, redis in conda-forge is redis-py and on pypi it is just redis.

https://anaconda.org/conda-forge/redis-py

Id rather us list the dependencies as conda packages, but would need a way to associate with the correct package when we build the wheel and the sdist tarballs?

[lucascolley]

ohh, maybe I understand what you mean — are you trying to avoid listing dependencies in pyproject.toml?

[millsks]

Correct. We are looking for a "single source of truth" when it comes to the dependencies so we dont have to sync packages in multiple places. If we could store it in the pixi configuration and then have the dependencies needed for the Python project be added dynamically to the project.dependencies during the build.

I believe this is a similar approach with the current pixi-build-python backend using recipe-stage0? It will generate the recipe file needed to kick off the build of the conda package? Is my understanding correct?

[lucascolley]

@nichmor maybe you have some input :)

[lucascolley]

does there exist the opportunity to run the pixi build sequentially allowing a user to matrix multiple build backends on a CI pipeline?

I am working on out-of-tree builds right now, which will provide at least a workaround for this. Basically you could have multiple Pixi manifests pointing to the same source, and build different packages in each.

[millsks]

Is this the scipy repo you pointed me at in the other discussion? I'd love to take a look and see how you are accomplishing this.

[lucascolley]

right now I am working on unblocking the SciPy repo at #4240, yes.