Abillity to split pixi.toml into multiple files

[jleibs]

Problem description

We have a large, and growing pixi.toml file

It has grown to the point where some kind of hierarchical file-based organization would be helpful.

There are clearly different ways to partition: by function such as task vs dependency, by environment, by project sub-directory, including dependencies and tasks related to a given part of the code-base.

However, at a minimum just a way to include other files so create a pixi.toml that is a concatenation of all the included files might help bring a little bit of organization to things.

[niemiaszek]

Interesting. I really appreciate the possibility to keep all my env-related stuff in one file instead of having multiple environment.yml's like I did with conda before. Moreover, possibility of bundling Pixi into pyproject.yml is also nice for python-only projects.

Would be interesting to see work on such a large project

[bollwyvl]

Yes, lovely stuff, and could potentially help reduce duplication in matrix use cases, and enable monorepos and skeleton-based tools to get spun up on pixi more quickly.

As long as there is a way to get a single inspectable file back out, fully realized, things are pretty ok. Otherwise, behold the nightmare that is GitHub actions uses or, worse still, the GitLab pipeline spaghetti: can't know until you try, and have already spun up 20 containers!

concatenation

Yep, TOML is quite resilient to cat. I've explored not actually checking in pixi.toml and having a wrapper script which doesn't need pixi... cleaned up a bit, my "run"-focused pxr:

#!/usr/bin/env bash
cat pixi.base.toml src/*/pixi.partial.toml > pixi.toml

set -eux
for task in "$@"; do
    pixi run $task
done

Writing those files can be a bit hard to reason about, however, as all those paths need to be pixi.toml-relative.

include

As with the inputs/outputs feature, task could provide some inspiration with its top-level includes which can be a simple path string or a rich object.

Typographically, this could be adaptable here, assuming a reserved namespace delimiter, e.g. ::

[project]
includes.docs = "docs/pixi.toml"
includes.atest = { manifest = "atest/pixi.toml", features = ["deps-atest"] }

[project.includes.ff]
manifest = "atest/pixi.toml"
features = ["tasks-atest"]
env = { BROWSER = "firefox" }

[project.includes.cr]
manifest = "atest/pixi.toml"
features = ["tasks-atest"]
env = { BROWSER="chromium" }

[tasks.all]
depends_on = ["docs:build", "ff:robot", "cr:robot"]
inputs = [
  "build/docs/.buildinfo", 
  "build/reports/robot/chromium", 
  "build/reports/robot/firefox",
]

[environments]
docs = ["docs:deps-docs", "docs:tasks-docs"]
atest = ["atest:deps-atest", "ff:tasks-atest", "cr:tasks-atest"]

Dangerously close to going down the spaghetti route of "post-install hooks": if an include could be declared as coming from an environment, all of a sudden partial pixi workflows could be packaged like proper software:

[project.includes.ex]
manifest = { environment = "default", path = "share/pixi/tasks/example.com/pixi.toml" }
env = { EX_PROJECT_NAME="my-project" }

[feature.baseline.dependencies]
example-com-pixi-tasks = "2024.04.*"

Having been down this road, it gets pretty hairy if a child wants to reference its parent files, features, tasks, etc. This could either be banned (which would limit the overall functionality) or would require some more syntax to make an includable partial contract more explicit and self-documenting, e.g.

[partial]
parent = "root"

# just use JSON Schema: don't get fancy
[partial.env.EX_PROJECT_REPO]
description = "the location of repo"

[partial.env.EX_PROJECT_NAME]
description = "a lovely project name according to your friendly example.com company policies"
pattern = "[a-z][a-z/d\\-]+"

[ruben-arts]

Looking at Rerun's pixi.toml, this would definitely benefit from a split.

Since the pixi project started we've thought about a "Workspace" feature. Which would allow you to have multiple manifest files which can be included in the main manifest file. Taken huge inspiration from https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html and https://docs.npmjs.com/cli/v7/using-npm/workspaces
Although we don't have a design for this laying around the question would be if that fixes the issue for you? If you think it will I would like to do a full write-up.

Quick summary on the workspace feature:

• Having member packages in subfolders
• Having a member environments specifically for the member but shared dependencies.
• Running tasks from the members from the root entrypoint.
• Depending on members as actual packages which can be shared and cached during development. (requires pixi build)

I know we're being credited for not splitting our configuration into multiple files which is something I take seriously as well.

[bollwyvl]

With a SAT hammer sitting around, it's nice if everything is a package nail, but needing build a as an extra, hidden depends_on of install:b to be used by run c would feel pretty heavy for something that is a another aspect of a project (a la docs, heavyweight tests). Further, it seems that anything touching an env (and presumably the pixi.lock) has the potential to degrade CI cacheability until something can generate bit-for-bit fully-reproducible .conda artifacts (which i have yet to see).

[jleibs]

Although we don't have a design for this laying around the question would be if that fixes the issue for you? If you think it will I would like to do a full write-up.

Yeah, workspaces very much sounds like a direction that would work well for us. This one definitely isn't urgent at all but came up when I was talking to Tim and Bas last week so I opened an issue to keep from forgetting about the idea.

[ruben-arts]

I would personally not be in favor of making a split manifest functionality without additional features(like workspaces) that can benefit from this as it would add additional complexity to both the pixi code base and when misusing this, also to the projects them self.

[baszalmstra]

This is a really good idea but Im not sure when we'll get to it. For now I moved this to the ideas discussion board to not clutter the issues.

[cheshirex]

I've just discovered Pixi and am in the process of converting some existing projects to this. I was looking for exactly this functionality, since my project is actually split between two different environments, and I want to ensure CI will only run for the relevant environment. Since my CI setup is triggered by which file(s) are modified, being able to split the pixi.toml into two (or three, with a common parent) files would be perfect.

I would be fine with concatenating the files, if there's a way to tie it into the pixi command so I don't need to manually generate the resulting pixi.toml before running a given command...

Thanks!

[cheshirex]

Actually, it occurs to me that wanting to tie it in to CI would require also splitting the pixi.lock file, so that each environment could be monitored separately. I imagine that might be trickier...

[bollwyvl]

pixi build workspaces pretty much handle a lot of the multi-pixi.toml workflows, but having a single pixi.toml still can be beneficial for providing a high-level, single-file description of a project shared by a whole team. I haven't used this feature in anger, but believe these result in a single, root-level pixi.lock.

In the single-file model, pixi.lock only changes when pixi.toml#/*/dependencies change. This is typically "good enough," for CI caching, but does many any environment change will cause a total cache miss.

For this specific concern, the output of pixi project export {kind} {output} as a cache key. Specifically the pixi project conda-explicit-spec {some-folder} will generate reproducible, topologically sorted environment solves (modulo virtual packages) that won't change based on irrelevant pixi.toml changes (like tasks), or, generally, the version of pixi, but almost certainly will change on a solve of a relevant environment in pixi.toml (and therefore pixi.lock).

These could be done offline, with the results checked in (not terrible), or regenerated as a preflight job.

The velocity of pixi.toml changes can be brought down even further by putting most business logic in scripts appropriate to the team's environment (e.g. .sh, .py) and stored in the subproject of interest so that existing tools would work.

[cheshirex]

I like that. We were essentially doing that with our conda envs before. So now instead of those being permanent files, we could generate them on the fly. Still a bit to work out as to how to use that, but it gives a direction to move forward. Thanks!

[vincent-pollet]

I would also be interested in this feature! My team is developing a lot of packages (~60) that have common structure and development tools and therefore similar pixi manifests. We would love to be able to include the common parts of the pixi.toml (eg test, coverage, documentation tasks) from a single place, possibly remote or from a package, to reduce duplication. It would be similar in spirit to how CI/CD components of gitlab allow us to not duplicate the CI definition.