rattler-build: frequent rebuilds

[burdiyan]

I'm using pixi 0.55.0 with these in the pixi.toml file:

[workspace]
preview = ["pixi-build"]

# ...
# ...

[package.build.backend]
name = "pixi-build-rattler-build"
version = "*"

[dependencies]
please = { path = "./build/pixi-packages/please" }
# ...

In ./build/pixi-packages/please I have a source dependency — a rattler-build recipe, which is a repackaging of a particular tool I need, roughly following the Repackaging example in rattler-build docs.

I then add this package as a dependency in my workspace pixi.toml file.

The problem I'm having is that a simple touch of the recipe file would trigger a rebuild during my next interaction with pixi. So sometimes when I switch branches, or somehow the recipe file gets touched, whenever I do something in pixi the entire recipe gets rebuilt, which in this case takes a while.

Reproduction Steps

Create the following files in an empty directory:

• pixi.toml

[workspace]
name = "repro"
channels = ["https://prefix.dev/conda-forge"]
platforms = ["linux-64", "osx-64", "osx-arm64", "win-64"]
version = "0.0.0"
requires-pixi = ">=0.55.0"
preview = ["pixi-build"]

[package.build.backend]
name = "pixi-build-rattler-build"
version = "*"

[dependencies]
please = { path = "./pixi-packages/please" }

• ./pixi-packages/please/recipe.yaml

context:
  version: "16.17.1"

package:
  name: please
  version: ${{ version }}

source:
  git: "https://github.com/thought-machine/please"
  tag: "v${{ version }}"

requirements:
  build:
    - ${{ compiler('go-cgo') }}
    - ${{ compiler('c') }}

tests:
  - script:
      - plz --version

build:
  script:
    - mkdir -p $PREFIX/bin
    - CGO_ENABLED=1 go build -o $PREFIX/bin/plz ./src/

about:
  homepage: https://please.build
  repository: https://github.com/thought-machine/please
  documentation: https://please.build
  summary: The Please build system
  description: |
    High-performance extensible build system for reproducible multi-language builds.
  license: Apache-2.0
  license_file:
    - LICENSE

• pixi install and see the package being built.
• pixi run which plz to see the plz binary being available in the environment.
• pixi install again, and see that everything is cached.
• touch ./pixi-packages/please/recipe.yaml.
• pixi install again, and see the package being rebuilt. At this step it also fetches the repodata, and it's taking a long time, despite the first repodata fetch being pretty fast, assuming due to using the https://prefix.dev/conda-forge channel with optimizations. But looks like on the second install it's not using it.

Expectations

1. No rebuilds when nothing relevant about the package has changed.
2. No unnecessary repodata fetches if rebuilds need to happen.

[ruben-arts]

Currently, this rebuild is by design, but I agree we should change this.

The "No rebuilds when nothing relevant" is the biggest problem, as it's complex to figure out what that means for a project.

When would you @burdiyan want your project to be rebuilt? Which file changes. And do you want this to trigger on every pixi command?

[baszalmstra]

I think what makes this worse is that the rattler-build backend doesnt do incremental compilation.

[baszalmstra]

Perhaps we could setup a go backend? That should make this much more comfortable! We could do incremental compilation and be much more precice about which files trigger a rebuild.

[wolfv]

I think there are a few optimizations that we could do without going for incremental compilation (which btw. could also be added to the rattler-build backend).

• Is there really something broken about not using the prefix.dev channels when rebuilding? Seems fishy if true, maybe someone can run with -vv to check?
• It doesn't seem to be "by design" to recompile the pacakge when the timestamp of the file changes. We already add the content of the recipe and some other adjacent files to the hash. So why would a timestamp change trigger a build? Or what does touch do in this case?
• Can we cache and re-use the build and host environments? In that case we wouldn't have to fetch repodata.
• For incremental builds with rattler-build users could add some testing for previous build output in their bash scripts. We could also add some extra markers to recipe files in the extra section that pixi would read (just thinking out loud).

[burdiyan]

The "No rebuilds when nothing relevant" is the biggest problem, as it's complex to figure out what that means for a project.

@ruben-arts yeah, I can totally understand the complexity here. I could probably tolerate a rebuild when the recipe file really changes, but marking it as dirty just because the timestamp changed seems like a divergence from the hash-based checks that Pixi usually does. So I agree with @wolfv's second point.

For a moment I thought that this problem was only happening in pixi-build-rattler-build, because I think when I tried building the same package directly with rattler-build subsequent builds were cached. And even touching the recipe file wasn't triggering rebuilds. But now, when I tried it again, seems like rattler-build rebuilds every time, even without touching the recipe file.

[wolfv]

You can use --skip-existing with rattler-build to skip packages that you had previously built. But it also doesn't take into account any source code changes or recipe changes (except when you change the version number or build string - which can also be changed by the variant config).

[burdiyan]

Is this also by design? I guess in theory rattler-build could hash the rendered recipe file and related scripts for checking whether it needs to be rebuilt, right?

[wolfv]

Yeah, that could all be done for sure. Like we could change the "hash" / build string of a package based on the recipe content. It's just not how rattler-build currently works.