v1/rattler-build adoption and anaconda/conda-forge alignment strategy in general.

[danpetry]

A summary of Anaconda's current state with regards to rattler-build/v1 format adoption is here.

The v0/v1 formats aren't miles apart but we already have to do some work to adapt recipes to anaconda's main distribution, this would increase the amount of that work. However, we don't want to block or stymie community efforts. In the end, it should be judged as to whether it's a net benefit here (conda-forge) to wait or go forward, I think. Hopefully aligning efforts is a net benefit to conda-forge, too.

We could also here discuss wider strategies on collaboration given the different working approaches.

[hmaarrfk]

There are 3 parts to the recipe:

1. The dependencies
2. The build.sh and related scripts.
3. The patches

If we can align on 2 and 3 that would be a win while Anaconda catches up??? Thoughts?

[jakirkham]

The long term plan is for conda-forge to move to the v1 format: conda-forge/conda-forge.github.io#2308

That said, this will take some time to do. Especially for more complex recipes (like PyTorch) or more advanced features (bots)

Additionally comfort level with the current state of v1/rattler-build varies amongst maintainers. Though expect we will continue to work through issues as a community to improve general acceptance. Should add this is informed by my own past experience of going through other large scale changes in conda-forge

[wolfv]

Beyond improvements in rendering speed and other niceties of the v1 format, rattler-build also massively outperforms conda-build in the relinking step, which is particularly important for large packages and packages with many files. rattler-build is up to 60x faster.

Here is a thread on Zulip: https://conda-forge.zulipchat.com/#narrow/channel/457337-general/topic/rattler.20v1.20auto.20version.20bump/near/494063869

I see no reasons to hold back on the v1 recipe. If Anaconda wants to stay on v0 then transforming the recipe back should be feasible and easy enough for something like pytorch. Applying / syncing changes between the formats is a small effort compared to the amount of work in testing and building the recipe.

If there is anything not working with rattler-build / v1 we at prefix.dev are 1000% committed to fixing it as soon as possible.

[baszalmstra]

Im very sure that with the number of files in the pytorch build switching to v1/rattler-build would also make a pretty big difference!

[mgorny]

more advanced features (bots)

Just for the record, bots are working since yesterday.

[danpetry]

In general if it's possible to have some level of co-ordination so that we aren't always scrambling to catch up with changes here while we synchronise, that would be helpful.

[h-vetinari]

In general if it's possible to have some level of co-ordination so that we aren't always scrambling to catch up with changes here while we synchronise, that would be helpful.

What did you have in mind? I think what would be useful is to turn https://github.com/AnacondaRecipes/pytorch-feedstock into a fork of this feedstock (doesn't need to be a fork from the POV of GitHub, --allow-unrelated-histories when merging works too). Then anaconda can still have some local delta, but as long as the changes here don't touch that part, it wouldn't cause conflicts when back-merging. And obviously we could reduce the delta over time, as in #318

[h-vetinari]

For the v1 discussion, I don't think pytorch is a good candidate yet. It's a feedstock where conda-forge barely became independent from Anaconda, and which is so heavy that any collaboration to improve packaging is welcome.

I want that collaboration to be smooth much more than we really need v1 here IMO, even though the potential speed gains from rattler-build would be nice.

If, over the next couple months, the collaboration doesn't actually materialise (doubtful, because @danpetry has been very involved and very helpful recently), then we can consider going solo. Or of course once Anaconda manages to switch to v1 (perhaps the pytorch feedstock could be a good vanguard for this effort Anaconda-internally?).

[mgorny]

As a semi-related idea, how would you feel about making some changes to the recipe that would make the eventual conversion a bit easier? Such as renaming bld.bat to build.bat, replacing {% if %} with inline selectors and {{ ... if ... else ... }}.

[h-vetinari]

Such as renaming bld.bat to build.bat

That won't work AFAICT, because it's not possible to specify a custom script name for the global build stage.

replacing {% if %} with inline selectors

Not all Jinja usages are replaceable with selectors in practice, unfortunately. Where it makes sense I'm not against it of course. But for example the whole if cuda_compiler_version != "None" block is better without selectors (especially since a move to rattler will have to change indentation for all the cuda* and libcu* packages anyway)

and {{ ... if ... else ... }}.

Where it actually reduces the diff that will be created in the v1 conversion, I'm open to changing things

[wolfv]

custom script name for the global build stage

really? that would be news to me

[h-vetinari]

Well, I'm happy to be proven wrong. Perhaps setting

build:
  script: build.bat  # [win]

does work after all, but my impression was that this is mainly used (and thus tested) in outputs, and that the global stage expects build.sh & bld.bat. 🤷‍♂️

[danpetry]

In general if it's possible to have some level of co-ordination so that we aren't always scrambling to catch up with changes here while we synchronise, that would be helpful.

What did you have in mind? I think what would be useful is to turn AnacondaRecipes/pytorch-feedstock into a fork of this feedstock (doesn't need to be a fork from the POV of GitHub, --allow-unrelated-histories when merging works too). Then anaconda can still have some local delta, but as long as the changes here don't touch that part, it wouldn't cause conflicts when back-merging. And obviously we could reduce the delta over time, as in #318

Yes this is what we're currently doing. It does however take some time to pull changes and debug over our end, and recently, changes have (happily) been prolific and progress has moved fast here, meaning it's a moving target to sync. I'm thinking more of discourse to help us not be on the back step catching up to changes on this side. Of course without hindering development speed here. Probably a zulip channel would be helpful.

any collaboration to improve packaging is welcome

Great to hear, I feel the same way. Sounds like the benefit and intention is there on both sides so I'm sure it will work.

[danpetry]

I can't create zulip channels, if that would be useful would somebody please be able to create one?