Responsible use of conda-forge infrastructure

[xhochy]

Hi @conda-forge/sagemaker-code-editor maintainers,

We have recently seen a lot of "mark as broken" PRs from your feedstock. This led me to review this feedstock again. There are several things here that are against the best practices of conda-forge:

• You opened a lot of Update version PRs that had no effect and were just lingering around. I have closed these.
• You seem to be updating versions (semi-automatically), but you don't rerender the feedstock.
• You are publishing dev and beta builds to the main label.
• There are several dev branches on the main label; these builds will also be uploaded to the main label. Please always use a fork and delete these branches.
• At one point, you switched from building from source to binary repackaging. Can you please revert to using (p)npm install?
• Optional: It seems like your build is platform independent, you could mark is as noarch: generic so that this package is also installable on other architectures.

cc @conda-forge/core

Please fix these issues immediately before uploading new version again.

[aws-prayags]

@xhochy Thank you for the detailed review and feedback. We're reviewing all the issues raised and will respond to each point with our plan to address them shortly.

[aws-prayags]

You opened a lot of Update version PRs that had no effect and were just lingering around. I have closed these.

Thank you for closing those. I've also reviewed the remaining open PRs and closed any that were no longer applicable.

[aws-prayags]

You are publishing dev and beta builds to the main label.

Acknowledged. This was an error on our team's part where branches intended for dev releases were created but the conda-forge channel label was not correctly configured. We have previously used a dedicated channel for dev builds and have shared the conda-forge docs on creating pre-release builds with our team to prevent this in the future.

There are several dev branches on the main label; these builds will also be uploaded to the main label. Please always use a fork and delete these branches.

We are working on cleaning up the dev branches now. I've also shared the conda-forge best practices with our team to ensure we use forks going forward.

For clarity: in addition to the main branch, we maintain version-specific branches (e.g., 1.7) for our release management. These are intentional and will remain, but all temporary dev branches will be removed.

[aws-prayags]

At one point, you switched from building from source to binary repackaging. Can you please revert to using (p)npm install?

@xhochy You're correct. We encountered build timeout issues when building from source and were unable to resolve them within the conda build system constraints at the time. To unblock our releases, we had to switch to binary repackaging.

We'll revisit the timeout issue and investigate how we can unblock the source build approach. We may need to reach out to the community for guidance if we continue to face challenges.

Additionally, could you please elaborate on the rationale for building from source in this context? We want to clearly understand how it helps and the specific concerns with binary repackaging.

Thank you for your patience.

[hmaarrfk]

We want to clearly understand how it helps and the specific concerns with binary repackaging.

ultimately conda-forge is a community and as such we have to prioritize the maintainability to the community.

You are using your company associated github account, that is fine, but if a maintainer from outside your company were to come in, and help improve maintainability and compatibility in a meaninful way, in a way that not entirely aligned with your company's goals, we would have to consider their contribution from conda-forge's perspective.

As such, as an OSS ecosystem, we believe it is important to have transparency in how packages are built and deployed. We make great strides to supporting older platforms and using shared libraries.

Have a read through: https://github.com/conda-forge/cfep

---

To me, conda's greatest achievement was to lower the bar to creating your own channel. From AWS's perspective, you should really considering managing your own channels for your own development purposes that bypass much of our infrastructure, and our rules. You will find it VERY beneficial to "align" yourself with conda-forge, and to have stable builds published here. You can of course use conda-smithy to rerender your own recipes, and upload them to your own channel.

There will be HUGE advantages to you working with us to publish your packages here, but you will find that there are challenges to doing so. We work through these through CFEP03 and in some cases, access to more powerful CIs.

However, as AWS, you are in a position to actually help this!
Companies (quantsight in this case) has worked with conda-forge to make more powerful CIs available. https://github.com/Quansight/open-gpu-server
Microsoft has donated many workers (which caused much of our infrastructure to migrate there).

[h-vetinari]

Additionally, could you please elaborate on the rationale for building from source in this context? We want to clearly understand how it helps and the specific concerns with binary repackaging.

The main concerns are:

• ensure compatibility, by carefully controlling the ABI of compilers and dependencies
• avoid bloated binaries due to vendoring or static linking
• avoid dependencies on opaque processes that are not introspectable or auditable for the community

[aws-prayags]

Thank you for sharing the context @hmaarrfk and @h-vetinari . We are taking some time to review this internally and explore the path forward. We'll circle back after that.

Thanks again for working with us on this.

[aws-prayags]

Another quick update - we cleaned up some of the dev branches that were publishing to main label - there are few more such branches. We are working on those.

[aws-prayags]

Couple updates :
1/ We have set a dev label for a particular branch to use for dev releases.
2/ While we are working on gathering issues in building from source, we do need to release a new version to patch a minor version. We will continue working on feedback raised in this issue.