Skip to content

Start article about the history of conda-forge #2298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 35 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
333e9fb
Start article about the history of conda-forge
jaimergp Sep 14, 2024
9bebd8b
Merge branch 'main' of github.com:conda-forge/conda-forge.github.io i…
jaimergp Apr 11, 2025
bea0898
pre-commit
jaimergp Apr 11, 2025
4e22a38
Address some feedback about the early history
jaimergp Apr 11, 2025
2af512e
Rebrand was in 2017
jaimergp Apr 11, 2025
d164985
Clarify Scitools
jaimergp Apr 11, 2025
a3c6f5f
add quote from Travis
jaimergp Apr 11, 2025
a58f100
Add link to binstar announcement in scipy
jaimergp Apr 11, 2025
352d669
More content in "how conda-forge came to be"
jaimergp Apr 11, 2025
96d4d5c
More details about the state of wheels in 2013
jaimergp Apr 11, 2025
17e5982
Mention Gohlke and Cournapeau
jaimergp Apr 11, 2025
ca6a3b4
Add link to SciTool's .travis.yml
jaimergp Apr 11, 2025
6e8b282
typo
jaimergp Apr 11, 2025
1583e8b
Improve language and grammar
jaimergp Apr 11, 2025
d881fe5
add manylinux link
jaimergp Apr 11, 2025
5cf119e
flesh out continuum history (#1)
msarahan Apr 14, 2025
271b95e
cosmetic adjustments
jaimergp Apr 14, 2025
8a9e5b9
pre-commit
jaimergp Apr 14, 2025
9930e5b
fix lost save
msarahan Apr 14, 2025
9c36534
Merge branch 'history-conda-forge' of github.com:jaimergp/conda-forge…
jaimergp Apr 14, 2025
d0a4839
add link to talk python episode #94
jaimergp Apr 14, 2025
b0135c7
reapply cosmetic changes
jaimergp Apr 14, 2025
e43258b
pre-commit
jaimergp Apr 14, 2025
2f8aaa3
add github users
jaimergp Apr 14, 2025
a0c439c
Update community/history.md
jaimergp Apr 15, 2025
734d2b6
link to pypackaging-native for abi explanation
jaimergp Apr 15, 2025
65f25b7
mention CB3 and gcc 7
jaimergp Apr 15, 2025
9f1463b
break up long paragraphs a bit and add more sections
jaimergp Apr 15, 2025
cdb662a
add reference to Anaconda 5.0 release
jaimergp Apr 15, 2025
063b395
Line breaks and a couple of new links (#2)
msarahan Apr 15, 2025
16cc130
fix links
jaimergp Apr 15, 2025
6eaed62
fold at 90
jaimergp Apr 15, 2025
2d96e5a
mention build/host/run split
jaimergp Apr 15, 2025
a3430e3
pre-commit
jaimergp Apr 15, 2025
25cde72
one more link to conda-build docs
jaimergp Apr 15, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions community/_sidebar.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
[
"index",
"history",
"getting-in-touch",
{
"type": "category",
Expand Down
61 changes: 61 additions & 0 deletions community/history.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: History
---

# History of conda-forge

conda-forge's origins cannot be explained without understanding the context of Python packaging back in the early 2010s. Back then, the installation of Python packages across operating systems was very challenging, specially on Windows, as it often meant compiling dependencies from source.

Python 2.x was the norm, the community was transitioning from `easy_install` to `pip`, and there wouldn't be an alternative for Python eggs [^eggs] until 2012, when wheels are introduced [^wheels]. To get Python, you'd get the official installers from Python.org, stick to the system provided one in Linux, or resort to ActiveState's or Enthought's distributions in macOS and Windows [^legacy-python-downloads].

## The origins of `conda`

In 2012, Continuum Analytics announces Anaconda 0.8 in the SciPy conference [^anaconda-history]. Later that year, in September, Continuum would release `conda` 1.0, the cross-platform, language-agnostic package manager for pre-compiled artifacts [^conda-changelog-1.0]. The motivation behind these efforts was to provide an easy way to ship all the compiled libraries and Python packages that users of the SciPy and numpy stacks needed [^packaging-and-deployment-with-conda] [^lex-fridman-podcast].

In constrast with Python eggs and wheels, conda packages were agnostic enough to ship Python itself, as well as the underlying shared libraries without having to statically vendor them under each Python package. This was particularly convenient for projects that relied on both compiled dependencies (e.g. C++ or Fortran libraries) and Python "glue code".
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above - this is jumping the gun. There was no binary shipping mechanism for Python - conda was the first environment manager to do so AFAIK.

Before this, on windows you used to go to Christoph Gohlk's website. On linux, I think you had to build from source.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Howdy @pelson! Long time.

Back around say 2005-2010 I remember it working out OK on platforms where you had strong package manager interfaces at the OS tier — like Debian/Ubuntu based distributions, MacPorts on OS X, or ports on FreeBSD. But of course trying to do so on Windows, or anywhere where you didn't have a well maintained compiler stack trivially installable was still a nightmare, especially if you needed to work with a broad collection of packages. In terms of Python specific package managers which did deal with the scientific Python distribution problem, I believe that Enthought had something even before Canopy, and definitely ActiveState did, but both of these were licensed products which hampered their use, particularly in the academic community.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added some more context here, hopefully it reads better now?


By June 2013, conda is using a SAT solver and includes the `conda build` subcommand [^new-advances-in-conda], along with the concept of recipes [^conda-recipes-repo] [^early-conda-build-docs]. This is also when the first Miniconda release is announced. By the end of the year, Continuum Analytics announces Binstar.org, the predecessor of the Anaconda.org channels. This meant that now any user could build their software stack as conda packages and redistribute them online at no cost.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think conda-recipes is worth more detail - it was the place that people would contribute their recipes.

It was really successful, but the recipes were of various quality, and typically only worked on one or two platforms. There was a high chance that a recipe you found there would no longer build, and you had to tweak it to get it to work.

To my knowledge, SciTools was the first repo to do CI based builds. Filipe borrowed the technical infra for IOOS to do the same. I borrowed some of the harder to build recipes back. It was a successful collaborative effort, but it was inefficient since we were working in separate repos. We often had duplicate recipes etc.

conda-forge came about because I could see that the conda-recipes repo was popular, there was demand for having high-quality recipes, and we needed a way to build them in a consistent way. I even built a tool (conda-build-all) to try to do this from our repositories in an efficient way. In the end, it got to the point where we wanted to de-centralise the responsibilities, and the one-repo-per-recipe concept fell out.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If @pelson is OK I think we should write this part in the blog in some form:

It was really successful, but the recipes were of various quality, and typically only worked on one or two platforms. There was a high chance that a recipe you found there would no longer build, and you had to tweak it to get it to work.

To my knowledge, SciTools was the first repo to do CI based builds. Filipe borrowed the technical infra for IOOS to do the same. I borrowed some of the harder to build recipes back. It was a successful collaborative effort, but it was inefficient since we were working in separate repos. We often had duplicate recipes etc.

conda-forge came about because I could see that the conda-recipes repo was popular, there was demand for having high-quality recipes, and we needed a way to build them in a consistent way. I even built a tool (conda-build-all) to try to do this from our repositories in an efficient way. In the end, it got to the point where we wanted to de-centralise the responsibilities, and the one-repo-per-recipe concept fell out.

What do you think @jaimergp?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added almost everything. I need more details on the monorepo -> recipe-per-repo transition. Do I remember correctly that conda-forge was initially set up as a monorepo?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing in that story that I'm curious about, and have a hazy understanding of at best, is the relationship to anaconda's free channel. It's possible that it's completely unrelated, but I have the vague recollection that conda-forge grew out of (or at least into) the same space as the free channel

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can explain this, but I am afk until Sunday.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added almost everything. I need more details on the monorepo -> recipe-per-repo transition. Do I remember correctly that conda-forge was initially set up as a monorepo?

Nope. The "proto" conda-forge, sscitools and ioos channels were, but conda-forge started as a distributed repos due to the monorepo limitations we had.


By 2015, several institutes and groups were using Binstar to distribute software packages they used daily: the [Omnia Molecular Dynamics](https://github.com/omnia-md) project started as early as March 2014 [^binstar-omnia], UK's [Scitools](https://scitools.org.uk/) joined in June 2014 [^binstar-scitools], the [US Integrated Ocean Observing System (IOOS)](http://www.ioos.noaa.gov/) started using it in July 2014 [^binstar-ioos]. The channel for conda-forge was not created until April 2015 [^binstar-conda-forge], and [Bioconda](https://anaconda.org/bioconda) waited until September of the same year.

In 2015, Continuum Analytics rebranded as Anaconda Inc, and Binstar.org became Anaconda.org.

## How conda-forge came to be

In 2014, Filipe Fernandes ([@ocefpaf](https://github.com/ocefpaf)) and Phil Elson ([@pelson](https://github.com/pelson)) get in touch [^chatting-ocefpaf]. They are maintaining the Binstar channels for IOOS and Scitools, respectively.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think there is a lot Filipe and I can add here 😉

We should give some context of the geospatial world in particular (hi gdal).

I would also get to the point of acknowledging both Christophe Gohlke and David Cournapeau, who especially helped me with the Windows builds of the whole SciPy stack (a topic on which I had no knowledge at all, yet needed to get building in a CI context on appveyor. In those days you had to pick particular compilers for particular Python versions, and this was a bit of a dark art to me).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the acknowledgements, but would love to hear more about the geospatial world if you want to add something!

<!-- to be continued -->

## References

[^cgohlke-shutdown]: [Christoph Gohlke's Windows Wheels site is shutting down by the end of the month](https://www.reddit.com/r/Python/comments/vcaibq/christoph_gohlkes_windows_wheels_site_is_shutting/), 2022.

[^anaconda-history]: [The Early History of the Anaconda Distribution](http://ilan.schnell-web.net/prog/anaconda-history/), Ilan Schnell, 2018.

[^lex-fridman-podcast]: [Travis Oliphant: NumPy, SciPy, Anaconda, Python & Scientific Programming](https://www.youtube.com/watch?v=gFEE3w7F0ww&t=7596s), Lex Fridman Podcast #224, 2022.

[^conda-changelog-1.0]: [`conda` 1.0 release notes](https://github.com/conda/conda/blob/24.7.1/CHANGELOG.md#100-2012-09-06), 2012.

[^early-conda-build-docs]: [Conda build framework documentation](https://web.archive.org/web/20141006141927/http://conda.pydata.org/docs/build.html), 2014.

[^conda-recipes-repo]: [ContinuumIO/conda-recipes](https://github.com/conda-archive/conda-recipes)

[^packaging-and-deployment-with-conda]: [Packaging and deployment with conda](https://speakerdeck.com/teoliphant/packaging-and-deployment-with-conda), Travis Oliphant, 2013.

[^new-advances-in-conda]: [New Advances in Conda](https://web.archive.org/web/20140331190645/http://continuum.io/blog/new-advances-in-conda), Ilan Schnell, 2013.

[^binstar-scitools]: https://anaconda.org/scitools, 2014.

[^binstar-ioos]: https://anaconda.org/ioos, 2014.

[^binstar-omnia]: https://anaconda.org/omnia, 2014.

[^binstar-conda-forge]: https://anaconda.org/conda-forge, 2015.

[^chatting-ocefpaf]: [Filipe Fernandes on the Evolution of conda-forge](https://www.youtube.com/watch?v=U2oa_RLbTVA), Chatting with the Conda Community #1, 2024.

[^wheels]: [PEP 427 – The Wheel Binary Package Format 1.0](https://peps.python.org/pep-0427/)

[^eggs]: [The Internal Structure of Python Eggs](https://setuptools.pypa.io/en/latest/deprecated/python_eggs.html).

[^legacy-python-downloads]: [Download Python for Windows (legacy docs)](https://legacy.python.org/download/windows/).
2 changes: 1 addition & 1 deletion community/index.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: 'conda-forge community'
title: 'The conda-forge community'
---

# Community
Expand Down