-
-
Notifications
You must be signed in to change notification settings - Fork 299
Start article about the history of conda-forge #2298
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 1 commit
333e9fb
9bebd8b
bea0898
4e22a38
2af512e
d164985
a3c6f5f
a58f100
352d669
96d4d5c
17e5982
ca6a3b4
6e8b282
1583e8b
d881fe5
5cf119e
271b95e
8a9e5b9
9930e5b
9c36534
d0a4839
b0135c7
e43258b
2f8aaa3
a0c439c
734d2b6
65f25b7
9f1463b
cdb662a
063b395
16cc130
6eaed62
2d96e5a
a3430e3
25cde72
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
[ | ||
"index", | ||
"history", | ||
"getting-in-touch", | ||
{ | ||
"type": "category", | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
--- | ||
title: History | ||
--- | ||
|
||
# History of conda-forge | ||
|
||
conda-forge's origins cannot be explained without understanding the context of Python packaging back in the early 2010s. Back then, the installation of Python packages across operating systems was very challenging, specially on Windows, as it often meant compiling dependencies from source. | ||
|
||
Python 2.x was the norm, the community was transitioning from `easy_install` to `pip`, and there wouldn't be an alternative for Python eggs [^eggs] until 2012, when wheels are introduced [^wheels]. To get Python, you'd get the official installers from Python.org, stick to the system provided one in Linux, or resort to ActiveState's or Enthought's distributions in macOS and Windows [^legacy-python-downloads]. | ||
jaimergp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## The origins of `conda` | ||
|
||
In 2012, Continuum Analytics announces Anaconda 0.8 in the SciPy conference [^anaconda-history]. Later that year, in September, Continuum would release `conda` 1.0, the cross-platform, language-agnostic package manager for pre-compiled artifacts [^conda-changelog-1.0]. The motivation behind these efforts was to provide an easy way to ship all the compiled libraries and Python packages that users of the SciPy and numpy stacks needed [^packaging-and-deployment-with-conda] [^lex-fridman-podcast]. | ||
jaimergp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
In constrast with Python eggs and wheels, conda packages were agnostic enough to ship Python itself, as well as the underlying shared libraries without having to statically vendor them under each Python package. This was particularly convenient for projects that relied on both compiled dependencies (e.g. C++ or Fortran libraries) and Python "glue code". | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As above - this is jumping the gun. There was no binary shipping mechanism for Python - Before this, on windows you used to go to Christoph Gohlk's website. On linux, I think you had to build from source. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Howdy @pelson! Long time. Back around say 2005-2010 I remember it working out OK on platforms where you had strong package manager interfaces at the OS tier — like Debian/Ubuntu based distributions, MacPorts on OS X, or ports on FreeBSD. But of course trying to do so on Windows, or anywhere where you didn't have a well maintained compiler stack trivially installable was still a nightmare, especially if you needed to work with a broad collection of packages. In terms of Python specific package managers which did deal with the scientific Python distribution problem, I believe that Enthought had something even before Canopy, and definitely ActiveState did, but both of these were licensed products which hampered their use, particularly in the academic community. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I added some more context here, hopefully it reads better now? |
||
|
||
By June 2013, conda is using a SAT solver and includes the `conda build` subcommand [^new-advances-in-conda], along with the concept of recipes [^conda-recipes-repo] [^early-conda-build-docs]. This is also when the first Miniconda release is announced. By the end of the year, Continuum Analytics announces Binstar.org, the predecessor of the Anaconda.org channels. This meant that now any user could build their software stack as conda packages and redistribute them online at no cost. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think It was really successful, but the recipes were of various quality, and typically only worked on one or two platforms. There was a high chance that a recipe you found there would no longer build, and you had to tweak it to get it to work. To my knowledge, SciTools was the first repo to do CI based builds. Filipe borrowed the technical infra for IOOS to do the same. I borrowed some of the harder to build recipes back. It was a successful collaborative effort, but it was inefficient since we were working in separate repos. We often had duplicate recipes etc. conda-forge came about because I could see that the conda-recipes repo was popular, there was demand for having high-quality recipes, and we needed a way to build them in a consistent way. I even built a tool ( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If @pelson is OK I think we should write this part in the blog in some form:
What do you think @jaimergp? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added almost everything. I need more details on the monorepo -> recipe-per-repo transition. Do I remember correctly that conda-forge was initially set up as a monorepo? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One thing in that story that I'm curious about, and have a hazy understanding of at best, is the relationship to anaconda's There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I can explain this, but I am afk until Sunday. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Nope. The "proto" conda-forge, sscitools and ioos channels were, but conda-forge started as a distributed repos due to the monorepo limitations we had. |
||
|
||
By 2015, several institutes and groups were using Binstar to distribute software packages they used daily: the [Omnia Molecular Dynamics](https://github.com/omnia-md) project started as early as March 2014 [^binstar-omnia], UK's [Scitools](https://scitools.org.uk/) joined in June 2014 [^binstar-scitools], the [US Integrated Ocean Observing System (IOOS)](http://www.ioos.noaa.gov/) started using it in July 2014 [^binstar-ioos]. The channel for conda-forge was not created until April 2015 [^binstar-conda-forge], and [Bioconda](https://anaconda.org/bioconda) waited until September of the same year. | ||
jaimergp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
In 2015, Continuum Analytics rebranded as Anaconda Inc, and Binstar.org became Anaconda.org. | ||
jaimergp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## How conda-forge came to be | ||
|
||
In 2014, Filipe Fernandes ([@ocefpaf](https://github.com/ocefpaf)) and Phil Elson ([@pelson](https://github.com/pelson)) get in touch [^chatting-ocefpaf]. They are maintaining the Binstar channels for IOOS and Scitools, respectively. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Think there is a lot Filipe and I can add here 😉 We should give some context of the geospatial world in particular (hi I would also get to the point of acknowledging both Christophe Gohlke and David Cournapeau, who especially helped me with the Windows builds of the whole SciPy stack (a topic on which I had no knowledge at all, yet needed to get building in a CI context on appveyor. In those days you had to pick particular compilers for particular Python versions, and this was a bit of a dark art to me). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added the acknowledgements, but would love to hear more about the geospatial world if you want to add something! |
||
<!-- to be continued --> | ||
|
||
## References | ||
|
||
[^cgohlke-shutdown]: [Christoph Gohlke's Windows Wheels site is shutting down by the end of the month](https://www.reddit.com/r/Python/comments/vcaibq/christoph_gohlkes_windows_wheels_site_is_shutting/), 2022. | ||
jaimergp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
[^anaconda-history]: [The Early History of the Anaconda Distribution](http://ilan.schnell-web.net/prog/anaconda-history/), Ilan Schnell, 2018. | ||
|
||
[^lex-fridman-podcast]: [Travis Oliphant: NumPy, SciPy, Anaconda, Python & Scientific Programming](https://www.youtube.com/watch?v=gFEE3w7F0ww&t=7596s), Lex Fridman Podcast #224, 2022. | ||
|
||
[^conda-changelog-1.0]: [`conda` 1.0 release notes](https://github.com/conda/conda/blob/24.7.1/CHANGELOG.md#100-2012-09-06), 2012. | ||
|
||
[^early-conda-build-docs]: [Conda build framework documentation](https://web.archive.org/web/20141006141927/http://conda.pydata.org/docs/build.html), 2014. | ||
|
||
[^conda-recipes-repo]: [ContinuumIO/conda-recipes](https://github.com/conda-archive/conda-recipes) | ||
|
||
[^packaging-and-deployment-with-conda]: [Packaging and deployment with conda](https://speakerdeck.com/teoliphant/packaging-and-deployment-with-conda), Travis Oliphant, 2013. | ||
|
||
[^new-advances-in-conda]: [New Advances in Conda](https://web.archive.org/web/20140331190645/http://continuum.io/blog/new-advances-in-conda), Ilan Schnell, 2013. | ||
jaimergp marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
[^binstar-scitools]: https://anaconda.org/scitools, 2014. | ||
|
||
[^binstar-ioos]: https://anaconda.org/ioos, 2014. | ||
|
||
[^binstar-omnia]: https://anaconda.org/omnia, 2014. | ||
|
||
[^binstar-conda-forge]: https://anaconda.org/conda-forge, 2015. | ||
|
||
[^chatting-ocefpaf]: [Filipe Fernandes on the Evolution of conda-forge](https://www.youtube.com/watch?v=U2oa_RLbTVA), Chatting with the Conda Community #1, 2024. | ||
|
||
[^wheels]: [PEP 427 – The Wheel Binary Package Format 1.0](https://peps.python.org/pep-0427/) | ||
|
||
[^eggs]: [The Internal Structure of Python Eggs](https://setuptools.pypa.io/en/latest/deprecated/python_eggs.html). | ||
|
||
[^legacy-python-downloads]: [Download Python for Windows (legacy docs)](https://legacy.python.org/download/windows/). |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
--- | ||
title: 'conda-forge community' | ||
title: 'The conda-forge community' | ||
--- | ||
|
||
# Community | ||
|
Uh oh!
There was an error while loading. Please reload this page.