add blog/2020-07-11-R-4.md

Co-authored-by: beckermr <[email protected]>
Co-authored-by: cj-wright <[email protected]>

Author: 3 people

blog/2020-07-11-R-4.md

@@ -0,0 +1,114 @@
+---
+authors:
+  - cj-wright
+  - beckermr
+tags: [scipy]
+---
+
+# R 4.0 Migration Retrospective
+
+While the R 4.0 migration has been functionally complete for quite a
+while, the recent migration of `r-java` and its dependents gives a good
+opportunity to write a retrospective on the technical issues with
+large-scale migrations in `conda-forge` and how we solved them.
+
+The R 4.0 migration rebuilt every package in `conda-forge` that had
+`r-base` as a requirement, including more than 2200 feedstocks. A
+migration of this size in `conda-forge` faces several hurdles. First,
+since every feedstock is a separate GitHub repository, one needs to
+merge more 2200 pull requests (PRs). Second, `conda-forge`'s packages
+on `anaconda.org` are behind a CDN (content delivery network). This
+service reduces web hosting costs for Anaconda Inc. but introduces an
+approximately 30 minute delay from when a package is uploaded to
+`anaconda.org` and when it will appear as available using `conda` from
+the command line. Thus, even if the dependencies of a package have been
+built, we have to wait until they appear on the CDN before we can
+successfully issue the next PR and have it build correctly. Finally, the
+existing bot and `conda` infrastructure limited the throughput of the
+migrations, due in part to the speed of the `conda` solver.
+
+Given the size of the R 4.0 migration, we took this opportunity to try
+out a bunch of new technology to speed up large-scale migrations. The
+main enhancements were using GitHub Actions to automerge PRs, using
+`mamba` to quickly check for solvability of package environments, and
+enabling long-running migration jobs for the autotick bot. All told, the
+bulk of the feedstocks for R 4.0 were rebuilt in less than a week, with
+many PRs being merged in 30 minutes or less from when they were issued.
+These enhancements to the autotick bot and `conda-forge` infrastructure
+can be used to enhance future migrations (e.g., Python 3.9) and reduce
+maintenance burdens for feedstocks.
+
+## Automerging conda-forge PRs
+
+In a typical migration on `conda-forge`, we issue a PR to a feedstock
+and then ask the feedstock maintainers to make sure it passes and merge
+it. In the case of the R 4.0 migration, the maintainers of R packages on
+`conda-forge` use a maintenance team (i.e., `@conda-forge/r`) on the
+vast majority of feedstocks. This team is small and so merging over 2000
+PRs by hand is a big undertaking. Thus, with their permission, we added
+the `conda-forge` automerge functionality to all R feedstocks that they
+maintain. The automerge bot, which relies on GitHub Actions, is able to
+automatically merge any PR from the autotick bot that passes the recipe
+linter, the continuous integration services, and has the special
+`[bot automerge]` slug in the PR title. This feature removed the
+bottleneck of waiting for maintainers to merge PRs and reduced the
+maintenance burden on the R maintenance team.
+
+## Checking Solvability with mamba
+
+While being able to automatically merge PRs removed much of the work of
+performing the R 4.0 migration, it relied on the PR building correctly
+the first time it was issued. Due to the CDN delays and the build times
+of a package's dependencies, the dependencies of a package may not be
+immediately available after all of their migration PRs are merged. If
+the bot issued the packages migration PR before the dependents are
+available, the PR would fail with an unsolvable environment and have to
+be restarted manually. This failure would negate any of the benefits of
+using automerge in the first place.
+
+To control for this edge case, we employed the `mamba` package to check
+for the solvability of a PR's environments before the PR was issued.
+`mamba` is a fast alternative to `conda` that produces solutions for
+environments orders of magnitude more quickly. Since, we have to perform
+our checks of PR environments many times, an extremely fast solver was
+essential for making the code efficient enough to run as part of the
+autotick bot. We ended up using mamba to try to install the dependencies
+for every variant produced by the feedstock to be migrated. With this
+check in place, the autotick bot was able to issue migration PRs that
+passed on the first try and were thus automatically merged, many within
+30 minutes or less.
+
+## Improving the Autotick Bot's Efficiency
+
+Finally, we made several upgrades to the autotick bot infrastructure to
+increase the uptime of the bot and its efficiency. First, we moved from
+an hourly cron job to a set of chained CI jobs. This change eliminated
+downtime between the runs of the bot. Second, we started to refactor the
+autotick bot from one monolithic piece of code into a distributed set of
+microservices which perform various independent tasks in parallel. These
+independent tasks, used for things like checking the statuses of
+previously issued PRs, are run separately allowing the bot to spend more
+time issuing PRs. Finally, we optimized the internal prioritization of
+the PRs to make sure the bot was spending more time on larger migrations
+where there is more work to do. More work on the autotick bot
+infrastructure, including work done by Vinicius Cerutti as part of the
+Google Summer of Code program, will further streamline the bot's
+operation.
+
+Despite some initial hiccups with the bot infrastructure, the migration
+ran quite smoothly for an endeavor of its size. The vast majority of
+migration PRs were completed within a week from when we started, which
+is a first for a migration of this size on `conda-forge`. The largest
+issue was solved recently, with the fixing of the `openjdk` recipe and
+the removal of `aarch64` and `ppc64le` builds from `r-java`, enabling
+the last large piece of the R ecosystem to be updated.
+
+Looking forward, the improvements we made for the R 4.0 migration seem
+broadly applicable to other migration tasks, including the yearly python
+minor version bump. These kinds of large-scale migrations are
+particularly suitable, since they usually involve few changes to the
+feedstock itself and usually fail on CI when a broken package would be
+produced. Faster migrations will help to provide the latest features to
+downstream users and keep transition times to a minimum, helping to
+foster greater stability of the ecosystem and the seamless experience
+users have come to expect from `conda-forge`.