4.0.0

These release notes are automatically extracted from the full changelog.

This release contains two potentially-breaking changes for existing usages. The circumstances and implications of each are described below.

Improvements

• It is now an error (instead of a warning) to use the --image option to nextstrain build when using the "native" runner (either explicitly via --native or implicitly via config set by nextstrain check-setup --set-default). The error message is:

  The --image option is incompatible with the "native" runner (…).
  
  If you need to use the "native" runner, please omit the --image option.
  
  If you need the --image option, please select another runner (e.g.
  with the --docker option) that supports it.  Currently --image is
  supported by the Docker (--docker) and AWS Batch (--aws-batch)
  runners.  You can check if your setup supports these runners with
  `nextstrain check-setup`.
  

  This is a potentially-breaking change as invocations using the "native" runner with the --image option may exist and be working for users as they expect. If you encounter this new error after upgrading but your results of running nextstrain commands has always been as-expected, then you can safely drop the --image option from your invocations and avoid the new error.

• When running a Snakemake workflow, nextstrain build now defaults Snakemake's --cores option to all unless build's own --cpus option is provided. If you provide your own --cores (or equivalent, e.g. -j) option to Snakemake via nextstrain build, as in nextstrain build . --cores 2, then this new default isn't applicable.

  This is a potentially-breaking change if you're…

  • …using Snakemake version 5.10 or earlier (such as via our standard Docker runtime image), and

  • …are not already providing a --cores (or equivalent, e.g. -j) option to Snakemake via nextstrain build, and

  • …expect your nextstrain build invocations to use only a single CPU/core instead of all CPUs/cores available on your computer.

  If this is the case, you can pass --cpus 1 to nextstrain build to regain the original behaviour, e.g. nextstrain build --cpus 1 ..

  This change will allow upgrading of Snakemake in our Docker runtime image without inflicting the addition of --cores (or equivalent) arguments onto every existing nextstrain build invocation that lacks it.

  For context, Snakemake requires the --cores option as of 5.11. This has spawned much discussion (1, 2, 3, 4) where it was made clear this is an intentional, permanent change and a default will not be added. By adding our own default, we can insulate our users from the upstream change and make Nextstrain builds fast-by-default. Our --cpus option can be used to limit CPU usage back from this default if necessary, and users can always specify --cores (or equivalents) themselves.

• The version --verbose and check-setup commands now indicate the default runner in their output, which is useful context when troubleshooting or just plain unsure what the default is.

• The check-setup command now exits with an error code if the default runner is not supported. Prior to this it only exited with an error code if no runners were supported.

Bug fixes

• When using the AWS Batch runner, the --cpus and --memory options for build now correctly override the corresponding resource requests in newer style AWS Batch job definitions. Prior to this they would be ignored by AWS Batch. Older style AWS Batch job definitions were never affected by this and continue to work with --cpus and --memory as expected. See #144 for more details.

• A deprecation warning from the cryptography module (version 37 and higher) when running on Python 3.6 is now suppressed as it's just noise to an end user. This probably moves us closer to dropping 3.6 support ourselves, but it's not so onerous yet.

• The rST to plain text conversion used to format --help text was fixed to avoid emitting \r\n in the wrong context.

• The stdout and stderr streams are now configured at program start to always emit UTF-8. Previously they used the Python defaults, determined in part by the system defaults, which often resolved to UTF-8, but not always. The code base assumes UTF-8, and now the streams are guaranteed to match. In particular, this fixes UnicodeEncodeError issues in some contexts on Windows even when UTF-8 is supported.

Documentation

• This changelog now sports a preamble to set the scene and provide context for the content.

Development

• The source repo now uses a +git local version part to distinguish actual releases from installations of unreleased code. Relatedly, the development builds created by CI use a +git.${commit} local version part to pin down the specific commit from which they were built. This is mostly helpful when reading CI logs or downloading the builds from the CI artifacts.

• The CI workflow has seen some significant sprucing up and additions, including sporting a more typical lifecycle with separate steps for build, test, and release (a new addition). The single test step is also now split between source tests (unit tests, type checking, linting, etc) and dist tests (functional tests, integration tests, interface tests, etc). The release step uploads to PyPI after a release tag is pushed, built, and tested, replacing manual uploads from a local development environment. Various other small improvements to CI were also made.

• We now run CI tests on Windows. \o/ It's not perfect, but this should help avoid basic Windows-only pitfalls which we might not otherwise notice in a timely fashion.

• Our CI now builds (and tests) standalone installation archives (.tar.gz for Linux and macOS, .zip for Windows) comprising of:

  1. A nextstrain executable containing a bundled Python interpreter + the Python stdlib + the Nextstrain CLI code + its dependencies.

  2. External files (lib, data, etc) that are necessary but can't (for a variety of reasons) be bundled into the executable.

  These installation archives can be downloaded, extracted, and run in-place without even a Python interpreter being installed on the host computer, hence the "standalone" moniker.

  Currently these are for development/testing/experimentation purposes only. We include them as assets on GitHub Releases, but do not provide an automated means of "installing" or unpacking them; those are ultimate goals, but this is just a first step towards those. If you try out the standalone archives in the meantime, though, please let us know how it goes (good or bad) by opening an issue with your experience/feedback/questions.

• GitHub Releases are now created by CI after making a release to PyPI. These are visible on the GitHub repo's releases page and various other places on GitHub. Each GitHub Release includes a copy of the relevant changelog section and release assets like the Python distributions and standalone installation archives (see above). Releases on GitHub are currently intended mostly for informational and notification purposes; the primary release distribution method is still PyPI and sources downstream of PyPI (e.g. Conda).