Skip to content

AiiDA Dependency Management

Jump to bottom
Carl Simon Adorf edited this page Apr 17, 2020 · 45 revisions

test-install

About

This page contains guidelines, concrete procedures, and any other information relevant to the management of dependencies for the aiida-core package and plugins.

General information

The dependencies for the aiida-core package must be managed according to AEP 002.

Dependency Manager (DM):

The current dependency manager should be a member of the dependency-manager team.

Specification of AiiDA core dependencies

The dependencies for the aiida-core package are specified in the setup.json file are divided into install_requires and extras_require categories with various sub-categories.

Furthermore, dependencies are also specified in

  • environment.yml: to create a conda environment
  • docs/requirements_for_rtd.txt: for building documentation
  • pyproject.toml: to specify requirements for building the package
  • requirements/requirements-py-3x.txt: well-defined Python environments for all supported Python versions in compliance with dependencies used for CI testing; these files are automatically maintained by the 'update-requirements' workflow

The specification of a dependency within the setup.json is to be considered authoritative.

In addition to above mentioned files, the DM must also be concerned with the modification of the following files:

  • setup.py: Indirectly specifies install dependencies sourced from setup.json
  • util/dependency_management.py: Utility script used for dependency management related tasks.
  • .github/workflows/test-install.yml: Specifies the CI workflow that tests for the ability to install aiida-core with the given dependencies within the current ecosystem.
  • .github/workflows/update-requirements.yml: Specifies the CI workflow that automatically updates the requirements/requirements-py-3x.txt files.

Utility script

The utils/dependency_management.py script within the aiida-core repository provides commands to validate the consistency of aforementioned files with the setup.json file as well as commands to re-generate them after updating a dependency.

Adding new dependencies

To add a new dependency for the aiida-core package as part of a related change, make sure that the dependency adheres to the following requirements:

The newly introduced dependency

  • is needed to close a non-trivial feature gap that could not be resolved easily otherwise,
  • supports all Python versions supported by aiida-core (as specified in setup.json),
  • is available on PyPI and conda-forge [1],
  • appears to be within a stable development stage, e.g., has reached a version 1.0 for projects that follow semantic versioning.

[1] In case that a dependency is not yet available through aforementioned channels, but is considered critical enough to be included anyways, the following steps can be taken:

  1. For lightweight dependencies, consider to vendor the package if permissible by license.
  2. Create request to current maintainer to make the dependency available.
  3. Maintain the PyPI project and conda-forge recipes ourselves.

Updating dependencies

This is the standard workflow for updating the aiida-core dependencies:

1. Update the setup.json file:

Modify the affected entries in setup.json and then update all dependent files with

./util/dependency_management.py generate-all

The command will also be executed by the pre-commit hook (if installed).

For packages that are named differently between PyPI and conda-forge, you might need to add an entry to the SETUPTOOLS_CONDA_MAPPINGS variable within the same script.

2. Update the requirements/* files

NOTE: THE FOLLOWING STEP CURRENTLY DOES NOT WORK FOR FORKS.

After pushing the updates to a branch on aiida-core, the aiida-bot will check whether the 'requirements/*' files must be updated and create a comment on the commit and any related pull-request. The aiida-bot can be instructed to trigger an update of those files by commenting with /update-requirements on the pull request.

Continuous Integration

The consistency of the various files with setup.json is checked with a pre-commit hook that is executing the generate-all and the validate-* commands of utils/dependency_management.py. This hook is also executed for all commits as part of the aiida-core CI workflow.

Furthermore, a dedicated workflow (test-install) to check whether the current dependencies can be installed with pip and conda, and tests pass for all supported Python versions, is defined in the .github/workflows/test-install.yml file. The test-install workflow is executed nightly, as well as for all pull-requests that modify any of the dependency-related files.

Problematic dependencies

This is a list of dependencies that might be problematic for some reason.

Impact on the build and install chain of aiida-core:

  1. Critical disruption/blocker
  2. Major disruption with user impact
  3. Minor disruption without user impact
  4. Concern, but so far no impact.
  • [1] tornado: old major version <5 required, causing issues for example in the context of AiiDA lab
  • [1] pymatgen: Their use of specifying numpy as a requirement in setup_requires in the call to setuptools.setup() is causing issues, because it will trigger an installation (and eventual build!) of numpy within the build environment for pymatgen, which will fail on Python 3.5 with a version of setuptools<38.2.0 and is suspected to cause other potential problems in the future.
  • [2] sqlalchemy-utils: not available through conda-forge
  • [3] Click: Minor release version 7.1 changed the help output formatting and thus broke our tests and CI workflows.
  • [4] shortuiid: A broken release (0.5.1) was removed from PyPI after upload.
Clone this wiki locally