Docs: Improve Conda docs - PyPI + lock files (#5531) [ci skip]

Signed-off-by: Phil Ewels <[email protected]>
Co-authored-by: Christopher Hakkaart <[email protected]>
Co-authored-by: Paolo Di Tommaso <[email protected]>

Author: 3 people

docs/conda.md

@@ -6,7 +6,7 @@
 
 Nextflow has built-in support for Conda that allows the configuration of workflow dependencies using Conda recipes and environment files.
 
-This allows Nextflow applications to use popular tool collections such as [Bioconda](https://bioconda.github.io) whilst taking advantage of the configuration flexibility provided by Nextflow.
+This allows Nextflow applications to use popular tool collections such as [Bioconda](https://bioconda.github.io) and the [Python Package index](https://pypi.org/), whilst taking advantage of the configuration flexibility provided by Nextflow.
 
 ## Prerequisites
 
@@ -22,7 +22,7 @@ Dependencies are specified by using the {ref}`process-conda` directive, providin
 Conda environments are stored on the file system. By default, Nextflow instructs Conda to save the required environments in the pipeline work directory. The same environment may be created/saved multiple times across multiple executions when using different work directories.
 :::
 
-You can specify the directory where the Conda environments are stored using the `conda.cacheDir` configuration property. When using a computing cluster, make sure to use a shared file system path accessible from all compute nodes. See the {ref}`configuration page <config-conda>` for details about Conda configuration. 
+You can specify the directory where the Conda environments are stored using the `conda.cacheDir` configuration property. When using a computing cluster, make sure to use a shared file system path accessible from all compute nodes. See the {ref}`configuration page <config-conda>` for details about Conda configuration.
 
 :::{warning}
 The Conda environment feature is not supported by executors that use remote object storage as a work directory. For example, AWS Batch.
@@ -62,6 +62,7 @@ The usual Conda package syntax and naming conventions can be used. The version o
 
 The name of the channel where a package is located can be specified prefixing the package with the channel name as shown here `bioconda::bwa=0.7.15`.
 
+(conda-env-files)=
 ### Use Conda environment files
 
 Conda environments can also be defined using one or more Conda environment files. This is a file that lists the required packages and channels structured using the YAML format. For example:
@@ -77,20 +78,6 @@ dependencies:
   - bwa=0.7.15
 ```
 
-This other example shows how to leverage a Conda environment file to install Python packages from the [PyPI repository](https://pypi.org/)), through the `pip` package manager (which must also be explicitly listed as a required package):
-
-```yaml
-name: my-env-2
-channels:
-  - defaults
-dependencies:
-  - pip
-  - pip:
-    - numpy
-    - pandas
-    - matplotlib
-```
-
 Read the Conda documentation for more details about how to create [environment files](https://conda.io/docs/user-guide/tasks/manage-environments.html#creating-an-environment-file-manually).
 
 The path of an environment file can be specified using the `conda` directive:
@@ -110,7 +97,26 @@ process foo {
 The environment file name **must** have a `.yml` or `.yaml` extension or else it won't be properly recognised.
 :::
 
-Alternatively, it is possible to provide the dependencies using a plain text file, just listing each package name as a separate line. For example:
+(conda-pypi)=
+### Python Packages from PyPI
+
+Conda environment files can also be used to install Python packages from the [PyPI repository](https://pypi.org/), through the `pip` package manager (which must also be explicitly listed as a required package):
+
+```yaml
+name: my-env-2
+channels:
+  - defaults
+dependencies:
+  - pip
+  - pip:
+    - numpy
+    - pandas
+    - matplotlib
+```
+
+### Conda text files
+
+It is possible to provide dependencies by listing each package name as a separate line in a plain text file. For example:
 
 ```
 bioconda::star=2.5.4a
@@ -122,6 +128,43 @@ bioconda::multiqc=1.4
 Like before, the extension matters. Make sure the dependencies file has a `.txt` extension.
 :::
 
+### Conda lock files
+
+The final way to provide packages to Conda is with [Conda lock files](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#identical-conda-envs).
+
+These are generated from existing Conda environments using the following command:
+
+```bash
+conda list --explicit > spec-file.txt
+```
+
+or if using Mamba / Micromamba:
+
+```bash
+micromamba env export --explicit > spec-file.txt
+```
+
+Conda lock files can also be downloaded from [Wave](https://seqera.io/wave/) build pages.
+
+These files include every package and their dependencies. As such, no Conda environment resolution step is needed. This is faster and more reproducible.
+
+The files contain package URLs and an optional md5hash for each download to confirm identity:
+
+```
+# micromamba env export --explicit
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: linux-64
+@EXPLICIT
+https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2#d7c89558ba9fa0495403155b64376d81
+https://conda.anaconda.org/conda-forge/linux-64/libgomp-13.2.0-h77fa898_7.conda#abf3fec87c2563697defa759dec3d639
+https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2#73aaf86a425cc6e73fcf236a5a46396d
+https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-13.2.0-h77fa898_7.conda#72ec1b1b04c4d15d4204ece1ecea5978
+# .. and so on
+```
+
+To use with Nextflow, simply set the `conda` directive to the lock file path.
+
 ### Use existing Conda environments
 
 If you already have a local Conda environment, you can use it in your workflow specifying the installation directory of such environment by using the `conda` directive:

docs/wave.md

@@ -90,6 +90,8 @@ conda.channels = 'conda-forge,bioconda'
 ```
 :::
 
+Packages from the [Python Package Index](https://pypi.org/) can also be added to a Conda `environment.yml` file. See {ref}`Conda and PyPI <conda-pypi>` for more information.
+
 (wave-singularity)=
 
 ### Build Singularity native images