Skip to content

use-cases.md

Latest commit

 

History

History
309 lines (199 loc) · 10.3 KB

use-cases.md

File metadata and controls

309 lines (199 loc) · 10.3 KB
title description date tags
Use cases
Learn how to use Wave with Nextflow for container management, building, and security scanning
2024-08-22
nextflow
wave
use cases

With Nextflow and Wave, you can build, upload, and manage the container images automatically and on demand during pipeline execution. The following sections describe several common use cases.

:::tip To get started with an example Nextflow pipeline that uses Wave, see Nextflow and Wave. :::

:::note Nextflow integration with Wave requires Nextflow 22.10.0 or later. :::

Access private container repositories

Use Wave to access private repositories for your Nextflow pipelines.

**Access private container repositories**

To enable private container repository access:

  1. Add the following to your Nextflow configuration:

    wave.enabled = true
tower.accessToken = '<TOWER_ACCESS_TOKEN>'

    Replace <TOWER_ACCESS_TOKEN> with your Seqera access token.

  2. (Optional) If you created your credentials in an organization workspace, add your workspace ID to your Nextflow configuration:

    tower.workspaceId = '<TOWER_WORKSPACE_ID>'

    Replace <TOWER_WORKSPACE_ID> with your Seqera workspace ID.

  3. Configure your repository access in Seqera. See Seqera Platform credentials for more information.

  4. Add your build and build cache repository to your Nextflow configuration:

    wave.build.repository = '<BUILD_REPOSITORY>'
wave.build.cacheRepository = '<CACHE_REPOSITORY>'

    Replace the following:

    • <BUILD_REPOSITORY>: the repository to store your built container images
    • <CACHE_REPOSITORY>: the repository to store image layers for caching

Build Nextflow module containers

Wave can build and provision container images on demand for your Nextflow pipelines.

**Build Nextflow module containers**

To enable Wave to build Nextflow module containers:

  1. Add your Dockerfile to the module directory where you define the pipeline process.

  2. Enable Wave in your Nextflow configuration:

    wave.enabled = true

  3. (Optional) Set your Wave strategy to prioritize Dockerfiles in your Nextflow configuration:

    wave.strategy = ['dockerfile','container']

    :::note The container directive takes precedence over a Dockerfile by default. :::

:::warning Wave does not support ADD, COPY, or other Dockerfile commands that access files in the host file system. :::

Build Conda-based containers

Wave can provision containers based on the conda directive. This allows you to use Conda packages in your pipeline, even on cloud-native platforms like AWS Batch and Kubernetes, which do not support the Conda package manager directly.

**Build Conda-based containers**

To enable Wave to provision Conda package containers:

  1. Define your Conda packages using the conda directive in your pipeline processes.

  2. Enable Wave in your Nextflow configuration:

    wave.enabled = true

  3. (Optional) Set your Wave strategy to prioritize conda in your Nextflow configuration:

    wave.strategy = ['conda']

    :::note The container directive or a Dockerfile takes precedence over the conda directive by default. :::

    :::info Nextflow 23.10.0 or later automatically includes the conda-forge::procps-ng package in provisioned containers. This package includes the ps command. :::

  4. Set your Conda channel priority:

    conda.channels = '<CONDA_CHANNELS>'

    Replace <CONDA_CHANNELS> with a comma-separated list of your channel priorities.

Build Singularity containers

Nextflow can build Singularity native images on demand using a Singularityfile or Conda packages. Images are uploaded to an OCI-compliant container registry of your choice and stored as an ORAS artifact.

:::note Requires Nextflow version 23.09.0-edge or later and a version of Singularity (or Apptainer) that supports pulling images using the oras: pseudo-protocol. :::

**Build Singularity containers**

To enable provisioning of Singularity images:

  1. Add the following to your Nextflow configuration:

    wave.enabled = true
wave.freeze = true
wave.strategy = ['conda']
singularity.enabled = true

  2. (Optional) To store your Singularity image files in a private registry:

    1. Configure your repository access in Seqera. See Seqera Platform credentials for more information.

    2. Add your build repository to your Nextflow configuration:

      wave.build.repository = '<BUILD_REPOSITORY>'

      Replace <BUILD_REPOSITORY> with your OCI-compliant container registry.

  3. Grant access to the repository on compute nodes:

    singularity remote login <REMOTE_ENDPOINT>

    Replace <REMOTE_ENDPOINT> with your Singularity remote endpoint. See Singularity remote login for more information.

  4. (Optional) To build Singularity native images, disable both singularity.ociAutoPull and singularity.ociMode in your Nextflow configuration. See Nextflow configuration for more information.

Mirror containers across registries

Wave enables mirroring by copying containers used by your pipeline to a container registry of your choice. Your pipeline can then pull containers from the target registry instead of the original registry.

**Mirror containers across registries**

To enable container mirroring:

  1. Add the following to your Nextflow configuration:

    wave.enabled = true
wave.mirror = true
tower.accessToken = '<TOWER_ACCESS_TOKEN>'

    Replace <TOWER_ACCESS_TOKEN> with your Seqera access token.

  2. Configure your private repository access in Seqera. See Seqera Platform credentials for more information.

  3. Add your build repository to your Nextflow configuration:

    wave.build.repository = '<BUILD_REPOSITORY>'

    Replace <BUILD_REPOSITORY> with your container registry.

Security scan containers

Wave scans containers used in your Nextflow pipelines for security vulnerabilities. This feature helps you ensure that your pipelines use secure container images by identifying potential security risks before and during pipeline execution.

**Security scan containers**

To enable container security scanning:

  1. Add the following to your Nextflow configuration:

     ```groovy
 wave.enabled = true
 wave.scan.mode = 'required'
 tower.accessToken = '<TOWER_ACCESS_TOKEN>'
 ```

    Replace <TOWER_ACCESS_TOKEN> with your Seqera access token.

  2. Add the acceptable vulnerability levels to your Nextflow configuration:

    wave.scan.allowedLevels = 'low,medium'

    Accepted vulnerability levels include: low, medium, high, and critical.

:::note When you set wave.scan.mode to required, Wave blocks pipeline execution if containers have vulnerabilities above the specified threshold. The scanning uses the Common Vulnerabilities Scoring System (CVSS) to assess security risks. :::

:::note Scan results expire after seven days. When a container is accessed after this period, Wave automatically re-scans it to ensure up-to-date security assessments. :::

Use Wave with Fusion

Wave containers allow you to run your containerized pipelines with the Fusion file system. Wave with Fusion enables you to use an object storage bucket, such as AWS S3 or Google Cloud Storage, as your pipeline work directory.

**Use Wave with Fusion**

To enable Wave with Fusion, add the following to your Nextflow configuration:

```groovy
wave.enabled = true
fusion.enabled = true
tower.accessToken = '<TOWER_ACCESS_TOKEN>'
```

Replace `<TOWER_ACCESS_TOKEN>` with your [Seqera access token](../tutorials/nextflow-wave.mdx#create-your-seqera-access-token).

:::note For more information about Fusion capabilities and configuration options, see the Fusion file system documentation. :::

Workflow bin scripts with Fusion

When using Wave with Fusion, your workflow's bin/ directory is automatically bundled into the container image. This differs from standard Nextflow behavior where bin scripts are staged separately to the work directory at runtime.

**How bin script bundling works**

With Fusion enabled:

  • The bin/ directory contents are bundled into the container at /usr/local/bin/
  • Changes to bin scripts trigger a container rebuild (the fingerprint includes file content hashes)
  • Remote bin directory upload is disabled

Without Fusion (standard Wave):

  • The bin/ directory is NOT bundled into the container by default
  • Scripts are uploaded separately to cloud storage at runtime
  • Changes to bin scripts do NOT trigger a container rebuild

To explicitly enable bin bundling without Fusion:

wave.enabled = true
wave.bundleProjectResources = true

:::warning If you freeze an image with wave.freeze=true and later run with Wave disabled (wave.enabled=false), your pipeline will use the bin scripts that were baked into the frozen image at build time. Local changes to bin scripts will not be reflected. To pick up script changes, re-enable Wave to trigger a rebuild with the updated fingerprint. :::