Skip to content

CLAUDE.md

Latest commit

 

History

History
141 lines (119 loc) · 7 KB

CLAUDE.md

File metadata and controls

141 lines (119 loc) · 7 KB

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Nextflow is a scientific workflow management system built primarily in Groovy and Java. It enables the creation of scalable, portable, and reproducible computational pipelines using a dataflow programming model. The project follows a modular architecture with a plugin-based system for cloud providers and specialized features.

Development Commands

Build System

  • Primary build tool: Gradle with wrapper (./gradlew)
  • Quick commands via Makefile:
    • make compile - Compile sources and export classpath
    • make assemble - Build info, compile, and assemble
    • make test - Run all tests
    • make check - Run verification tasks
    • make clean - Clean build artifacts

Testing

  • Unit tests: make test or ./gradlew test
    • Uses Spock Framework (Groovy-based testing)
    • JaCoCo integration for code coverage
  • Specific test class: ./gradlew test --tests "SomeTestClass"
  • Specific module: make test module=nextflow
  • Smoke tests: make smoke or NXF_SMOKE=1 ./gradlew test (skips long-running tests, network-dependent tests, and cloud provider integration tests)
  • Integration tests: Run from tests/ directory using test runner scripts
  • Cloud validation tests: Located in validation/ directory (requires credentials)

Development Workflow

  • Development launcher: ./launch.sh run script.nf (uses development build)
  • Dependency analysis: make deps or make deps config=runtime
  • Install locally: make install (installs to Maven local)

Architecture

Core Modules (modules/)

  • nextflow: Main application module with core workflow engine, CLI, AST transformations, executors
  • nf-commons: Shared utilities, plugin system infrastructure, extension methods
  • nf-httpfs: HTTP filesystem support and custom providers
  • nf-lang: Language parsing, ANTLR grammars, AST implementation
  • nf-lineage: Data lineage tracking and workflow execution history

Plugin System (plugins/)

  • Cloud providers: nf-amazon (AWS), nf-azure (Azure), nf-google (GCP)
  • Execution platforms: nf-k8s (Kubernetes)
  • Services: nf-tower (Seqera Platform), nf-wave (container management)
  • Other: nf-console (interactive interface), nf-cloudcache (cloud caching)

Key Technologies

  • Language: Groovy 4.0.29 (Java-compatible, targeting Java 17)
  • Concurrency: GPars 1.2.1 (Actor model, parallel/concurrent programming)
  • Build: Gradle with Java 21 toolchain
  • Parsing: ANTLR for Nextflow DSL
  • Serialization: Kryo
  • Database: LevelDB for local caching
  • Version Control: JGit integration

Testing Structure

  • Unit tests: Each module has src/test/groovy/ with Spock Framework tests
  • Integration tests: tests/ directory with .nf workflows and expected outputs
  • Legacy tests: tests-v1/ for DSL v1 compatibility
  • Validation tests: validation/ directory for cloud provider end-to-end testing
  • Documentation tests: docs/snippets/ for verifying documentation examples

Development Notes

Code Standards

  • All code must include Apache 2.0 license headers
  • Contributions require Developer Certificate of Origin (DCO) sign-off
  • Use existing code patterns and conventions from similar modules
  • Follow Groovy idioms and leverage the Nextflow DSL patterns

Common Development Tasks

  • Local development: Use make compile to build and ./launch.sh to test changes
  • Adding features: First check modules like nextflow for core features or create plugins for specialized functionality
  • Plugin development: Follow existing plugin patterns in plugins/ directory
  • Testing changes: Always run make test before committing
  • Cloud testing: Use validation scripts in validation/ directory with appropriate credentials

Build Configuration

  • Java toolchain uses version 21 for development, targets Java 17 compatibility
  • Uses shadow plugin for creating fat JARs
  • Maven publication to S3-based Seqera repositories
  • Multi-module project with shared dependencies managed in root build.gradle

Git conventions

  • DCO sign-off required: All commits must be signed by adding a Signed-off-by line to the commit message or by using the -s option (see CONTRIBUTING.md for details).
  • Always use sign-off: Use git commit -s or git commit --signoff for commits to avoid DCO bot issues
  • CI control tags: Use special tags in commit messages to control CI behavior:
    • [ci skip] - Skip the execution of CI tests
    • [ci fast] - Run only unit tests and skip integration tests
    • [e2e stage] - Run end-to-end tests vs Seqera platform stage environment
    • [e2e prod] - Same but against production platform
    • [release] - Trigger release process

Important Files

  • VERSION: Define the current version number
  • nextflow: Launch wrapper script (updated by build process)
  • .launch.classpath: Development classpath (generated by make compile)
  • build.gradle: Root build configuration with multi-module setup
  • settings.gradle: Gradle project structure definition
  • plugins/*/VERSION: Define the version of the corresponding plugin sub-project.
  • adr/: Architecture Decision Records (ADRs) documenting significant structural and technical decisions in the project

Release process

Follow these actions to make a new release:

  • Update the changelog.txt file in each plugin sub-project (if any change has been done).
  • Update the VERSION file in in each plugin sub-project. Use a semantic version number depending the impact of the change, or do not change if no changes have been done to the plugin.
  • Update nextflowVersion attribute in the build.gradle file for plugins requiring specific Nextflow versions.
  • Commit the version and changelog files changes independently for each plugin. Use as commit message the template Bump plugin-name@version e.g. `Bump nf-amazon@2.0.0.
  • Update VERSION file in the project root using a calendar-like versioning scheme. Versions in the 4-th and 10-th month are "stable releases", e.g. 25.10.0, while versions in all other months are "edge releases", e.g. 25.09.0-edge.
  • Update the project root changelog.txt with changes since the past release. Use the git log command to determine what changed e.g. git log v<PREVIOUS VERSION>..
  • Run make releaseInfo to update the version number and generate checksums.
  • Run this command to stage for commit the release files: 
    git add \
  VERSION \
  changelog.txt \
  nextflow \
  nextflow.md5 \
  nextflow.sha1 \
  nextflow.sha256 \
  modules/nextflow/src/main/resources/META-INF/plugins-info.txt \
  modules/nextflow/src/main/resources/META-INF/build-info.properties
  • Make a commit using the [release] and [e2e prod] tags in the comment and push it upstream to trigger the release automation with GitHub action: 
    git commit -m "[release] Nextflow version 25.09.0-edge"
git push origin master