Skip to content

CLAUDE.md

Latest commit

 

History

History
651 lines (493 loc) · 23.8 KB

CLAUDE.md

File metadata and controls

651 lines (493 loc) · 23.8 KB

CLAUDE.md -- Developer guide for niworkflows

Project overview

niworkflows (NeuroImaging Workflows) is the shared library of reusable Nipype interfaces, workflows, utilities, and visualization tools that underpin the NiPreps ecosystem. It is consumed as a dependency by fMRIPrep, sMRIPrep, dMRIPrep, and other NiPreps pipelines. It is not a standalone BIDS-App; it is a building-block library.

Key responsibilities:

  • Interfaces: Nipype interface wrappers for ANTs, FreeSurfer, FSL, NiBabel, NiTransforms, and custom Python processing (BIDS I/O, image manipulation, confounds, header fixing, morphology, reportlets, etc.).
  • Workflows: Reusable sub-workflows (brain extraction, BOLD reference generation, EPI reference alignment, spatial normalization, coregistration).
  • Engine extensions: LiterateWorkflow (methods boilerplate via __desc__), MultiProcPlugin (lightweight execution plugin), and workflow splicer (node-level hot-swap for tagged workflows).
  • Utilities: BIDS querying/validation, space tracking (SpatialReferences), connection helpers, image manipulation, and miscellaneous tools.
  • Visualization: Plotting routines for reportlets (registration overlays, brain masks, segmentation contours, carpet plots, confound time series).
  • Reports: Jinja2-based HTML report builder for BIDS-Apps.

License: Apache 2.0.

Repository structure

niworkflows/
  __init__.py            # Package init; sets up logging, acres Loader
  __about__.py           # Package metadata (name, credits)
  _version.py            # Auto-generated by hatch-vcs (do not edit)

  anat/                  # Anatomical processing workflows & interfaces
    ants.py              # ANTs brain extraction workflow (init_brain_extraction_wf)
    coregistration.py    # Coregistration workflows
    freesurfer.py        # FreeSurfer-based anatomical workflows
    skullstrip.py        # Skull-stripping utilities
    tests/

  cli/                   # Command-line entry points
    boldref.py           # `niworkflows-boldref` CLI

  data/                  # Package data files (loaded via acres.Loader)
    nipreps.json         # BIDS-Derivatives entity config for PyBIDS
    *.json               # ANTs registration parameter sets
    sentinel.nii.gz      # Sentinel NIfTI for testing
    tests/               # Test fixture data

  dwi/                   # DWI utilities (currently minimal)

  engine/                # Nipype engine extensions
    __init__.py          # Re-exports LiterateWorkflow as Workflow, splicer
    workflows.py         # LiterateWorkflow class (__desc__, __postdesc__)
    plugin.py            # MultiProcPlugin (lightweight ProcessPoolExecutor)
    splicer.py           # Workflow splicing: tag() decorator + splice_workflow()
    tests/

  func/                  # Functional (BOLD) workflows
    util.py              # init_bold_reference_wf (reference + mask generation)
    tests/

  interfaces/            # Nipype interfaces (the largest subpackage)
    bids.py              # BIDSInfo, BIDSDataGrabber, DerivativesDataSink,
                         #   PrepareDerivative, SaveDerivative, BIDSURI
    bold.py              # NonsteadyStatesDetector and BOLD-specific interfaces
    cifti.py             # CIFTI-related interfaces
    confounds.py         # Confound computation interfaces
    fixes.py             # FixHeaderRegistration, FixHeaderApplyTransforms, FixN4
    freesurfer.py        # FreeSurfer interface wrappers
    header.py            # ValidateImage, CopyXForm, CopyHeader, SanitizeImage
    images.py            # RegridToZooms, RobustAverage, SignalExtraction, etc.
    itk.py               # ITK transform utilities
    morphology.py        # Morphological operations on images
    nibabel.py           # IntensityClip, ApplyMask, Binarize, etc.
    nilearn.py           # Nilearn-based interfaces
    nitransforms.py      # NiTransforms interface (ConvertAffine)
    norm.py              # SpatialNormalization (robust ANTs T1-to-MNI)
    patches.py           # Monkey-patches for upstream bugs
    plotting.py          # Confound correlation plot interfaces
    probmaps.py          # Probability map interfaces
    space.py             # Space-handling interfaces
    surf.py              # Surface-related interfaces
    utility.py           # KeySelect, AddTSVHeader, JoinTSVColumns, DictMerge
    workbench.py         # Connectome Workbench interface wrappers
    reportlets/          # Report-capable interface mixins
      base.py            # RegistrationRC, SegmentationRC, SurfaceRC mixins
      masks.py           # Mask overlay reportlets
      registration.py    # Registration overlay reportlets (RPT variants)
      segmentation.py    # Segmentation overlay reportlets
    tests/               # Interface unit tests

  reports/               # HTML report generation
    core.py              # Report class (Jinja2 template rendering)
    default.yml          # Default report configuration
    report.tpl           # Jinja2 HTML template
    tests/

  tests/                 # Top-level integration tests
    conftest.py          # Shared fixtures (testdata_dir, ds000030_dir, etc.)
    data/                # Test data for integration tests
    test_confounds.py
    test_registration.py
    test_segmentation.py
    test_utils.py
    test_viz.py

  testing.py             # Test environment setup (data paths, canaries, tool checks)
  conftest.py            # Root conftest (doctest namespace, session fixtures)

  utils/                 # General utilities
    bids.py              # collect_data, BIDS layout helpers, DEFAULT_BIDS_QUERIES
    connections.py       # listify(), pop_file() -- inline helpers for connect()
    debug.py             # Debugging utilities
    images.py            # Low-level NIfTI manipulation (_copyxform, etc.)
    misc.py              # get_template_specs, _copy_any, unlink, etc.
    spaces.py            # SpatialReferences, Reference (space tracking)
    testing.py           # Utilities for test helpers
    timeseries.py        # Time-series manipulation utilities
    tests/

  viz/                   # Visualization (matplotlib/SVG-based)
    notebook.py          # Jupyter notebook display helpers
    plots.py             # fMRIPlot (carpet plot), confound correlation plots
    utils.py             # compose_view, plot_registration, cuts_from_bbox

  workflows/             # Reusable Nipype workflows
    epi/
      refmap.py          # init_epi_reference_wf (EPI reference generation)
      tests/

Development setup

Preferred: pixi (linux-64)

pixi install
pixi run -e test pytest niworkflows/

The pixi configuration is in pyproject.toml under [tool.pixi.*] sections. It pins Python 3.12 and includes conda packages for ANTs, FSL tools, and system libraries.

Alternative: pip editable install

python -m pip install -e ".[tests]"

Environment variables

Variable Purpose
TEST_DATA_HOME Path to test data (default: ~/.cache/stanford-crn)
TEST_OUTPUT_DIR Where to write test outputs
TEST_WORK_DIR Nipype working directory for tests
TEMPLATEFLOW_HOME TemplateFlow cache directory
FREESURFER_HOME FreeSurfer installation (for FS-dependent tests)
NO_ET Set to 1 to disable error telemetry (always set in tests)

Fetching test data

Test data is managed via DataLad from nipreps-data/niworkflows-data:

pip install datalad datalad-osf
datalad install -r -s https://github.com/nipreps-data/niworkflows-data.git ~/.cache/stanford-crn
cd ~/.cache/stanford-crn
datalad get -J 2 -r ds000003 ds000030/sub-10228/func

Fetching TemplateFlow templates

Before running tests, pre-cache required templates:

python docker/fetch_templates.py

Code style and linting

Ruff (primary linter and formatter)

Configuration is in pyproject.toml under [tool.ruff].

  • Line length: 99
  • Quote style: single quotes (enforced by ruff format)
  • Import sorting: isort-compatible via ruff's I rule
  • Enabled rule categories: I, UP, S, B, A, C4, PT, PERF, RUF, and more
  • Per-file ignores: F401 in __init__.py, S101 in test files

Run manually:

ruff check --fix niworkflows/
ruff format niworkflows/

Pre-commit hooks

Configured in .pre-commit-config.yaml:

  • trailing-whitespace, end-of-file-fixer, check-yaml/json/toml, check-added-large-files
  • ruff check + ruff format

Data files are excluded from pre-commit (exclude: ".*/data/.*").

Codespell

codespell . --skip="*/data/*,*/docs/_build/*,./examples/viz-report.*"

tox environments

tox -e style       # Check style (ruff check + ruff format --diff)
tox -e style-fix   # Auto-fix style
tox -e spellcheck   # Run codespell

Testing

Running tests

# Full test suite (requires test data + templates pre-cached)
pytest niworkflows/ -n auto

# Single test file
pytest niworkflows/interfaces/tests/test_bids.py -svx

# Single test
pytest niworkflows/interfaces/tests/test_bids.py::test_name -svx

# Doctests are enabled globally (--doctest-modules in pyproject.toml)
# To run just doctests for a module:
pytest --doctest-modules niworkflows/utils/connections.py

# Via tox (fetches templates first)
tox

Pytest configuration

Defined in pyproject.toml under [tool.pytest.ini_options]:

  • testpaths = ["niworkflows"] -- tests live alongside source
  • xfail_strict = true -- xfail must actually fail
  • addopts includes -svx, --doctest-modules, --cov=niworkflows
  • env = "PYTHONHASHSEED=0" -- deterministic hashing (via pytest-env)

Test fixtures

Root conftest (niworkflows/conftest.py):

  • legacy_printoptions (session): numpy print compatibility
  • _add_np (autouse): populates doctest namespace with np, nb, pd, datadir, bids_collect_data, test_data, nifti_fname; creates a temp directory and chdir into it for doctests
  • testdata_dir: path to testing.data_dir
  • ds000030_dir: path to ds000030 (skips if unavailable)
  • workdir, outdir: from env vars

Interfaces conftest (niworkflows/interfaces/conftest.py):

  • data_dir (module-scoped): points to interfaces/tests/data/
  • _docdir (autouse): copies test data to tmp_path for doctests

Test data locations

  • niworkflows/data/tests/ -- small fixture data shipped with the package
  • niworkflows/tests/data/ -- integration test data (excluded from wheel)
  • niworkflows/interfaces/tests/data/ -- interface test fixtures
  • ~/.cache/stanford-crn/ -- large test datasets (DataLad-managed)

Writing tests

  • Place tests in a tests/ subdirectory adjacent to the module being tested
  • Name test files test_<module>.py
  • Use pytest.mark.skipif or canary decorators for tests requiring external tools (FSL, FreeSurfer, AFNI) or test data
  • Use pytest.mark.parametrize for combinatorial testing
  • Prefer doctests for pure-function utilities (they serve as documentation)
  • For interfaces, test both the _run_interface logic and the output spec

Nipype interface conventions

Interface naming pattern

# Input spec: _<InterfaceName>InputSpec
class _MyInterfaceInputSpec(BaseInterfaceInputSpec):
    in_file = File(exists=True, mandatory=True, desc='input image')

# Output spec: _<InterfaceName>OutputSpec
class _MyInterfaceOutputSpec(TraitedSpec):
    out_file = File(exists=True, desc='output image')

# Interface class
class MyInterface(SimpleInterface):
    input_spec = _MyInterfaceInputSpec
    output_spec = _MyInterfaceOutputSpec

    def _run_interface(self, runtime):
        # Processing logic
        self._results['out_file'] = '/path/to/output'
        return runtime

Report-capable interfaces (RPT pattern)

Interfaces that generate visual reportlets follow a mixin pattern:

# In interfaces/reportlets/registration.py
class SpatialNormalizationRPT(RegistrationRC, SpatialNormalization):
    input_spec = _SpatialNormalizationInputSpecRPT
    output_spec = _SpatialNormalizationOutputSpecRPT

The RegistrationRC mixin (from reportlets/base.py) adds _generate_report() which produces SVG overlays. Similar mixins exist for segmentation and masks.

Fix interfaces

interfaces/fixes.py provides header-fixing wrappers around ANTs interfaces:

  • FixHeaderRegistration -- fixes sform/qform after ANTs registration
  • FixHeaderApplyTransforms -- fixes header after resampling
  • FixN4BiasFieldCorrection -- handles edge cases in N4

Always prefer these over vanilla nipype ANTs interfaces.

Nipype workflow conventions

Factory function pattern

All workflows follow the factory function convention:

def init_<name>_wf(
    omp_nthreads,
    name='<name>_wf',
    # ... other parameters
):
    """
    Build a workflow that does X.

    Workflow Graph
        .. workflow::
            :graph2use: orig
            :simple_form: yes

            from niworkflows.<module> import init_<name>_wf
            wf = init_<name>_wf(omp_nthreads=1)

    Parameters
    ----------
    ...

    Inputs
    ------
    ...

    Outputs
    -------
    ...
    """
    from niworkflows.engine.workflows import LiterateWorkflow as Workflow

    wf = Workflow(name=name)

    inputnode = pe.Node(
        niu.IdentityInterface(fields=[...]),
        name='inputnode',
    )
    outputnode = pe.Node(
        niu.IdentityInterface(fields=[...]),
        name='outputnode',
    )

    # ... internal nodes ...

    # fmt: off
    wf.connect([
        (inputnode, node_a, [('field', 'input')]),
        (node_a, outputnode, [('output', 'field')]),
    ])
    # fmt: on

    return wf

Key patterns

  1. LiterateWorkflow: Use from niworkflows.engine import Workflow instead of pe.Workflow. Supports __desc__ and __postdesc__ for auto-generated methods sections. Call wf.visit_desc() to collect boilerplate.

  2. inputnode/outputnode contract: Every workflow exposes its interface via IdentityInterface nodes named inputnode and outputnode.

  3. Connection formatting: Use # fmt: off / # fmt: on around wf.connect([...]) blocks to prevent ruff from reformatting them.

  4. Inline functions in connections: Small transform functions can be passed inline: (('field', function), 'target_field'). Import inside the function body for nipype serialization.

  5. KeySelect: For dynamic dispatch from keyed collections (e.g., selecting a template's transforms from a list keyed by space name). Defined in interfaces/utility.py.

  6. Workflow splicing (engine/splicer.py): Use @tag('name') to mark workflow factories, then splice_workflow(root_wf, {'name': replacement}) to hot-swap nodes in a built workflow graph.

BIDS interfaces

DerivativesDataSink

The primary interface for writing BIDS-Derivatives compliant outputs. Located in interfaces/bids.py. Key features:

  • Extracts entities from source files automatically
  • Supports custom entities via allowed_entities constructor parameter
  • Handles compression, data type coercion, and header fixing
  • Generates sidecar JSON metadata
  • Supports BIDSURI for provenance tracking

PrepareDerivative + SaveDerivative (new split pattern)

The refactored two-step approach:

  • PrepareDerivative: determines output path, fixes headers, builds metadata
  • SaveDerivative: copies the prepared file to the derivatives directory

This split enables the BIDSURI pattern where provenance metadata can be injected between preparation and saving.

BIDSURI

Interface for tracking data provenance via the BIDS URI scheme (bids::relative/path). Used by fMRIPrep and downstream tools.

Key abstractions

SpatialReferences (utils/spaces.py)

Tracks and filters output spaces. Uses attrs-based Reference objects with template names, cohort/resolution/density specs. Distinguishes standard (TemplateFlow) from nonstandard (T1w, fsnative, func, etc.) references.

Resource loading (data/)

Uses acres.Loader for package data access:

from niworkflows import load_resource
path = load_resource('data/nipreps.json')

from niworkflows.data import load
content = load.readable('nipreps.json').read_text()

Connection helpers (utils/connections.py)

  • listify(value) -- normalize to list (or pass through None)
  • pop_file(in_files) -- select first file from a list

These are designed for inline use in Nipype connections:

(inputnode, node, [(('in_files', listify), 'input_image')])

CI/CD

GitHub Actions (.github/workflows/tox.yml)

Primary CI pipeline:

  • build: builds sdist/wheel, checks with twine
  • get_data: fetches test data via DataLad (cached)
  • test: matrix of Python 3.12-3.14 x {min, latest, pre} deps
  • checks: ruff style + codespell

CircleCI (.circleci/config.yml)

Docker-based integration tests with full neuroimaging tool stack:

  • Builds Docker image with ANTs, FSL, FreeSurfer
  • Runs integration tests against real neuroimaging data
  • Tests that require external tools run here (not in GitHub Actions)

codecov

Coverage config in pyproject.toml under [tool.coverage.*]. Reports uploaded to Codecov from both CI systems.

Common tasks

Adding a new interface

  1. Create or edit the appropriate file in niworkflows/interfaces/
  2. Follow the _InputSpec / _OutputSpec / Interface(SimpleInterface) pattern
  3. If the interface should generate a reportlet, create an RPT variant in interfaces/reportlets/ using the mixin pattern
  4. Add doctests with examples
  5. Add unit tests in interfaces/tests/test_<module>.py

Adding a new workflow

  1. Create or edit a file in the appropriate subpackage (anat/, func/, workflows/epi/, etc.)
  2. Follow the init_<name>_wf() factory function pattern
  3. Use LiterateWorkflow (imported as Workflow from niworkflows.engine)
  4. Define inputnode/outputnode with IdentityInterface
  5. Add a .. workflow:: directive in the docstring for auto-generated graphs
  6. Add __desc__ text for methods boilerplate if appropriate
  7. Add tests in the adjacent tests/ directory

Modifying an ANTs registration parameter set

Registration parameters are stored as JSON files in niworkflows/data/:

  • antsBrainExtraction_precise.json / antsBrainExtraction_testing.json
  • t1w-mni_registration_*.json
  • boldref-mni_registration_*.json

Edit the JSON files directly. The SpatialNormalization interface in interfaces/norm.py loads these via flavor parameter.

Adding a new BIDS entity for derivatives

Edit niworkflows/data/nipreps.json to add the entity definition under entities and update default_path_patterns if needed. The PrepareDerivative and DerivativesDataSink interfaces will pick it up automatically.

Build system

  • Build backend: hatchling with hatch-vcs (version from git tags)
  • Version scheme: release-branch-semver
  • Version file: niworkflows/_version.py (auto-generated, do not edit)
  • Python support: >= 3.12

Git conventions

  • Main branch: master
  • Feature branches: enh/<description>, fix/<description>, maint/<description>, doc/<description>
  • Release tags: standard semver (e.g., 1.14.4)
  • Changelog: CHANGES.rst (RST format)
  • Pre-commit hooks run automatically; CI enforces style checks
  • Dependabot manages dependency updates

Architecture notes

Separation of concerns

niworkflows deliberately does not include:

  • A config singleton (that is fMRIPrep's pattern)
  • Fit/transform level gating (that is the BIDS-App's responsibility)
  • Full pipeline orchestration (that is the App layer)

It provides the building blocks that Apps compose.

Header-fixing philosophy

Many ANTs operations produce outputs with incorrect NIfTI headers (mismatched qform/sform). The fixes.py module wraps these interfaces to ensure headers are always consistent. Always use FixHeader* variants in workflows.

Data loading via acres

The acres.Loader pattern replaces pkg_resources / importlib.resources. All package data access goes through niworkflows.data.load or niworkflows.load_resource. This works correctly with both installed packages and editable installs.

Current state and known issues (as of 2026-02-18)

86 open issues. The following areas are most relevant to active development:

Active development: upstreaming from fMRIPrep

The biggest ongoing effort is centralizing general-purpose code from fMRIPrep into niworkflows (#854 is the master tracking issue). Specific items:

  • BIDSURI interface (#1024, PR #1031): provenance tracking via bids:: URIs. Being upstreamed. Critical for dMRIPrep and NiBabies.
  • ConvertAffine interface (#1025): transform format conversion (ITK/FSL/LTA).
  • Workbench interfaces (#1026): MetricDilate, MetricResample, VolumeToSurfaceMapping, etc. Critical for CIFTI/surface pipeline reuse.
  • Clip / Label2Mask (#1027): general-purpose image math interfaces.
  • extract_entities / check_pipeline_version (#1028): BIDS utility functions.
  • Full fmriprep.interfaces.bids port (#985): planned but not yet itemized.

Active development: BIDS entity configuration

PR #1023 proposes replacing the hand-maintained nipreps.json with schema-driven config generation from the BIDS schema. This would systematically fix entity ordering bugs (#1018) and missing PET metadata (#946). Until this lands, be aware that:

  • Entity ordering in nipreps.json may not match BIDS spec (e.g., seg entity missing from some path patterns causes it to appear after desc).
  • PET metadata (petref suffix, pet datatype) is incomplete.

Active development: NiReports migration

Visual elements (plots, reportlets) are being migrated to the NiReports package (#787, PR #789). Shim imports with deprecation warnings are being added. Code in niworkflows/viz/ and niworkflows/interfaces/reportlets/ should be treated as transitional -- new visualization work should target NiReports instead.

Known bugs to watch for

  • #1029 (HIGH): DerivativesDataSink produces corrupted NIfTI files when input is big-endian and data type coercion is needed. The byte order of the header becomes inconsistent with the data volume. Affects unsafe_write() in interfaces/bids.py and _copyxform() in utils/images.py.
  • #1000 (HIGH): antsAI can fail randomly during EPI-to-template initialization, producing wildly wrong affines. PR #1001 splits a premask workflow to mitigate this.
  • #1021: create_cfm() in interfaces/norm.py has an orientation bug when combining lesion masks: one volume is canonicalized with nb.as_closest_canonical() but the other is not.
  • #959: numpy 2.3+ deprecates np.bool_ as integer, breaking nipype's Select interface. Upstream nipype fix needed. Affects anat/coregistration.py where np.bool_ output feeds Select.index.
  • #928: brain_extraction_wf.inu_n4_final can produce extreme values that crash downstream Atropos. Workaround: use inu_n4 outputs instead.
  • #804: BIDSFreeSurferDir has a filesystem race when parallel jobs copy FreeSurfer directories simultaneously.

Planned changes to space handling

  • Surface/volume space distinction (#881, PR #883): Reference will need to handle combined surface+volume space specifications for CIFTIs. Current direction: use + in space entity (e.g., space-dhcpAsym42+MNIInfant1).
  • Custom resolutions (#997): proposed syntax like res-1p5mm or res-6x6x3mm for arbitrary output resolutions.
  • nativemax/nativemin (#889): isotropic resampling to min/max native voxel size. May use a register-based API for new space types.
  • SUIT template (#969): not yet in valid spaces list; blocked on TemplateFlow availability.

Deprecations in progress

  • niworkflows.viz and niworkflows.interfaces.reportlets: being replaced by NiReports. Import from NiReports for new code.
  • nipreps.json manual editing: being replaced by schema-driven generation (PR #1023). Avoid adding new entities manually if possible.
  • nilearn.image.resample_img (#967): nilearn 0.13 will change default to force_resample=True. Downstream impact to be evaluated.