Singularity: No space left on device

[Hierakonpolis]

The processing pipeline fails because of errors such as:
OSError: [Errno 28] No space left on device: '/opt/conda/lib/python3.8/site-packages/skimage/data/../data/brick.png' -> '/home/fmriprep/.cache/scikit-image/0.18.3/data/../data/brick.png'
I tried setting the workdir explicitly following nipreps/fmriprep#2120 and nipreps/fmriprep#1291 and the issue remains.

What version of fMRIPrep are you using?

Latest fmriprep-rodents from docker as of April 19th

What kind of installation are you using? Containers (Singularity, Docker), or "bare-metal"?

Singularity

What is the exact command-line you used?

singularity run --cleanenv ~/singularity_images/fmriprep-rodents19042022.simg /home/mri/dataset /home/mri/test_fmriprep participant --fs-license-file licensefile -w ~/scratchdir

Have you checked that your inputs are BIDS valid?

The validator checks out

Did fMRIPrep generate the visual report for this particular subject? If yes, could you share it?

No

Can you find some traces of the error reported in the visual report (at the bottom) or in crashfiles?

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/opt/conda/lib/python3.8/site-packages/niworkflows/interfaces/nibabel.py", line 396, in _run_interface
    self._results["out_file"] = _advanced_clip(
  File "/opt/conda/lib/python3.8/site-packages/niworkflows/interfaces/nibabel.py", line 517, in _advanced_clip
    from skimage.morphology import ball
  File "/opt/conda/lib/python3.8/site-packages/skimage/__init__.py", line 135, in <module>
    from .data import data_dir
  File "/opt/conda/lib/python3.8/site-packages/skimage/data/__init__.py", line 270, in <module>
    _init_pooch()
  File "/opt/conda/lib/python3.8/site-packages/skimage/data/__init__.py", line 262, in _init_pooch
    _fetch(filename)
  File "/opt/conda/lib/python3.8/site-packages/skimage/data/__init__.py", line 222, in _fetch
    shutil.copy2(gh_repository_path, resolved_path)
  File "/opt/conda/lib/python3.8/shutil.py", line 432, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/conda/lib/python3.8/shutil.py", line 272, in copyfile
    _fastcopy_sendfile(fsrc, fdst)
  File "/opt/conda/lib/python3.8/shutil.py", line 163, in _fastcopy_sendfile
    raise err from None
  File "/opt/conda/lib/python3.8/shutil.py", line 149, in _fastcopy_sendfile
    sent = os.sendfile(outfd, infd, offset, blocksize)
OSError: [Errno 28] No space left on device: '/opt/conda/lib/python3.8/site-packages/skimage/data/../data/brick.png' -> '/home/fmriprep/.cache/scikit-image/0.18.3/data/../data/brick.png'

Are you reusing previously computed results (e.g., FreeSurfer, Anatomical derivatives, work directory of previous run)?

No

[eilidhmacnicol]

This looks like an issue with the mounted volumes for singularity rather than a problem with fmriprep-rodents, since similar issues were found in other NiPreps tools (see nipreps/mriqc#850 (comment) and https://sylabs.io/guides/2.5/user-guide/troubleshooting.html)

[effigies]

This looks like an issue with scikit-image downloading files we're not requesting. Are we pinning a specific version? I remember them fixing something like this a while back.

[eilidhmacnicol]

Are we pinning a specific version

Nope. In fact, I don't think it's listed as a dependency anywhere outside of this GitHub action:

fmriprep-rodents/.github/workflows/pythonpackage.yml

Line 70 in edfe415

python -m pip install numpy scipy "Cython >= 0.28.5" # sklearn needs this

Does that ring any bells?

Otherwise, it might be a problem with one of the dependencies. I'll try to recreate with the latest singularity image.

[effigies]

This is the relevant issue: scikit-image/scikit-image#6069

We need to pin scikit-image >= 0.19 in the docker container.

[Hierakonpolis]

Is there any workaround I could use in the meantime, for example to allow for some writable space that would prevent this error from happening?

[eilidhmacnicol]

Okay, so I have been trying to work out how to change .circleci/config.yml for a good chunk of the afternoon, so that tags actually push to dockerhub. @oesteban @effigies - do either of you have the bandwidth to help with this change?

Edited to add that I have also tried to do this with NiRodents lately, but I am clearly missing something.

[effigies]

Is there any workaround I could use in the meantime, for example to allow for some writable space that would prevent this error from happening?

Bind a writeable directory into the data directory that failed to be written to.