Skip to content

Reduce inode usage in participant_job.sh #333

@tien-tong

Description

@tien-tong

Summary

In participant_job.sh, the first step is a full-tree datalad get -n of the input dataset and then prunes other subjects/sessions with find/rm -rf:

# pull down input data (but don't retrieve the data content) and remove other sub's data:
echo "# Pull down the input subject (or dataset) but don't retrieve data contents:"
datalad get -n "inputs/data/BIDS_dl/${subid}"
(cd inputs/data/BIDS_dl && find . -type d -name 'sub*' | grep -v "$subid" | xargs rm -rf)
(cd inputs/data/BIDS_dl/"${subid}" && find . -type d -name 'ses*' | grep -v "$sesid" | xargs rm -rf)

Even without retrieving file contents, this still materializes the full directory tree for the whole input RIA (all subjects/sessions). On large BIDS datasets this consumes too many inodes, and we repeatedly hit inode limits on shared filesystems.

Additional details

Next steps

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions