-
Notifications
You must be signed in to change notification settings - Fork 7
Reduce inode usage in participant_job.sh #333
Copy link
Copy link
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Summary
In participant_job.sh, the first step is a full-tree datalad get -n of the input dataset and then prunes other subjects/sessions with find/rm -rf:
# pull down input data (but don't retrieve the data content) and remove other sub's data:
echo "# Pull down the input subject (or dataset) but don't retrieve data contents:"
datalad get -n "inputs/data/BIDS_dl/${subid}"
(cd inputs/data/BIDS_dl && find . -type d -name 'sub*' | grep -v "$subid" | xargs rm -rf)
(cd inputs/data/BIDS_dl/"${subid}" && find . -type d -name 'ses*' | grep -v "$sesid" | xargs rm -rf)
Even without retrieving file contents, this still materializes the full directory tree for the whole input RIA (all subjects/sessions). On large BIDS datasets this consumes too many inodes, and we repeatedly hit inode limits on shared filesystems.
Additional details
Next steps
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request