RClone options tuning

[sanderegg]

Rclone currently uses the following parameters:

rclone  --transfers 5 --use-mmap --buffer-size 0M --use-json-log --stats 200ms --verbose sync mys3:project/node/workspace/ test/ --links

A few things here are questionable:

• --buffer-size 0M this one is something that should not be, the default is 16M and is almost never hit. it is a performance hit.
• use-mmap this one seems pretty useless and even maybe problematic
• --transfers 5 if we have many small files, this is very limiting. Nevertheless this is probably also directly linked to the amount of models

We have some examples from TN where the dataset has >300k files and total size >170GiB.

tested

rclone --transfers 32 with TN dataset brought the download time from 1 hour 30 to 15 minutes!

• need to check if this could be set for normal operation

[GitHK]

To change --transfers R_CLONE_OPTION_TRANSFERS=5 is already present on all deployments and can be used to change the number of parallel downloads.

[GitHK]

To change --buffer-size R_CLONE_OPTION_BUFFER_SIZE=0M is already present on all deployments.

[GitHK]

Summary for Event Horizon sprint

Done

• adjuster rclone defaults to allow for better performance ♻️ adjust rclone defaults ⚠️ #6885
• fine tuned existing defaults to achieve the above goal