Scripts to streamline the process of filtering and downloading large numbers of files for local preservation and storage.
- rclone_list.sh
- rclone_listsize.sh
- rclone_copy.sh
- unzip_directory.sh
- chd_clean.sh
Download the scripts and extract them in any directory. You can pass target directories as arguments when starting a script. They're shell scripts, so you should probably be using a Linux machine or WSL. You will also need rclone installed. In the terminal, navigate to the script directory.
Typical Workflow:
- Check the total number of files using rclone_list. Look at returned results for any additional filters that may be needed. Modify and retry if desired.
- Once filters have been refined, run rclone_listsize to see the total file size. Ensure enough storage is available for the desired transfer.
- Run rclone_copy to transfer all files to local storage.
- If .zip files are not functional, run unzip_directory to unzip all files in the target directory.
- Many disc-based games can utilize .chd files. Use chdman (external, not included) or similar tools to convert various file types and tracks to .chd format.
- Run chd_clean to remove all .bin, .cue, .iso files that have already been converted to .chd and have a matching .chd present in the directory.
- Happy Preserving!
Headless search with filters. Provides a quick list of files that match your search criteria with a total number. No file size information is provided.
- Usage: ./rclone_list.sh <http_url> [optional additional filters]
- Example: rclone_list.sh http://example.com "+ *Japan*" "- *Europe*"
- Defaults to filtering for only USA releases.
- Default filters can be easily edited by modifying the script file.
- Wrap filters in quotes.
- Adding too many additional filter conditions can slow or break things, so just be cautious.
- Add asterisks to search for files with the keyword anywhere in the file name. Be careful; filtering *re* for re-releases will skip all files that contain "re," such as pResident shmEvil.
- Open the script in Notepad, Word (if you're a maniac), Notepad++ (the right answer), or any IDE to edit the DEFAULT_FILTERS section and set new defaults.
Searches WITH headers enabled. This search will take longer, depending on the quantity of matches and file sizes, but will provide the total size of matches along with the number of files.
- Usage: ./rclone_listsize.sh <http_url> [optional additional filters]
- Example: rclone_listsize.sh http://example.com "+ *Japan*" "- *Europe*"
- Functions the same as the other list script but provides information on total file size.
- This script can take significantly longer depending on the number of matching files and their sizes. Expect minutes or more for directories over 100GB.
Copies files matching criteria to the specified local directory.
- Usage: ./rclone_copy.sh <http_url> <destination_directory> [optional additional filters]
- Example: rclone_copy.sh http://example.com /path/to/destination "+ *Japan*" "- *Europe*"
- This script will take time to process depending on the number and size of matching files, as well as your network connection. If the terminal is closed or the system goes to sleep, the download will stop.
Unzips all .zip files in the specified directory. Prompts the user before the script starts to delete .zip files after extraction or to retain them for future use.
- Usage: unzip_directory.sh <source_directory> <destination_directory>
- Example: unzip_directory.sh /path/to/source /path/to/destination (CAN be the same)
- Make sure you have sufficient storage if you choose to retain .zip files. Extraction will significantly increase storage requirements for larger files.
Removes all precursor files from a directory after conversion to .chd. Any (and only) .bin, .cue, and .iso files that have a matching .chd also present in the directory will be removed. Users will be prompted for confirmation for each file or can approve en masse.
- Usage: chd_clean.sh <target_directory>
- Example: chd_clean.sh /path/to/directory
- Option to approve or deny each file individually.
- Option to approve all available files after each file.
Scans directory for similar file titles, such as "Rev 1", "re-release", and files included in 2-in-1 packs. Prompts the user to decide which files to delete.
- Usage: directory_clean.sh <target_directory>
- Example: directory_clean.sh /path/to/directory
- Option to approve or deny each file individually.
I'm pretty new to all of this. GPT did the heavy lifting to get the scripts started. I refined and tested. If you have suggestions or run into issues, please let me know. There isn't really anything fancy here, just a way to expedite things and reduce repetitive typing.
Use the scripts with caution. Some filter combinations have unexpected results. I tried to add as many precautions as possible when permanently altering or removing files, but it's possible I missed something.