Skip to content

Commit f85dd2f

Browse files
committed
Initial commit
0 parents  commit f85dd2f

File tree

4 files changed

+175
-0
lines changed

4 files changed

+175
-0
lines changed

README.md

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
## Cloud sync ignore files
2+
3+
> Make Cloud sync ignore certain project files
4+
5+
### Why?
6+
7+
I am using iCloud to share data between my macs, including my project files. I noticed that `node_modules` and similar project dependencies caused iCloud to take awfully long to sync thousands upon thousands of files, despite fast internet connection.
8+
9+
I was searching for a solution to prevent `node_modules` from uploading, but iCloud doesn't seem to offer ignoring files. So I started to put together this solution/hack. It's not iCloud specific so it can be used with any service which doesn't provide selective sync and/or ignoring certain files.
10+
11+
### How it works?
12+
13+
It's really dumb solution and I tried my best to avoid this solution, but it was the only thing that worked.
14+
15+
The script expects a local project directory, which is not included in cloud synced folders. This directory is kept in sync with duplicate project folder inside the cloud drive, but this clone doesn't contain project dependencies. This makes the size of clone a bit smaller and most importantly, faster to sync, thanks to a lot less files.
16+
17+
18+
Syncing works both ways, so when new files are downloaded from the cloud, they will be copied over to local project directory.
19+
20+
The syncing is done with [unison](https://www.cis.upenn.edu/~bcpierce/unison/index.html) CLI tool. Initially, I tried with rsync, but it doesn't support bidirectional sync.
21+
22+
The sync script will ignore:
23+
- `node_modules` folder
24+
- `bower_components` folder
25+
- `*.log` files (rails apps can produce large logs and those are stored directly inside the project directory)
26+
- `.DS_Store` files
27+
28+
These files can be customized by modifying `ignore_files` variable in `install.sh`. With current version, this is based off personal project types (mostly rails and node), but it's open to suggestions.
29+
30+
### Installing
31+
32+
1. Install [unison](https://www.cis.upenn.edu/~bcpierce/unison/download.html) CLI tool. The easiest way is `brew install unison`.
33+
2. Clone or download this repository and `cd` into the folder.
34+
3. Check `install.sh` script and edit paths to match your system setup. Check variables `local_path`, `cloud_path` and `ignore_files`, which can be found at the top of the script.
35+
4. Run `./install.sh`.
36+
37+
By default, the script is configured to add log files in `/var/log`. To create log files, the requires `sudo` access, so it will ask for password during installation. You can disable logging by running install script as `./install.sh --no-logs`.
38+
39+
### Updating config
40+
41+
If you make changes to `install.sh` or templates and want to update your config, just run `./install.sh` again and it will re-generate and reload config. The updated config should start working immediately. But it might take `unison` a moment to sync stuff, depending on your project size
42+
43+
### Uninstalling
44+
45+
Running `./install.sh --uninstall` will unload config, remove it, remove the syncing script and log files.
46+
47+
48+
### Performance
49+
50+
As far as software goes, all comes down to `unison` performance, which seems to be quite fast. On my MacBook Pro (13" retina, late 2013: 2.4 GHz CPU, 8gb RAM, 256gb SSD) with projects folder size of 2.5GB with ~125k files or 62k without counting `node_modules`, it took about 75s to init, i.e. clone the whole project folder. Real cases will probably see only a few files changed here and there, which should be synced instantaneously.
51+
52+
53+
### Credits
54+
55+
Thanks you @tatums for the [rsyc-icloud-hack](https://github.com/tatums/rsyc-icloud-hack) project, it helped me a lot to shape this project!

install.sh

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
#!/bin/bash
2+
#
3+
# v1.0.0
4+
#
5+
# Usage:
6+
# - Call script to register sync script with launchd.
7+
# - Call with `--no-logs` to disable logging.
8+
# - Call with `--uninstall` or `--remove` to unregister from launchd and clean up files.
9+
10+
# Adjust the paths to match your system (do not end the path with /).
11+
# Path to local (working) projects folder
12+
local_path="${HOME}/LocalDocs/Projects"
13+
14+
# Path to cloud projects folder (node_modules, etc. are omitted).
15+
#
16+
# Note: if you're using iCloud on a system before Sierra, the Documents folder
17+
# can be found at "${HOME}/Library/Mobile Documents/com~apple~CloudDocs"
18+
cloud_path="${HOME}/Documents/Projects"
19+
20+
# Comma-separated list of files to ignore.
21+
# Example: "node_modules,*.log" -> ignore all paths containing `node_modules` and any files ending with `*.log`.
22+
# For more details see: http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html#ignore
23+
ignore_files="node_modules,bower_components,*.log,.DS_Store"
24+
25+
# If you want, change log destination here (irellevant with --no-logs flag).
26+
log_file="/var/log/${label}.out.log"
27+
err_file="/var/log/${label}.err.log"
28+
29+
##########################################################################
30+
# No need to modify the code below, unless you know what you're doing :D #
31+
##########################################################################
32+
33+
# Path to script and launchd config.
34+
label="com.markogresak.projects.CloudSyncIgnore"
35+
script_path="/usr/local/bin/${label}.sh"
36+
plist_path="${HOME}/Library/LaunchAgents/${label}.plist"
37+
38+
# If config already exists, unload it before updating it.
39+
if [ -f $plist_path ]; then
40+
launchctl unload $plist_path
41+
fi
42+
43+
if [[ "$1" == "--uninstall" || "$1" == "--remove" ]]; then
44+
rm -f $script_path $plist_path
45+
if [ -f $log_file ] || [ -f $err_file ]; then
46+
echo "The script will attempt to remove log files. This requires sudo access, so the shell will ask you for password."
47+
sudo rm -f $log_file $err_file
48+
fi
49+
echo "Sync script successfully removed. Thanks for giving it a chance. If you have any suggestions for improvement, please let me know by submitting an issue."
50+
exit
51+
fi
52+
53+
# Check for unison command and fail if not found.
54+
if ! command -v unison >/dev/null 2>&1; then
55+
echo "Command 'unison' not found. Install it (brew install unison) and try this script again."
56+
exit 1
57+
fi
58+
59+
# If `--no-logs` flag is used, use /dev/null as stdout and stderr.
60+
if [[ "$1" == "--no-logs" ]]; then
61+
log_file="/dev/null"
62+
err_file="/dev/null"
63+
else
64+
echo "The script will attempt to create log files. This requires sudo access, so the shell will ask you for password."
65+
66+
# Create/clear log files (requires sudo to allow modifying files in /var/log) and fix log file permissions.
67+
sudo sh -c 'echo "" > $0' "$log_file"
68+
sudo sh -c 'echo "" > $0' "$err_file"
69+
sudo chown `whoami` "$log_file" "$err_file"
70+
echo -e "Log files were successfully created.\n"
71+
fi
72+
73+
# Create actual files based of .template files.
74+
sed "s|{{LOCAL_PATH}}|${local_path}|;
75+
s|{{CLOUD_PATH}}|${cloud_path}|;
76+
s|{{SCRIPT_PATH}}|${script_path}|;
77+
s|{{LABEL}}|${label}|;
78+
s|{{LOG_FILE}}|${log_file}|;
79+
s|{{ERR_FILE}}|${err_file}|" plist.template > $plist_path
80+
sed "s|{{UNISON_PATH}}|$(which unison)|;
81+
s|{{IGNORE_FILES}}|${ignore_files}|;
82+
s|{{LOCAL_PATH}}|${local_path}|;
83+
s|{{CLOUD_PATH}}|${cloud_path}|;" script.template > $script_path
84+
85+
# Load launchd config.
86+
launchctl load $plist_path
87+
88+
echo "Sync script added. It will be triggered any time any of files inside local or iCloud project folder changes."
89+
echo "I hope this script will help make your life a little easier :)"

plist.template

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
3+
<plist version="1.0">
4+
<dict>
5+
<key>Label</key>
6+
<string>{{LABEL}}</string>
7+
<key>ProgramArguments</key>
8+
<array>
9+
<string>/bin/bash</string>
10+
<string>{{SCRIPT_PATH}}</string>
11+
</array>
12+
<key>WatchPaths</key>
13+
<array>
14+
<string>{{LOCAL_PATH}}</string>
15+
<string>{{CLOUD_PATH}}</string>
16+
</array>
17+
<key>RunAtLoad</key>
18+
<true/>
19+
<key>StandardOutPath</key>
20+
<string>{{LOG_FILE}}</string>
21+
<key>StandardErrorPath</key>
22+
<string>{{ERR_FILE}}</string>
23+
<key>LowPriorityIO</key>
24+
<true/>
25+
<key>Nice</key>
26+
<integer>1</integer>
27+
</dict>
28+
</plist>

script.template

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/bin/bash
2+
3+
{{UNISON_PATH}} -batch -ignore="Name {{{IGNORE_FILES}}}" "{{LOCAL_PATH}}" "{{CLOUD_PATH}}"

0 commit comments

Comments
 (0)