Skip to content

Commit 16201da

Browse files
jianghao0718jianghao0718
authored
Merge pull request #164 from github/hao/sync-private-to-public
sync backup-utils-private to public repo backup-utils
2 parents 39c4d28 + adfdc07 commit 16201da

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+621
-1006
lines changed

‎.github/workflows/main.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,16 +13,16 @@ jobs:
1313
- name: Install Dependencies (Linux)
1414
run: |
1515
sudo apt-get update -y
16-
sudo apt-get install -y devscripts debhelper moreutils fakeroot jq
16+
sudo apt-get install -y devscripts debhelper moreutils fakeroot jq pigz
1717
wget "https://storage.googleapis.com/shellcheck/shellcheck-v0.7.0.linux.x86_64.tar.xz"
1818
tar --xz -xvf "shellcheck-v0.7.0.linux.x86_64.tar.xz"
1919
sudo cp shellcheck-v0.7.0/shellcheck /usr/bin/shellcheck
2020
if: matrix.os == 'ubuntu-latest'
2121
- name: Install Dependencies (macOS)
2222
run: |
23-
brew install gnu-tar shellcheck jq
23+
brew install gnu-tar shellcheck jq pigz coreutils
2424
brew unlink parallel
25-
brew install moreutils
25+
brew install moreutils gawk
2626
if: matrix.os == 'macos-latest'
2727
- name: Get Sources
2828
uses: actions/checkout@v2

‎.travis.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,12 @@ matrix:
55
- os: osx
66
before_install:
77
- brew update
8+
- brew install gawk
89
- brew install gnu-tar
910
- brew install moreutils
1011
- brew install shellcheck
1112
- brew install jq
13+
- brew install pigz
1214
script: make test
1315
- os: linux
1416
dist: trusty
@@ -24,4 +26,6 @@ matrix:
2426
- moreutils
2527
- fakeroot
2628
- jq
29+
- coreutils
30+
- pigz
2731
script: debuild -uc -us

‎Dockerfile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ RUN apt-get -q -y update && \
77
ca-certificates \
88
ssh \
99
git \
10+
moreutils \
11+
gawk \
1012
&& rm -rf /var/lib/apt/lists/*
1113

1214
WORKDIR /backup-utils

‎README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@
33
This repository includes backup and recovery utilities for
44
[GitHub Enterprise Server][1].
55

6+
**UPDATE**: The new parallel backup and restore beta feature will require [GNU awk](https://www.gnu.org/software/gawk) and [moreutils](https://joeyh.name/code/moreutils) to be installed.
7+
68
**Note**: the [GitHub Enterprise Server version requirements][2] have
79
changed starting with Backup Utilities v2.13.0, released on 27 March 2018.
810

‎RELEASING.md

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,10 @@ Coming :soon:
2424

2525
## Automatic Process from CLI
2626

27-
1. Install the Debian `devscripts` package:
28-
`sudo apt-get install devscripts`
29-
2. Run...
27+
1. Install the Debian `devscripts` and `moreutils` packages:
28+
`sudo apt-get install devscripts moreutils`
29+
2. Generate a PAT through github.com with access to the `github/backup-utils` repository. This will be used for the `GH_RELEASE_TOKEN` environment variable in the next step.
30+
3. Run...
3031
- Feature release:
3132
`GH_AUTHOR="Bob Smith <[email protected]>" GH_RELEASE_TOKEN=your-amazing-secure-token script/release 2.13.0 2.11.0`
3233
- Patch release:
@@ -36,8 +37,8 @@ Coming :soon:
3637

3738
In the event you can't perform the automatic process, or a problem is encountered with the automatic process, these are the manual steps you need to perform for a release.
3839

39-
1. Install the Debian `devscripts` package:
40-
`sudo apt-get install devscripts`
40+
1. Install the Debian `devscripts` and `moreutils` packages:
41+
`sudo apt-get install devscripts moreutils`
4142
2. Add a new version and release notes to the `debian/changelog` file:
4243
`dch --newversion 2.13.0 --release-heuristic log`
4344
You can use `make pending-prs` to craft the release notes.

‎backup.config-example

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,10 @@ GHE_NUM_SNAPSHOTS=10
2323
#
2424
#GHE_RESTORE_HOST="github-standby.example.com"
2525

26+
# If set to 'yes', ghe-restore will omit the restore of audit logs.
27+
#
28+
#GHE_RESTORE_SKIP_AUDIT_LOGS=no
29+
2630
# When verbose output is enabled with `-v`, it's written to stdout by default. If
2731
# you'd prefer it to be written to a separate file, set this option.
2832
#
@@ -48,3 +52,28 @@ GHE_NUM_SNAPSHOTS=10
4852
#
4953
# WARNING: do not enable this, only useful for debugging/development
5054
#GHE_BACKUP_FSCK=no
55+
56+
# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'.
57+
#
58+
# WARNING: this feature is in beta.
59+
#GHE_PARALLEL_ENABLED=yes
60+
61+
# Sets the maximum number of jobs to run in parallel. Defaults to the number
62+
# of available processing units on the machine.
63+
#
64+
# WARNING: this feature is in beta.
65+
#GHE_PARALLEL_MAX_JOBS=2
66+
67+
# Sets the maximum number of rsync jobs to run in parallel. Defaults to the
68+
# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing
69+
# units on the machine.
70+
#
71+
# WARNING: this feature is in beta.
72+
# GHE_PARALLEL_RSYNC_MAX_JOBS=3
73+
74+
# When jobs are running in parallel wait as needed to avoid starting new jobs
75+
# when the system's load average is not below the specified percentage. Defaults to
76+
# unrestricted.
77+
#
78+
# WARNING: this feature is in beta.
79+
#GHE_PARALLEL_MAX_LOAD=50

‎bin/ghe-backup

Lines changed: 47 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,12 @@ done
4141
# shellcheck source=share/github-backup-utils/ghe-backup-config
4242
. "$( dirname "${BASH_SOURCE[0]}" )/../share/github-backup-utils/ghe-backup-config"
4343

44+
# Check to make sure moreutils parallel is installed and working properly
45+
ghe_parallel_check
46+
4447
# Used to record failed backup steps
4548
failures=
49+
failures_file="$(mktemp -t backup-utils-backup-failures-XXXXXX)"
4650

4751
# CPU and IO throttling to keep backups from thrashing around.
4852
export GHE_NICE=${GHE_NICE:-"nice -n 19"}
@@ -81,25 +85,6 @@ if [ "$(ls -il dest1/testfile | awk '{ print $1 }')" != "$(ls -il dest2/testfile
8185
fi
8286
rm -rf src dest1 dest2
8387

84-
# if we should use gitbackups to backup repositories
85-
should_use_gitbackups_for_repositories(){
86-
ghe-ssh "$GHE_HOSTNAME" ghe-config --true "app.github.gitbackups"
87-
}
88-
89-
# check that the appliance supports using gitbackups for repositories
90-
can_use_gitbackups_for_repositories(){
91-
ghe-ssh "$GHE_HOSTNAME" test -e /data/github/current/bin/backup-repositories
92-
}
93-
94-
# Exit early if the appliance is missing script to backup repositories using gitbackups
95-
if should_use_gitbackups_for_repositories; then
96-
if ! can_use_gitbackups_for_repositories; then
97-
echo "Error: Configuration setting 'app.github.gitbackups' is enabled but this version of GHES cannot use gitbackups to back up repositories via 'ghe-backup'."
98-
echo "Disable configuration setting 'app.github.gitbackups' and re-run 'ghe-backup' to use rsync."
99-
exit 1
100-
fi
101-
fi
102-
10388
# To prevent multiple backup runs happening at the same time, we create a
10489
# in-progress file with the timestamp and pid of the backup process,
10590
# giving us a form of locking.
@@ -119,6 +104,8 @@ cleanup () {
119104
fi
120105
fi
121106

107+
rm -rf "$failures_file"
108+
122109
# Cleanup SSH multiplexing
123110
ghe-ssh --clean
124111
}
@@ -193,49 +180,57 @@ ghe-ssh "$GHE_HOSTNAME" -- 'ghe-export-ssh-host-keys' > ssh-host-keys.tar ||
193180
failures="$failures ssh-host-keys"
194181
bm_end "ghe-export-ssh-host-keys"
195182

196-
# if we are going to take a binary backup
197-
is_binary_backup(){
198-
ghe-ssh "$GHE_HOSTNAME" ghe-config --true "mysql.backup.binary"
199-
}
200-
201-
echo "Backing up MySQL database ..."
202-
bm_start "ghe-export-mysql"
203-
echo 'set -o pipefail; ghe-export-mysql | gzip' |
204-
ghe-ssh "$GHE_HOSTNAME" -- /bin/bash > mysql.sql.gz || failures="$failures mysql"
205-
if is_binary_backup; then
206-
touch mysql-binary-backup-sentinel
183+
if is_binary_backup_feature_on; then
184+
echo "Backing up MySQL database using binary backup strategy ..."
185+
else
186+
echo "Backing up MySQL database using logical backup strategy ..."
207187
fi
208-
bm_end "ghe-export-mysql"
188+
ghe-backup-mysql || failures="$failures mysql"
209189

210-
echo "Backing up Redis database ..."
211-
ghe-backup-redis > redis.rdb || failures="$failures redis"
190+
commands=("
191+
echo \"Backing up Redis database ...\"
192+
ghe-backup-redis > redis.rdb || printf %s \"redis \" >> \"$failures_file\"")
212193

213-
echo "Backing up audit log ..."
214-
ghe-backup-audit-log || failures="$failures audit-log"
194+
commands+=("
195+
echo \"Backing up audit log ...\"
196+
ghe-backup-es-audit-log || printf %s \"audit-log \" >> \"$failures_file\"")
215197

216-
echo "Backing up hookshot logs ..."
217-
ghe-backup-es-hookshot || failures="$failures hookshot"
198+
commands+=("
199+
echo \"Backing up hookshot logs ...\"
200+
ghe-backup-es-hookshot || printf %s \"hookshot \" >> \"$failures_file\"")
218201

219-
if should_use_gitbackups_for_repositories && can_use_gitbackups_for_repositories; then
220-
echo "Backing up Git repositories using gitbackups ..."
221-
ghe-backup-repositories-gitbackups || failures="$failures repositories"
222-
else
223-
echo "Backing up Git repositories using rsync ..."
224-
ghe-backup-repositories-rsync || failures="$failures repositories"
225-
fi
202+
commands+=("
203+
echo \"Backing up Git repositories ...\"
204+
ghe-backup-repositories || printf %s \"repositories \" >> \"$failures_file\"")
226205

227-
echo "Backing up GitHub Pages ..."
228-
ghe-backup-pages || failures="$failures pages"
206+
commands+=("
207+
echo \"Backing up GitHub Pages artifacts ...\"
208+
ghe-backup-pages || printf %s \"pages \" >> \"$failures_file\"")
229209

230-
echo "Backing up storage data ..."
231-
ghe-backup-storage || failures="$failures storage"
210+
commands+=("
211+
echo \"Backing up storage data ...\"
212+
ghe-backup-storage || printf %s \"storage \" >> \"$failures_file\"")
232213

233-
echo "Backing up custom Git hooks ..."
234-
ghe-backup-git-hooks || failures="$failures git-hooks"
214+
commands+=("
215+
echo \"Backing up custom Git hooks ...\"
216+
ghe-backup-git-hooks || printf %s \"git-hooks \" >> \"$failures_file\"")
235217

236218
if [ "$GHE_BACKUP_STRATEGY" = "rsync" ]; then
237-
echo "Backing up Elasticsearch indices ..."
238-
ghe-backup-es-rsync || failures="$failures elasticsearch"
219+
commands+=("
220+
echo \"Backing up Elasticsearch indices ...\"
221+
ghe-backup-es-rsync || printf %s \"elasticsearch \" >> \"$failures_file\"")
222+
fi
223+
224+
if [ "$GHE_PARALLEL_ENABLED" = "yes" ]; then
225+
$GHE_PARALLEL_COMMAND $GHE_PARALLEL_COMMAND_OPTIONS -- "${commands[@]}"
226+
else
227+
for c in "${commands[@]}"; do
228+
eval "$c"
229+
done
230+
fi
231+
232+
if [ -s "$failures_file" ]; then
233+
failures="$failures $(cat "$failures_file")"
239234
fi
240235

241236
# git fsck repositories after the backup

‎bin/ghe-host-check

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ fi
113113

114114
# backup-utils 2.13 onwards limits support to the current and previous two releases
115115
# of GitHub Enterprise Server.
116-
supported_minimum_version="2.17.0"
116+
supported_minimum_version="2.18.0"
117117

118118
if [ "$(version $version)" -ge "$(version $supported_minimum_version)" ]; then
119119
supported=1

‎bin/ghe-restore

Lines changed: 57 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,9 @@ cleanup () {
8585
# shellcheck source=share/github-backup-utils/ghe-backup-config
8686
. "$( dirname "${BASH_SOURCE[0]}" )/../share/github-backup-utils/ghe-backup-config"
8787

88+
# Check to make sure moreutils parallel is installed and working properly
89+
ghe_parallel_check
90+
8891
# Grab the host arg
8992
GHE_HOSTNAME="${GHE_RESTORE_HOST_OPT:-$GHE_RESTORE_HOST}"
9093

@@ -252,45 +255,53 @@ if [ -s "$GHE_RESTORE_SNAPSHOT_PATH/uuid" ] && ! $CLUSTER; then
252255
ghe-ssh "$GHE_HOSTNAME" -- "sudo rm -rf /data/user/consul/raft"
253256
fi
254257

255-
echo "Restoring MySQL database ..."
256-
ghe-restore-mysql "$GHE_HOSTNAME" 1>&3
258+
if is_binary_backup_feature_on; then
259+
appliance_strategy="binary"
260+
else
261+
appliance_strategy="logical"
262+
fi
257263

258-
echo "Restoring Redis database ..."
259-
bm_start "ghe-import-redis"
260-
ghe-ssh "$GHE_HOSTNAME" -- 'ghe-import-redis' < "$GHE_RESTORE_SNAPSHOT_PATH/redis.rdb" 1>&3
261-
bm_end "ghe-import-redis"
264+
if is_binary_backup "$GHE_DATA_DIR/$GHE_RESTORE_SNAPSHOT"; then
265+
backup_snapshot_strategy="binary"
266+
else
267+
backup_snapshot_strategy="logical"
268+
fi
262269

263-
# Unified and enhanced restore method to 2.13.0 and newer
264-
if $CLUSTER || [ "$(version $GHE_REMOTE_VERSION)" -ge "$(version 2.13.0)" ]; then
265-
echo "Restoring Git repositories ..."
266-
ghe-restore-repositories "$GHE_HOSTNAME"
270+
echo "Restoring MySQL database from ${backup_snapshot_strategy} backup snapshot on an appliance configured for ${appliance_strategy} backups ..."
271+
ghe-restore-mysql "$GHE_HOSTNAME" 1>&3
267272

268-
echo "Restoring Gists ..."
269-
ghe-restore-repositories-gist "$GHE_HOSTNAME"
273+
commands=("
274+
echo \"Restoring Redis database ...\"
275+
ghe-ssh \"$GHE_HOSTNAME\" -- 'ghe-import-redis' < \"$GHE_RESTORE_SNAPSHOT_PATH/redis.rdb\" 1>&3")
270276

271-
echo "Restoring GitHub Pages ..."
272-
ghe-restore-pages "$GHE_HOSTNAME" 1>&3
273-
else
274-
echo "Restoring Git repositories and Gists ..."
275-
ghe-restore-repositories-rsync "$GHE_HOSTNAME" 1>&3
277+
commands+=("
278+
echo \"Restoring Git repositories ...\"
279+
ghe-restore-repositories \"$GHE_HOSTNAME\"")
276280

277-
echo "Restoring GitHub Pages ..."
278-
ghe-restore-pages-rsync "$GHE_HOSTNAME" 1>&3
279-
fi
281+
commands+=("
282+
echo \"Restoring Gists ...\"
283+
ghe-restore-repositories-gist \"$GHE_HOSTNAME\"")
280284

285+
commands+=("
286+
echo \"Restoring GitHub Pages artifacts ...\"
287+
ghe-restore-pages \"$GHE_HOSTNAME\" 1>&3")
281288

282-
echo "Restoring SSH authorized keys ..."
283-
ghe-ssh "$GHE_HOSTNAME" -- 'ghe-import-authorized-keys' < "$GHE_RESTORE_SNAPSHOT_PATH/authorized-keys.json" 1>&3
289+
commands+=("
290+
echo \"Restoring SSH authorized keys ...\"
291+
ghe-ssh \"$GHE_HOSTNAME\" -- 'ghe-import-authorized-keys' < \"$GHE_RESTORE_SNAPSHOT_PATH/authorized-keys.json\" 1>&3")
284292

285-
echo "Restoring storage data ..."
286-
ghe-restore-storage "$GHE_HOSTNAME" 1>&3
293+
commands+=("
294+
echo \"Restoring storage data ...\"
295+
ghe-restore-storage \"$GHE_HOSTNAME\" 1>&3")
287296

288-
echo "Restoring custom Git hooks ..."
289-
ghe-restore-git-hooks "$GHE_HOSTNAME" 1>&3
297+
commands+=("
298+
echo \"Restoring custom Git hooks ...\"
299+
ghe-restore-git-hooks \"$GHE_HOSTNAME\" 1>&3")
290300

291301
if ! $CLUSTER && [ -d "$GHE_RESTORE_SNAPSHOT_PATH/elasticsearch" ]; then
292-
echo "Restoring Elasticsearch indices ..."
293-
ghe-restore-es-rsync "$GHE_HOSTNAME" 1>&3
302+
commands+=("
303+
echo \"Restoring Elasticsearch indices ...\"
304+
ghe-restore-es-rsync \"$GHE_HOSTNAME\" 1>&3")
294305
fi
295306

296307
# Restore the audit log migration sentinel file, if it exists in the snapshot
@@ -301,11 +312,25 @@ fi
301312
# Restore exported audit and hookshot logs to 2.12.9 and newer single nodes and
302313
# all releases of cluster
303314
if $CLUSTER || [ "$(version $GHE_REMOTE_VERSION)" -ge "$(version 2.12.9)" ]; then
304-
echo "Restoring Audit logs ..."
305-
ghe-restore-audit-log "$GHE_HOSTNAME" 1>&3
315+
if [[ "$GHE_RESTORE_SKIP_AUDIT_LOGS" = "yes" ]]; then
316+
echo "Skipping restore of audit logs."
317+
else
318+
commands+=("
319+
echo \"Restoring Audit logs ...\"
320+
ghe-restore-es-audit-log \"$GHE_HOSTNAME\" 1>&3")
321+
fi
306322

307-
echo "Restoring hookshot logs ..."
308-
ghe-restore-es-hookshot "$GHE_HOSTNAME" 1>&3
323+
commands+=("
324+
echo \"Restoring hookshot logs ...\"
325+
ghe-restore-es-hookshot \"$GHE_HOSTNAME\" 1>&3")
326+
fi
327+
328+
if [ "$GHE_PARALLEL_ENABLED" = "yes" ]; then
329+
$GHE_PARALLEL_COMMAND $GHE_PARALLEL_COMMAND_OPTIONS -- "${commands[@]}"
330+
else
331+
for c in "${commands[@]}"; do
332+
eval "$c"
333+
done
309334
fi
310335

311336
# Restart an already running memcached to reset the cache after restore

0 commit comments

Comments
 (0)