Merge branch 'master' into master

Author: oakeyc

.github/workflows/lint.yml

@@ -0,0 +1,18 @@
+name: Lint Code Base
+
+on:
+  pull_request:
+    branches-ignore:
+      - 'master'
+
+jobs:
+  build:
+    name: Lint Code Base
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Code
+        uses: actions/checkout@v2
+      - name: Lint Code Base
+        uses: docker://github/super-linter:v2.1.1
+        env:
+          VALIDATE_ALL_CODEBASE: false

.github/workflows/main.yml

@@ -0,0 +1,38 @@
+name: Test and build
+
+on: [pull_request]
+
+jobs:
+  build:
+    strategy:
+      matrix:
+        os: ['ubuntu-latest', 'macos-latest']
+        # os: ['ubuntu-20.04', 'ubuntu-18.04', 'ubuntu-16.04', 'macos-latest']
+      fail-fast: false
+    runs-on: ${{ matrix.os }}
+      
+    steps:
+    - name: Install Dependencies (Linux)
+      run: |
+        sudo apt-get update -y
+        sudo apt-get install -y devscripts debhelper moreutils fakeroot jq pigz
+        wget "https://github.com/koalaman/shellcheck/releases/download/v0.7.0/shellcheck-v0.7.0.linux.x86_64.tar.xz"
+        tar --xz -xvf "shellcheck-v0.7.0.linux.x86_64.tar.xz"
+        sudo cp shellcheck-v0.7.0/shellcheck /usr/bin/shellcheck
+      if: matrix.os == 'ubuntu-latest'
+    - name: Install Dependencies (macOS)
+      run: |
+        brew install gnu-tar shellcheck jq pigz coreutils gnu-sed gnu-getopt
+        brew unlink parallel
+        brew install moreutils gawk
+      if: matrix.os == 'macos-latest'
+    - name: Get Sources
+      uses: actions/checkout@v2
+    - name: Test
+      run: |
+        export PATH="$PATH:/snap/bin"
+        make test
+      shell: bash
+    - name: Build (Linux)
+      run: DEB_BUILD_OPTIONS=nocheck debuild -us -uc
+      if: matrix.os == 'ubuntu-latest'

.travis.yml

@@ -7,6 +7,8 @@ RUN apt-get -q -y update && \
     ca-certificates \
     ssh \
     git \
+    moreutils \
+    gawk \
     && rm -rf /var/lib/apt/lists/*
 
 WORKDIR /backup-utils

Dockerfile

@@ -3,6 +3,8 @@
 This repository includes backup and recovery utilities for
 [GitHub Enterprise Server][1].
 
+**UPDATE**: The new parallel backup and restore beta feature will require [GNU awk](https://www.gnu.org/software/gawk) and [moreutils](https://joeyh.name/code/moreutils) to be installed.
+
 **Note**: the [GitHub Enterprise Server version requirements][2] have
 changed starting with Backup Utilities v2.13.0, released on 27 March 2018.
 

README.md

@@ -8,32 +8,39 @@ There is no need to align Backup Utilities patch releases with GitHub Enterprise
 
 When making a `.0` release, you will need to specify the minimum supported version of GitHub Enterprise Server that that release supports.
 
-## Automatic Process from chatops (internal to GitHub only) - Coming :soon:
+Only repo administrator is allowed to run the release script, otherwise it will fail.
 
-### Feature release:
+## Pre-release Actions
 
-`.ghe backup-utils-release 2.13.0 2.11.0`
+Prior to making a release,
 
-### Patch release:
+1. Go through the list of open pull requests and merge any that are ready for merging.
+2. Go through the list of closed pull requests since the last release and ensure those that should be included in the release notes:
+  - have a "bug", "enhancement" or "feature" label,
+  - have a title that clearly describes the changes in that pull request. Reword if necessary.
+3. Perform a dry run (add `--dry-run` to one of the commands below) and verify the version strings are going to be changed and verify the release notes.
 
-`.ghe backup-utils-release 2.13.1`
+## Automatic Process from chatops (internal to GitHub only)
+
+Coming :soon:
 
 ## Automatic Process from CLI
 
-1. Install the Debian `devscripts` package:
-  `sudo apt-get install devscripts`
-2. Run...
-  - Feature release:  
+1. Install the Debian `devscripts` and `moreutils` packages:
+  `sudo apt-get install devscripts moreutils`
+2. Generate a PAT through github.com with access to the `github/backup-utils` repository. This will be used for the `GH_RELEASE_TOKEN` environment variable in the next step.
+3. Run...
+  - Feature release:
   `GH_AUTHOR="Bob Smith <[email protected]>" GH_RELEASE_TOKEN=your-amazing-secure-token script/release 2.13.0 2.11.0`
-  - Patch release:  
+  - Patch release:
   `GH_AUTHOR="Bob Smith <[email protected]>" GH_RELEASE_TOKEN=your-amazing-secure-token script/release 2.13.1`
 
 ## Manual Process
 
 In the event you can't perform the automatic process, or a problem is encountered with the automatic process, these are the manual steps you need to perform for a release.
 
-1. Install the Debian `devscripts` package:
-  `sudo apt-get install devscripts`
+1. Install the Debian `devscripts` and `moreutils` packages:
+  `sudo apt-get install devscripts moreutils`
 2. Add a new version and release notes to the `debian/changelog` file:
   `dch --newversion 2.13.0 --release-heuristic log`
   You can use `make pending-prs` to craft the release notes.
@@ -45,3 +52,13 @@ In the event you can't perform the automatic process, or a problem is encountere
 8. Draft a new release at https://github.com/github/backup-utils/releases, including the release notes and attaching the tarball and deb packages.
   The dist tarball you should upload has the revision in the file name, i.e. something like `github-backup-utils-v2.13.0.tar.gz`
 9. Push the head of the release to the 'stable' branch.
+
+## Post-release Actions
+
+Immediately after making a release using one of the methods above, verify the release has succeeded by checking:
+
+- latest release at https://github.com/github/backup-utils/releases is correct,
+- release at https://github.com/github/backup-utils/releases is linked to the vX.Y.Z tag,
+- release has the notes you expect to see,
+- asset download links for the latest release at https://github.com/github/backup-utils/releases all download the correct version of Backup Utilities,
+- the stable branch is inline with master - https://github.com/github/backup-utils/compare/stable...master.

RELEASING.md

@@ -23,6 +23,10 @@ GHE_NUM_SNAPSHOTS=10
 #
 #GHE_RESTORE_HOST="github-standby.example.com"
 
+# If set to 'yes', ghe-restore will omit the restore of audit logs.
+#
+#GHE_RESTORE_SKIP_AUDIT_LOGS=no
+
 # When verbose output is enabled with `-v`, it's written to stdout by default. If
 # you'd prefer it to be written to a separate file, set this option.
 #
@@ -48,3 +52,28 @@ GHE_NUM_SNAPSHOTS=10
 #
 # WARNING: do not enable this, only useful for debugging/development
 #GHE_BACKUP_FSCK=no
+
+# If set to 'yes', ghe-backup jobs will run in parallel. Defaults to 'no'.
+#
+# WARNING: this feature is in beta.
+#GHE_PARALLEL_ENABLED=yes
+
+# Sets the maximum number of jobs to run in parallel. Defaults to the number
+# of available processing units on the machine.
+#
+# WARNING: this feature is in beta.
+#GHE_PARALLEL_MAX_JOBS=2
+
+# Sets the maximum number of rsync jobs to run in parallel. Defaults to the
+# configured GHE_PARALLEL_MAX_JOBS, or the number of available processing 
+# units on the machine.
+#
+# WARNING: this feature is in beta.
+# GHE_PARALLEL_RSYNC_MAX_JOBS=3
+
+# When jobs are running in parallel wait as needed to avoid starting new jobs
+# when the system's load average is not below the specified percentage. Defaults to
+# unrestricted.
+#
+# WARNING: this feature is in beta.
+#GHE_PARALLEL_MAX_LOAD=50

backup.config-example

@@ -41,8 +41,12 @@ done
 # shellcheck source=share/github-backup-utils/ghe-backup-config
 . "$( dirname "${BASH_SOURCE[0]}" )/../share/github-backup-utils/ghe-backup-config"
 
+# Check to make sure moreutils parallel is installed and working properly
+ghe_parallel_check
+
 # Used to record failed backup steps
 failures=
+failures_file="$(mktemp -t backup-utils-backup-failures-XXXXXX)"
 
 # CPU and IO throttling to keep backups from thrashing around.
 export GHE_NICE=${GHE_NICE:-"nice -n 19"}
@@ -100,6 +104,8 @@ cleanup () {
     fi
   fi
 
+  rm -rf "$failures_file"
+
   # Cleanup SSH multiplexing
   ghe-ssh --clean
 }
@@ -174,36 +180,57 @@ ghe-ssh "$GHE_HOSTNAME" -- 'ghe-export-ssh-host-keys' > ssh-host-keys.tar ||
 failures="$failures ssh-host-keys"
 bm_end "ghe-export-ssh-host-keys"
 
-echo "Backing up MySQL database ..."
-bm_start "ghe-export-mysql"
-echo 'set -o pipefail; ghe-export-mysql | gzip' |
-ghe-ssh "$GHE_HOSTNAME" -- /bin/bash > mysql.sql.gz || failures="$failures mysql"
-bm_end "ghe-export-mysql"
+if is_binary_backup_feature_on; then
+  echo "Backing up MySQL database using binary backup strategy ..."
+else
+  echo "Backing up MySQL database using logical backup strategy ..."
+fi
+ghe-backup-mysql || failures="$failures mysql"
 
-echo "Backing up Redis database ..."
-ghe-backup-redis > redis.rdb || failures="$failures redis"
+commands=("
+echo \"Backing up Redis database ...\"
+ghe-backup-redis > redis.rdb || printf %s \"redis \" >> \"$failures_file\"")
 
-echo "Backing up audit log ..."
-ghe-backup-es-audit-log || failures="$failures audit-log"
+commands+=("
+echo \"Backing up audit log ...\"
+ghe-backup-es-audit-log || printf %s \"audit-log \" >> \"$failures_file\"")
 
-echo "Backing up hookshot logs ..."
-ghe-backup-es-hookshot || failures="$failures hookshot"
+commands+=("
+echo \"Backing up hookshot logs ...\"
+ghe-backup-es-hookshot || printf %s \"hookshot \" >> \"$failures_file\"")
 
-echo "Backing up Git repositories ..."
-ghe-backup-repositories || failures="$failures repositories"
+commands+=("
+echo \"Backing up Git repositories ...\"
+ghe-backup-repositories || printf %s \"repositories \" >> \"$failures_file\"")
 
-echo "Backing up GitHub Pages ..."
-ghe-backup-pages || failures="$failures pages"
+commands+=("
+echo \"Backing up GitHub Pages artifacts ...\"
+ghe-backup-pages || printf %s \"pages \" >> \"$failures_file\"")
 
-echo "Backing up storage data ..."
-ghe-backup-storage || failures="$failures storage"
+commands+=("
+echo \"Backing up storage data ...\"
+ghe-backup-storage || printf %s \"storage \" >> \"$failures_file\"")
 
-echo "Backing up custom Git hooks ..."
-ghe-backup-git-hooks || failures="$failures git-hooks"
+commands+=("
+echo \"Backing up custom Git hooks ...\"
+ghe-backup-git-hooks || printf %s \"git-hooks \" >> \"$failures_file\"")
 
 if [ "$GHE_BACKUP_STRATEGY" = "rsync" ]; then
-  echo "Backing up Elasticsearch indices ..."
-  ghe-backup-es-rsync || failures="$failures elasticsearch"
+  commands+=("
+  echo \"Backing up Elasticsearch indices ...\"
+  ghe-backup-es-rsync || printf %s \"elasticsearch \" >> \"$failures_file\"")
+fi
+
+if [ "$GHE_PARALLEL_ENABLED" = "yes" ]; then
+  $GHE_PARALLEL_COMMAND $GHE_PARALLEL_COMMAND_OPTIONS -- "${commands[@]}"
+else
+  for c in "${commands[@]}"; do
+    eval "$c"
+  done
+fi
+
+if [ -s "$failures_file" ]; then
+  failures="$failures $(cat "$failures_file")"
 fi
 
 # git fsck repositories after the backup

bin/ghe-backup

@@ -84,7 +84,7 @@ if [ $rc -ne 0 ]; then
   exit $rc
 fi
 
-version=$(echo "$output" | sed -E -n 's/GitHub Enterprise( Server)? version (.*)/\2/p')
+version=$(echo "$output" | grep "GitHub Enterprise" | awk '{print $NF}')
 
 if [ -z "$version" ]; then
   echo "Error: failed to parse version on '$host' or this isn't a GitHub appliance." 1>&2
@@ -106,14 +106,14 @@ if [ -z "$GHE_ALLOW_REPLICA_BACKUP" ]; then
   if [ "$(ghe-ssh $host -- cat $GHE_REMOTE_ROOT_DIR/etc/github/repl-state 2>/dev/null || true)" = "replica" ]; then
     echo "Error: high availability replica detected." 1>&2
     echo "Backup Utilities should be used to backup from the primary node in" 1>&2
-    echo "high availability environments to ensure consistent and repliable backups." 1>&2
+    echo "high availability environments to ensure consistent and reliable backups." 1>&2
     exit 1
   fi
 fi
 
 # backup-utils 2.13 onwards limits support to the current and previous two releases
 # of GitHub Enterprise Server.
-supported_minimum_version="2.14.0"
+supported_minimum_version="2.19.0"
 
 if [ "$(version $version)" -ge "$(version $supported_minimum_version)" ]; then
   supported=1