Merge branch 'master' into tests-updates

Author: Shlomi Noach

.github/PULL_REQUEST_TEMPLATE.md

@@ -11,7 +11,7 @@ Related issue: https://github.com/github/gh-ost/issues/0123456789
 
 ### Description
 
-This PR [briefly explain what is does]
+This PR [briefly explain what it does]
 
 > In case this PR introduced Go code changes:
 

.travis.yml

@@ -1,20 +1,31 @@
 # http://docs.travis-ci.com/user/languages/go/
 language: go
 
-go: 1.8
+go:
+  - "1.9"
+  - "1.10"
 
 os:
   - linux
 
 env:
 - MYSQL_USER=root
+- CURRENT_CI_ENV=travis
+
+addons:
+  apt:
+    packages:
+    - git
+    - numactl
+    - libaio1
 
 before_install:
   - mysql -e 'CREATE DATABASE IF NOT EXISTS test;'
 
 install: true
 
-script: script/cibuild
+script:
+  - script/cibuild
 
 notifications:
   email: false

README.md

@@ -94,7 +94,7 @@ Please see [Coding gh-ost](doc/coding-ghost.md) for a guide to getting started d
 
 [Download latest release here](https://github.com/github/gh-ost/releases/latest)
 
-`gh-ost` is a Go project; it is built with Go `1.8` (though `1.7` should work as well). To build on your own, use either:
+`gh-ost` is a Go project; it is built with Go `1.9` and above. To build on your own, use either:
 - [script/build](https://github.com/github/gh-ost/blob/master/script/build) - this is the same build script used by CI hence the authoritative; artifact is `./bin/gh-ost` binary.
 - [build.sh](https://github.com/github/gh-ost/blob/master/build.sh) for building `tar.gz` artifacts in `/tmp/gh-ost`
 
@@ -107,3 +107,5 @@ Generally speaking, `master` branch is stable, but only [releases](https://githu
 - [@ggunson](https://github.com/ggunson)
 - [@tomkrouper](https://github.com/tomkrouper)
 - [@shlomi-noach](https://github.com/shlomi-noach)
+- [@jessbreckenridge](https://github.com/jessbreckenridge)
+- [@gtowey](https://github.com/gtowey)

RELEASE_VERSION

@@ -1 +1 @@
-1.0.42
+1.0.47

build.sh

@@ -2,36 +2,72 @@
 #
 #
 
-RELEASE_VERSION=$(cat RELEASE_VERSION)
+RELEASE_VERSION=
+buildpath=
+
+function setuptree() {
+  b=$( mktemp -d $buildpath/gh-ostXXXXXX ) || return 1
+  mkdir -p $b/gh-ost
+  mkdir -p $b/gh-ost/usr/bin
+  echo $b
+}
 
 function build {
-    osname=$1
-    osshort=$2
-    GOOS=$3
-    GOARCH=$4
-
-    echo "Building ${osname} binary"
-    export GOOS
-    export GOARCH
-    go build -ldflags "$ldflags" -o $buildpath/$target go/cmd/gh-ost/main.go
-
-    if [ $? -ne 0 ]; then
-        echo "Build failed for ${osname}"
-        exit 1
-    fi
-
-    (cd $buildpath && tar cfz ./gh-ost-binary-${osshort}-${timestamp}.tar.gz $target)
+  osname=$1
+  osshort=$2
+  GOOS=$3
+  GOARCH=$4
+
+
+
+  if ! go version | egrep -q 'go(1[.]9|1[.]1[0-9])' ; then
+    echo "go version is too low. Must use 1.9 or above"
+    exit 1
+  fi
+
+  echo "Building ${osname} binary"
+  export GOOS
+  export GOARCH
+  go build -ldflags "$ldflags" -o $buildpath/$target go/cmd/gh-ost/main.go
+
+  if [ $? -ne 0 ]; then
+      echo "Build failed for ${osname}"
+      exit 1
+  fi
+
+  (cd $buildpath && tar cfz ./gh-ost-binary-${osshort}-${timestamp}.tar.gz $target)
+
+  if [ "$GOOS" == "linux" ] ; then
+    echo "Creating Distro full packages"
+    builddir=$(setuptree)
+    cp $buildpath/$target $builddir/gh-ost/usr/bin
+    cd $buildpath
+    fpm -v "${RELEASE_VERSION}" --epoch 1 -f -s dir -n gh-ost -m shlomi-noach --description "GitHub's Online Schema Migrations for MySQL " --url "https://github.com/github/gh-ost" --vendor "GitHub" --license "Apache 2.0" -C $builddir/gh-ost --prefix=/ -t rpm .
+    fpm -v "${RELEASE_VERSION}" --epoch 1 -f -s dir -n gh-ost -m shlomi-noach --description "GitHub's Online Schema Migrations for MySQL " --url "https://github.com/github/gh-ost" --vendor "GitHub" --license "Apache 2.0" -C $builddir/gh-ost --prefix=/ -t deb --deb-no-default-config-files .
+  fi
 }
 
-buildpath=/tmp/gh-ost
-target=gh-ost
-timestamp=$(date "+%Y%m%d%H%M%S")
-ldflags="-X main.AppVersion=${RELEASE_VERSION}"
-export GO15VENDOREXPERIMENT=1
+main() {
+  if [ -z "${RELEASE_VERSION}" ] ; then
+    RELEASE_VERSION=$(git describe --abbrev=0 --tags | tr -d 'v')
+  fi
+  if [ -z "${RELEASE_VERSION}" ] ; then
+    RELEASE_VERSION=$(cat RELEASE_VERSION)
+  fi
+
 
-mkdir -p ${buildpath}
-build macOS osx darwin amd64
-build GNU/Linux linux linux amd64
+  buildpath=/tmp/gh-ost-release
+  target=gh-ost
+  timestamp=$(date "+%Y%m%d%H%M%S")
+  ldflags="-X main.AppVersion=${RELEASE_VERSION}"
+
+  mkdir -p ${buildpath}
+  rm -rf ${buildpath:?}/*
+  build macOS osx darwin amd64
+  build GNU/Linux linux linux amd64
+
+  echo "Binaries found in:"
+  ls -1 $buildpath/gh-ost-binary*${timestamp}.tar.gz
+}
 
-echo "Binaries found in:"
-ls -1 $buildpath/gh-ost-binary*${timestamp}.tar.gz
+main "$@"

doc/command-line-flags.md

@@ -2,6 +2,10 @@
 
 A more in-depth discussion of various `gh-ost` command line flags: implementation, implication, use cases.
 
+### aliyun-rds
+
+Add this flag when executing on Aliyun RDS.
+
 ### allow-master-master
 
 See [`--assume-master-host`](#assume-master-host).
@@ -65,6 +69,10 @@ This is somewhat similar to a Nagios `n`-times test, where `n` in our case is al
 
 Optional. Default is `safe`. See more discussion in [`cut-over`](cut-over.md)
 
+### cut-over-lock-timeout-seconds
+
+Default `3`.  Max number of seconds to hold locks on tables while attempting to cut-over (retry attempted when lock exceeds timeout).
+
 ### discard-foreign-keys
 
 **Danger**: this flag will _silently_ discard any foreign keys existing on your table.
@@ -82,7 +90,7 @@ The `--dml-batch-size` flag controls the size of the batched write. Allowed valu
 
 Why is this behavior configurable? Different workloads have different characteristics. Some workloads have very large writes, such that aggregating even `50` writes into a transaction makes for a significant transaction size. On other workloads write rate is high such that one just can't allow for a hundred more syncs to disk per second. The default value of `10` is a modest compromise that should probably work very well for most workloads. Your mileage may vary.
 
-Noteworthy is that setting `--dml-batch-size` to higher value _does not_ mean `gh-ost` blocks or waits on writes. The batch size is an upper limit on transaction size, not a minimal one. If `gh-ost` doesn't have "enough" events in the pipe, it does not wait on the binary log, it just writes what it already has. This conveniently suggests that if write load is light enough for `gh-ost` to only see a few events in the binary log at a given time, then it is also light neough for `gh-ost` to apply a fraction of the batch size.
+Noteworthy is that setting `--dml-batch-size` to higher value _does not_ mean `gh-ost` blocks or waits on writes. The batch size is an upper limit on transaction size, not a minimal one. If `gh-ost` doesn't have "enough" events in the pipe, it does not wait on the binary log, it just writes what it already has. This conveniently suggests that if write load is light enough for `gh-ost` to only see a few events in the binary log at a given time, then it is also light enough for `gh-ost` to apply a fraction of the batch size.
 
 ### exact-rowcount
 
@@ -103,6 +111,14 @@ While the ongoing estimated number of rows is still heuristic, it's almost exact
 
 Without this parameter, migration is a _noop_: testing table creation and validity of migration, but not touching data.
 
+### force-table-names
+
+Table name prefix to be used on the temporary tables.
+
+### gcp
+
+Add this flag when executing on a 1st generation Google Cloud Platform (GCP).
+
 ### heartbeat-interval-millis
 
 Default 100. See [`subsecond-lag`](subsecond-lag.md) for details.
@@ -117,6 +133,10 @@ We think `gh-ost` should not take chances or make assumptions about the user's t
 
 See [`initially-drop-ghost-table`](#initially-drop-ghost-table)
 
+### initially-drop-socket-file
+
+Default False. Should `gh-ost` forcibly delete an existing socket file. Be careful: this might drop the socket file of a running migration!
+
 ### max-lag-millis
 
 On a replication topology, this is perhaps the most important migration throttling factor: the maximum lag allowed for migration to work. If lag exceeds this value, migration throttles.
@@ -151,7 +171,7 @@ See also: [`concurrent-migrations`](cheatsheet.md#concurrent-migrations) on the
 
 ### skip-foreign-key-checks
 
-By default `gh-ost` verifies no foreign keys exist on the migrated table. On servers with large number of tables this check can take a long time. If you're absolutely certain no foreign keys exist (table does not referenece other table nor is referenced by other tables) and wish to save the check time, provide with `--skip-foreign-key-checks`.
+By default `gh-ost` verifies no foreign keys exist on the migrated table. On servers with large number of tables this check can take a long time. If you're absolutely certain no foreign keys exist (table does not reference other table nor is referenced by other tables) and wish to save the check time, provide with `--skip-foreign-key-checks`.
 
 ### skip-renamed-columns
 
@@ -161,6 +181,10 @@ See [`approve-renamed-columns`](#approve-renamed-columns)
 
 Issue the migration on a replica; do not modify data on master. Useful for validating, testing and benchmarking. See [`testing-on-replica`](testing-on-replica.md)
 
+### test-on-replica-skip-replica-stop
+
+Default `False`. When `--test-on-replica` is enabled, do not issue commands stop replication (requires `--test-on-replica`).
+
 ### throttle-control-replicas
 
 Provide a command delimited list of replicas; `gh-ost` will throttle when any of the given replicas lag beyond [`--max-lag-millis`](#max-lag-millis). The list can be queried and updated dynamically via [interactive commands](interactive-commands.md)

doc/hooks.md

@@ -69,6 +69,7 @@ The following variables are available on all hooks:
 - `GH_OST_INSPECTED_HOST`
 - `GH_OST_EXECUTING_HOST`
 - `GH_OST_HOOKS_HINT` - copy of `--hooks-hint` value
+- `GH_OST_DRY_RUN` - whether or not the `gh-ost` run is a dry run
 
 The following variable are available on particular hooks:
 

doc/interactive-commands.md

@@ -43,7 +43,7 @@ Both interfaces may serve at the same time. Both respond to simple text command,
 
 ### Querying for data
 
-For commands that accept an argumetn as value, pass `?` (question mark) to _get_ current value rather than _set_ a new one.
+For commands that accept an argument as value, pass `?` (question mark) to _get_ current value rather than _set_ a new one.
 
 ### Examples
 

doc/questions.md

@@ -28,3 +28,9 @@ It is therefore unlikely that `gh-ost` will support this behavior.
 Yes. TL;DR if running all on same replica/master, make sure to provide `--replica-server-id`. [Read more](cheatsheet.md#concurrent-migrations)
 
 # Why
+
+### Why Is the "Connect to Replica" mode preferred? 
+
+To avoid placing extra load on the master. `gh-ost` connects as a replication client. Each additional replica adds some load to the master. 
+
+To monitor replication lag from a replica. This makes the replication lag throttle, `--max-lag-millis`, more representative of the lag experienced by other replicas following the master (perhaps N levels deep in a tree of replicas).

doc/rds.md

@@ -1,4 +1,4 @@
-`gh-ost` has been updated to work with Amazon RDS however due to GitHub not relying using AWS for databases, this documentation is community driven so if you find a bug please [open an issue][new_issue]!
+`gh-ost` has been updated to work with Amazon RDS however due to GitHub not using AWS for databases, this documentation is community driven so if you find a bug please [open an issue][new_issue]!
 
 # Amazon RDS