Skip to content

Commit 176aabe

Browse files
author
Shlomi Noach
authored
Merge pull request #190 from github/hooks
WIP: Hooks
2 parents f5fb984 + 7450910 commit 176aabe

20 files changed

+367
-9
lines changed

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ In addition, it offers many [operational perks](doc/perks.md) that make it safer
2929
- Dynamic control: you can [interactively](doc/interactive-commands.md) reconfigure `gh-ost`, even as migration still runs. You may forcibly initiate throttling.
3030
- Auditing: you may query `gh-ost` for status. `gh-ost` listens on unix socket or TCP.
3131
- Control over cut-over phase: `gh-ost` can be instructed to postpone what is probably the most critical step: the swap of tables, until such time that you're comfortably available. No need to worry about ETA being outside office hours.
32+
- External [hooks](doc/hooks.md) can couple `gh-ost` with your particular environment.
3233

3334
Please refer to the [docs](doc) for more information. No, really, read the [docs](doc).
3435

build.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
#
33
#
44

5-
RELEASE_VERSION="1.0.14"
5+
RELEASE_VERSION="1.0.15"
66

77
function build {
88
osname=$1

doc/hooks.md

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Hooks
2+
3+
`gh-ost` supports _hooks_: external processes which `gh-ost` executes at particular points of interest.
4+
5+
Use cases include:
6+
7+
- You wish to be notified by mail when a migration completes/fails
8+
- You wish to be notified when `gh-ost` postpones cut-over (at your demand), thus ready to complete (at your leisure)
9+
- RDS users who wish to `--test-on-replica`, but who cannot have `gh-ost` issue a `STOP SLAVE`, would use a hook to command RDS to stop replication
10+
- Send a status message to your chatops every hour
11+
- Perform cleanup on the _ghost_ table (drop/rename/nibble) once migration completes
12+
- etc.
13+
14+
`gh-ost` defines certain points of interest (event types), and executes hooks at those points.
15+
16+
Notes:
17+
18+
- You may have more than one hook per event type.
19+
- `gh-ost` will invoke relevant hooks _sequentially_ and _synchronously_
20+
- thus, you would generally like the hooks to execute as fast as possible, or otherwise issue tasks in the background
21+
- A hook returning with error code will propagate the error in `gh-ost`. Thus, you are able to force `gh-ost` to fail migration on your conditions.
22+
- Make sure to only return an error code when you do indeed wish to fail the rest of the migration
23+
24+
### Creating hooks
25+
26+
All hooks are expected to reside in a single directory. This directory is indicated by `--hooks-path`. When not provided, no hooks are executed.
27+
28+
`gh-ost` will dynamically search for hooks in said directory. You may add and remove hooks to/from this directory as `gh-ost` makes progress (though likely you don't want to). Hook files are expected to be executable processes.
29+
30+
In an effort to simplify code and to standardize usage, `gh-ost` expects hooks in explicit naming conventions. As an example, the `onStartup` hook expects processes named `gh-ost-on-startup*`. It will match and accept files named:
31+
32+
- `gh-ost-on-startup`
33+
- `gh-ost-on-startup--send-notification-mail`
34+
- `gh-ost-on-startup12345`
35+
- etc.
36+
37+
The full list of supported hooks is best found in code: [hooks.go](https://github.com/github/gh-ost/blob/master/go/logic/hooks.go). Documentation will always be a bit behind. At this time, though, the following are recognized:
38+
39+
- `gh-ost-on-startup`
40+
- `gh-ost-on-validated`
41+
- `gh-ost-on-rowcount-complete`
42+
- `gh-ost-on-before-row-copy`
43+
- `gh-ost-on-status`
44+
- `gh-ost-on-interactive-command`
45+
- `gh-ost-on-row-copy-complete`
46+
- `gh-ost-on-stop-replication`
47+
- `gh-ost-on-begin-postponed`
48+
- `gh-ost-on-before-cut-over`
49+
- `gh-ost-on-success`
50+
- `gh-ost-on-failure`
51+
52+
### Context
53+
54+
`gh-ost` will set environment variables per hook invocation. Hooks are then able to read those variables, indicating schema name, table name, `alter` statement, migrated host name etc. Some variables are available on all hooks, and some are available on relevant hooks.
55+
56+
The following variables are available on all hooks:
57+
58+
- `GH_OST_DATABASE_NAME`
59+
- `GH_OST_TABLE_NAME`
60+
- `GH_OST_GHOST_TABLE_NAME`
61+
- `GH_OST_OLD_TABLE_NAME`
62+
- `GH_OST_DDL`
63+
- `GH_OST_ELAPSED_SECONDS`
64+
- `GH_OST_MIGRATED_HOST`
65+
- `GH_OST_INSPECTED_HOST`
66+
- `GH_OST_EXECUTING_HOST`
67+
- `GH_OST_HOOKS_HINT`
68+
69+
The following variable are available on particular hooks:
70+
71+
- `GH_OST_COMMAND` is only available in `gh-ost-on-interactive-command`
72+
- `GH_OST_STATUS` is only available in `gh-ost-on-status`
73+
74+
### Examples
75+
76+
See [sample hooks](https://github.com/github/gh-ost/tree/master/resources/hooks-sample), as `bash` implementation samples.

go/base/context.go

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,8 @@ type MigrationContext struct {
7979
PostponeCutOverFlagFile string
8080
CutOverLockTimeoutSeconds int64
8181
PanicFlagFile string
82+
HooksPath string
83+
HooksHintMessage string
8284

8385
DropServeSocket bool
8486
ServeSocketFile string
@@ -93,6 +95,7 @@ type MigrationContext struct {
9395
InitiallyDropGhostTable bool
9496
CutOverType CutOver
9597

98+
Hostname string
9699
TableEngine string
97100
RowsEstimate int64
98101
RowsDeltaEstimate int64

go/cmd/gh-ost/main.go

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,9 @@ func main() {
9090
flag.StringVar(&migrationContext.ServeSocketFile, "serve-socket-file", "", "Unix socket file to serve on. Default: auto-determined and advertised upon startup")
9191
flag.Int64Var(&migrationContext.ServeTCPPort, "serve-tcp-port", 0, "TCP port to serve on. Default: disabled")
9292

93+
flag.StringVar(&migrationContext.HooksPath, "hooks-path", "", "directory where hook files are found (default: empty, ie. hooks disabled). Hook files found on this path, and conforming to hook naming conventions will be executed")
94+
flag.StringVar(&migrationContext.HooksHintMessage, "hooks-hint", "", "arbitrary message to be injected to hooks via GH_OST_HOOKS_HINT, for your convenience")
95+
9396
maxLoad := flag.String("max-load", "", "Comma delimited status-name=threshold. e.g: 'Threads_running=100,Threads_connected=500'. When status exceeds threshold, app throttles writes")
9497
criticalLoad := flag.String("critical-load", "", "Comma delimited status-name=threshold, same format as `--max-load`. When status exceeds threshold, app panics and quits")
9598
quiet := flag.Bool("quiet", false, "quiet")
@@ -198,6 +201,7 @@ func main() {
198201
migrator := logic.NewMigrator()
199202
err := migrator.Migrate()
200203
if err != nil {
204+
migrator.ExecOnFailureHook()
201205
log.Fatale(err)
202206
}
203207
log.Info("Done")

go/logic/hooks.go

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
/*
2+
/*
3+
Copyright 2016 GitHub Inc.
4+
See https://github.com/github/gh-ost/blob/master/LICENSE
5+
*/
6+
7+
package logic
8+
9+
import (
10+
"fmt"
11+
"os"
12+
"os/exec"
13+
"path/filepath"
14+
15+
"github.com/github/gh-ost/go/base"
16+
"github.com/openark/golib/log"
17+
)
18+
19+
const (
20+
onStartup = "gh-ost-on-startup"
21+
onValidated = "gh-ost-on-validated"
22+
onRowCountComplete = "gh-ost-on-rowcount-complete"
23+
onBeforeRowCopy = "gh-ost-on-before-row-copy"
24+
onRowCopyComplete = "gh-ost-on-row-copy-complete"
25+
onBeginPostponed = "gh-ost-on-begin-postponed"
26+
onBeforeCutOver = "gh-ost-on-before-cut-over"
27+
onInteractiveCommand = "gh-ost-on-interactive-command"
28+
onSuccess = "gh-ost-on-success"
29+
onFailure = "gh-ost-on-failure"
30+
onStatus = "gh-ost-on-status"
31+
onStopReplication = "gh-ost-on-stop-replication"
32+
)
33+
34+
type HooksExecutor struct {
35+
migrationContext *base.MigrationContext
36+
}
37+
38+
func NewHooksExecutor() *HooksExecutor {
39+
return &HooksExecutor{
40+
migrationContext: base.GetMigrationContext(),
41+
}
42+
}
43+
44+
func (this *HooksExecutor) initHooks() error {
45+
return nil
46+
}
47+
48+
func (this *HooksExecutor) applyEnvironmentVairables(extraVariables ...string) []string {
49+
env := os.Environ()
50+
env = append(env, fmt.Sprintf("GH_OST_DATABASE_NAME=%s", this.migrationContext.DatabaseName))
51+
env = append(env, fmt.Sprintf("GH_OST_TABLE_NAME=%s", this.migrationContext.OriginalTableName))
52+
env = append(env, fmt.Sprintf("GH_OST_GHOST_TABLE_NAME=%s", this.migrationContext.GetGhostTableName()))
53+
env = append(env, fmt.Sprintf("GH_OST_OLD_TABLE_NAME=%s", this.migrationContext.GetOldTableName()))
54+
env = append(env, fmt.Sprintf("GH_OST_DDL=%s", this.migrationContext.AlterStatement))
55+
env = append(env, fmt.Sprintf("GH_OST_ELAPSED_SECONDS=%f", this.migrationContext.ElapsedTime().Seconds()))
56+
env = append(env, fmt.Sprintf("GH_OST_MIGRATED_HOST=%s", this.migrationContext.ApplierConnectionConfig.ImpliedKey.Hostname))
57+
env = append(env, fmt.Sprintf("GH_OST_INSPECTED_HOST=%s", this.migrationContext.InspectorConnectionConfig.ImpliedKey.Hostname))
58+
env = append(env, fmt.Sprintf("GH_OST_EXECUTING_HOST=%s", this.migrationContext.Hostname))
59+
env = append(env, fmt.Sprintf("GH_OST_HOOKS_HINT=%s", this.migrationContext.HooksHintMessage))
60+
61+
for _, variable := range extraVariables {
62+
env = append(env, variable)
63+
}
64+
return env
65+
}
66+
67+
// executeHook executes a command, and sets relevant environment variables
68+
// combined output & error are printed to gh-ost's standard error.
69+
func (this *HooksExecutor) executeHook(hook string, extraVariables ...string) error {
70+
cmd := exec.Command(hook)
71+
cmd.Env = this.applyEnvironmentVairables(extraVariables...)
72+
73+
combinedOutput, err := cmd.CombinedOutput()
74+
fmt.Fprintln(os.Stderr, string(combinedOutput))
75+
return log.Errore(err)
76+
}
77+
78+
func (this *HooksExecutor) detectHooks(baseName string) (hooks []string, err error) {
79+
if this.migrationContext.HooksPath == "" {
80+
return hooks, err
81+
}
82+
pattern := fmt.Sprintf("%s/%s*", this.migrationContext.HooksPath, baseName)
83+
hooks, err = filepath.Glob(pattern)
84+
return hooks, err
85+
}
86+
87+
func (this *HooksExecutor) executeHooks(baseName string, extraVariables ...string) error {
88+
hooks, err := this.detectHooks(baseName)
89+
if err != nil {
90+
return err
91+
}
92+
for _, hook := range hooks {
93+
log.Infof("executing %+v hook: %+v", baseName, hook)
94+
if err := this.executeHook(hook, extraVariables...); err != nil {
95+
return err
96+
}
97+
}
98+
return nil
99+
}
100+
101+
func (this *HooksExecutor) onStartup() error {
102+
return this.executeHooks(onStartup)
103+
}
104+
105+
func (this *HooksExecutor) onValidated() error {
106+
return this.executeHooks(onValidated)
107+
}
108+
109+
func (this *HooksExecutor) onRowCountComplete() error {
110+
return this.executeHooks(onRowCountComplete)
111+
}
112+
func (this *HooksExecutor) onBeforeRowCopy() error {
113+
return this.executeHooks(onBeforeRowCopy)
114+
}
115+
116+
func (this *HooksExecutor) onRowCopyComplete() error {
117+
return this.executeHooks(onRowCopyComplete)
118+
}
119+
120+
func (this *HooksExecutor) onBeginPostponed() error {
121+
return this.executeHooks(onBeginPostponed)
122+
}
123+
124+
func (this *HooksExecutor) onBeforeCutOver() error {
125+
return this.executeHooks(onBeforeCutOver)
126+
}
127+
128+
func (this *HooksExecutor) onInteractiveCommand(command string) error {
129+
v := fmt.Sprintf("GH_OST_COMMAND='%s'", command)
130+
return this.executeHooks(onInteractiveCommand, v)
131+
}
132+
133+
func (this *HooksExecutor) onSuccess() error {
134+
return this.executeHooks(onSuccess)
135+
}
136+
137+
func (this *HooksExecutor) onFailure() error {
138+
return this.executeHooks(onFailure)
139+
}
140+
141+
func (this *HooksExecutor) onStatus(statusMessage string) error {
142+
v := fmt.Sprintf("GH_OST_STATUS='%s'", statusMessage)
143+
return this.executeHooks(onStatus, v)
144+
}
145+
146+
func (this *HooksExecutor) onStopReplication() error {
147+
return this.executeHooks(onStopReplication)
148+
}

0 commit comments

Comments
 (0)