Merge pull request #43 from github/documentation

Beginning documentation

Author: Shlomi Noach

.github/CONTRIBUTING.md

@@ -0,0 +1,26 @@
+## Contributing
+
+Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great.
+
+This project adheres to the [Open Code of Conduct](http://todogroup.org/opencodeofconduct/#gh-ost/[email protected]). By participating, you are expected to uphold this code.
+
+## Submitting a pull request
+
+0. [Fork](https://github.com/github/gh-ost/fork) and clone the repository
+0. Create a new branch: `git checkout -b my-branch-name`
+0. Make your change, add tests, and make sure the tests still pass
+0. Push to your fork and [submit a pull request](https://github.com/github/gh-ost/compare)
+0. Pat your self on the back and wait for your pull request to be reviewed and merged.
+
+Here are a few things you can do that will increase the likelihood of your pull request being accepted:
+
+- Follow the [style guide](https://golang.org/doc/effective_go.html#formatting).
+- Write tests.
+- Keep your change as focused as possible. If there are multiple changes you would like to make that are not dependent upon each other, consider submitting them as separate pull requests.
+- Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
+
+## Resources
+
+- [Contributing to Open Source on GitHub](https://guides.github.com/activities/contributing-to-open-source/)
+- [Using Pull Requests](https://help.github.com/articles/using-pull-requests/)
+- [GitHub Help](https://help.github.com)

.github/ISSUE_TEMPLATE.md

@@ -1,2 +1,30 @@
 # gh-ost
-GitHub's Online Schema Change for MySQL
+
+#### GitHub's online schema migration for MySQL
+
+`gh-ost` allows for online schema migrations in MySQL which are:
+- Triggerless
+- Testable
+- Pausable
+- Operations-friendly
+
+## How?
+
+WORK IN PROGRESS
+
+Please meanwhile refer to the [docs](doc) for more information.
+
+## What's in a name?
+
+Originally this was named `gh-osc`: GitHub Online Schema Change, in the likes of [Facebook online schema change](https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932/) and [pt-online-schema-change](https://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html).
+
+But then a rare genetic mutation happened, and the `s` transformed into `t`. And that sent us down the path of trying to figure out a new acronym. Right now, `gh-ost` (pronounce: _Ghost_), stands for:
+- GitHub Online Schema Translator/Transformer/Transfigurator
+
+## Authors
+
+`gh-ost` is designed, authored, reviewed and tested by the database infrastructure team at GitHub:
+- [@jonahberquist](https://github.com/jonahberquist)
+- [@ggunson](https://github.com/ggunson)
+- [@tomkrouper](https://github.com/tomkrouper)
+- [@shlomi-noach](https://github.com/shlomi-noach)

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,16 @@
+# Command line flags
+
+A more in-depth discussion of various `gh-ost` command line flags: implementation, implication, use cases.
+
+##### exact-rowcount
+
+A `gh-ost` execution need to copy whatever rows you have in your existing table onto the ghost table. This can, and often be, a large number. Exactly what that number is?
+`gh-ost` initially estimates the number of rows in your table by issuing an `explain select * from your_table`. This will use statistics on your table and return with a rough estimate. How rough? It might go as low as half or as high as double the actual number of rows in your table. This is the same method as used in [`pt-online-schema-change`](https://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html).
+
+`gh-ost` also supports the `--exact-rowcount` flag. When this flag is given, two things happen:
+- An initial, authoritative `select count(*) from your_table`.
+  This query may take a long time to complete, but is performed before we begin the massive operations.
+- A continuous update to the estimate as we make progress applying events.
+  We heuristically update the number of rows based on the queries we process from the binlogs.
+
+While the ongoing estimated number of rows is still heuristic, it's almost exact, such that the reported  [ETA](understanding-output.md) or percentage progress is typically accurate to the second throughout a multiple-hour operation.

README.md

@@ -0,0 +1,24 @@
+# Migrating with Statement Based Replication
+
+Even though `gh-ost` relies on Row Based Replication (RBR), it does not mean you can't keep your Statement Based Replication (SBR).
+
+`gh-ost` is happy to, and actually prefers and suggests so, connect to a replica. On this replica, it is happy to:
+- issue the heavyweight `INFORMATION_SCHEMA` queries that make a table structure analysis
+- issue a `select count(*) from mydb.mytable`, should `--exact-rowcount` be provided
+- connect itself as a fake replica to get the binary log stream
+
+All of the above can be executed on the master, but we're more comfortable that they execute on a replica.
+
+Please note the third item: `gh-ost` connects as a fake replica and pulls the binary logs. This is how `gh-ost` finds the table's changelog: it looks up entries in the binary log.
+
+The magic is that your master can still produce SRB, but if you have a replica with `log-slave-updates`, you can also configure it to have `binlog_format='ROW'`. Such a replica accepts SBR statements from its master, and produces RBR statements onto its binary logs.
+
+`gh-ost` is happy to modify the `binlog_format` on the replica for you:
+- If you supply `--switch-to-rbr`, `gh-ost` will convert the binlog format for you, and restart replication to make sure this takes effect.
+- If your replica is an intermediate master, i.e. further serves as a master to other replicas, `gh-ost` will not convert the `binlog_format`.
+- At any case, `gh-ost` **will not** convert back to `STATEMENT` (SBR). This is because you may be running multiple migrations concurrently. Being able to run concurrent migrations is one of the design goals of this tool. It's your own responsibility to switch back to SBR once all pending migrations are complete.
+
+### Summary
+
+- If you're already using RBR, all is well for you
+- If not, convert one of your replicas to `binlog_format='ROW'`, or let `gh-ost` do this for you.

doc/command-line-flags.md

@@ -0,0 +1,14 @@
+# Swapping the tables
+
+The table-swap is the final major step of the migration: it's the moment where your original table is pushed aside, and the ghost table (the one we secretly altered and operated on throughout the process) takes its place.
+
+MySQL poses some limitations on how the table swap can take place. While it supports an atomic swap, it does not allow for a swap under controlled lock.
+
+The [facebook OSC](https://www.facebook.com/notes/mysql-at-facebook/online-schema-change-for-mysql/430801045932/) tool documents this nicely. Look for **"Cut-over phase"**.
+
+`gh-ost` supports various types of table-swap / cut-over options:
+
+- `--quick-and-bumpy-swap-tables` - this method is similar to the one taken by the facebook OSC. It's non-blocking but also non-atomic. The original table is first renames and pushed aside, then the ghost table is renamed to take its place. In between the two renames there's a brief period of time where your table just does not exist, and queries will fail.
+- Voluntary lock based solution (default at this time): as depicted in [Solving the Facebook-OSC non-atomic table swap problem](http://code.openark.org/blog/mysql/solving-the-facebook-osc-non-atomic-table-swap-problem), this solution uses voluntary MySQL locks, and makes for a blocking swap, where your queries do not fail, but block until operation is complete. This effect is desired. There is danger in this solution, since connection failure of the two sessions involved in creating the lock, would result in a premature swap of the tables, hence with potentially corrupted data.
+- We are working at this time on a blocking, safe, atomic solution, using wait conditions and via User Defined Functions which will need to be dynamically loaded onto your MySQL server.
+- With [`--test-on-replica`](testing-on-replica.md) there is no table swap.

doc/migrating-with-sbr.md

@@ -0,0 +1,59 @@
+# Testing on replica
+
+`gh-ost`'s design allows for trusted and reliable tests of the migration without compromising production data integrity.
+
+Test on replica if you:
+- Are unsure of `gh-ost`, have not gained confidence into its workings
+- Just want to experiment with a real migration without affecting production (maybe measure migration time?)
+- Wish to observe data change impact
+
+## What testing on replica means
+
+TL;DR `gh-ost` will make all changes on a replica and leave both original and ghost tables for you to compare.
+
+## Issuing a test drive
+
+Apply `--test-on-replica --host=<a.replica>`.
+- `gh-ost` would connect to the indicated server
+- Will verify this is indeed a replica and not a master
+- Will perform _everything_ on this replica. Other then checking who the master is, it will otherwise not touch it.
+  - All `INFORMATION_SCHEMA` and `SELECT` queries run on the replica
+  - Ghost table is created on the replica
+  - Rows are copied onto the ghost table on the replica
+  - Binlog events are read from the replica and applied to ghost table on the replica
+  - So... everything
+
+`gh-ost` will sync the ghost table with the original table.
+- When it is satisfied, it will issue a `STOP SLAVE IO_THREAD`, effectively stopping replication
+- Will finalize last few statements
+- Will terminate. No table swap takes place. No table is dropped.
+
+You are now left with the original table **and** the ghost table. They _should_ be identical.
+
+You now have the time to verify the tool works correctly. You may checksum the entire table data if you like.
+- e.g.
+  `mysql -e 'select * from mydb.mytable order by id' | md5sum`
+  `mysql -e 'select * from mydb._mytable_gst order by id' | md5sum`
+- or of course only select the shared columns before/after the migration
+- We use the trivial `engine=innodb` for `alter` when testing. This way the resulting ghost table is identical in structure to the original table (including indexes) and we expect data to be completely identical. We use `md5sum` on the entire dataset to confirm the test result.
+
+### Cleanup
+
+It's your job to:
+- Drop the ghost table (at your leisure, you should be aware that a `DROP` can be a lengthy operation)
+- Start replication back (via `START SLAVE`)
+
+### Examples
+
+Simple:
+```shell
+$ gh-osc --host=myhost.com --conf=/etc/gh-ost.cnf --database=test --table=sample_table --alter="engine=innodb" --chunk-size=2000 --max-load=Threads_connected=20 --initially-drop-ghost-table --initially-drop-old-table --test-on-replica --verbose --execute
+```
+
+Elaborate:
+```shell
+$ gh-osc --host=myhost.com --conf=/etc/gh-ost.cnf --database=test --table=sample_table --alter="engine=innodb" --chunk-size=2000 --max-load=Threads_connected=20 --switch-to-rbr --initially-drop-ghost-table --initially-drop-old-table --test-on-replica --postpone-swap-tables-flag-file=/tmp/ghost-postpone.flag --exact-rowcount --allow-nullable-unique-key --verbose --execute
+```
+- Count exact number of rows (makes ETA estimation very good). This goes at the expense of paying the time for issuing a `SELECT COUNT(*)` on your table. We use this lovingly.
+- Automatically switch to `RBR` if replica is configured as `SBR`. See also: [migrating with SBR](migrating-with-sbr.md)
+- allow iterating on a `UNIQUE KEY` that has `NULL`able columns (at your own risk)