Weird errors in packets.go (mysql-driver layer, don't think it actually causes issue in production)

[shuhaowu]

There are some really weird errors when running Ghostferry for an extended amount of time. Errors like the following shows up in the logs when I run Ghostferry moving data to move a large amount of data:

11786:[mysql] 2017/08/29 17:02:37 connection.go:67: invalid connection
12772:[mysql] 2017/08/29 17:15:40 packets.go:130: write tcp [REDACTED]->[REDACTED]: write: broken pipe
12773:[mysql] 2017/08/29 17:15:40 packets.go:130: write tcp [REDACTED]->[REDACTED]: write: broken pipe
12774:[mysql] 2017/08/29 17:15:40 connection.go:97: write tcp [REDACTED]->[REDACTED]: write: broken pipe
13756:[mysql] 2017/08/29 17:28:17 packets.go:66: unexpected EOF
13757:[mysql] 2017/08/29 17:28:17 packets.go:412: busy buffer
13760:[mysql] 2017/08/29 17:28:17 connection.go:67: invalid connection
15713:[mysql] 2017/08/29 17:55:37 packets.go:33: unexpected EOF
15718:[mysql] 2017/08/29 17:55:40 connection.go:67: invalid connection
16444:[mysql] 2017/08/29 18:12:27 packets.go:66: unexpected EOF
16445:[mysql] 2017/08/29 18:12:27 packets.go:412: busy buffer
16448:[mysql] 2017/08/29 18:12:27 connection.go:67: invalid connection
17208:[mysql] 2017/08/29 18:27:57 packets.go:33: unexpected EOF
17211:[mysql] 2017/08/29 18:27:57 connection.go:67: invalid connection
17295:[mysql] 2017/08/29 18:29:37 packets.go:66: unexpected EOF
17296:[mysql] 2017/08/29 18:29:37 packets.go:412: busy buffer
17299:[mysql] 2017/08/29 18:29:37 connection.go:67: invalid connection

No issues seem to arise from this as I think the underlying driver just reconnects, but I cannot be certain. Upstream has a lot of bugs complaining of similar behaviour. All the comments seems to be all over the place so I don't really know what to make of them (go-sql-driver/mysql#582, go-sql-driver/mysql#674, go-sql-driver/mysql#673 just for a few).

Also I just learned that go-sql-driver/mysql#302 merged about two weeks ago, which supposedly fixed an issue with potentially sending duplicate queries to MySQL. However, another bug seems to be present after: go-sql-driver/mysql#657

Also I noticed while tcpdumping that the errors are caused by connection reset by the mysql server, rather than the client (ghostferry). This maybe a red herring however as I only looked at a few cases and I don't have the logs from that anymore.

I haven't ran a detailed investigation into this or seen any case where this causes any issues during a real run.

[shuhaowu]

Likely not an issue as Ghostferry has ran with hundreds of TBs of data.