Handling corruption/unreadable data in partitions

[jcsp]

Currently, redpanda generally assumes that on-disk content is valid, and if not then it may assert out.

Yesterday we discussed a little around this:

• Already have checksums on incoming data but not on what we write to disk
• Vassertions on unexpected version during decode, and/or other decode failures.
• asserting out is definitely better than ignoring the issue!
• On-disk corruption (bitflips etc) can be quite rare but become more probable linearly with number of users and system size
• Why do structured error handling?
  • because a large multiuser system should not become unavailable to 100 users' topics if only 1 topic got corrupted.  Minimize the blast radius to what is actually broken.

Can we auto-heal?

• For on-disk errors on only one node, we can potentially rebuild
• Simplest thing is to declare all of a node's storage for a partition untrusted, drop it and let raft replay everything
• More advanced would be to selectively fetch missing parts, but that only makes sense if there was only a localized bitrot, rather than some systemic issue like a software bug.

[emaxerrno]

So each batch has 2 checksums. For header and for data. Are we thinking for silent things like a batch we wrote a year ago?

s3 has minions that try to randomly read data and auto heals

[jcsp]

Silent corruptions, but also things like subtle software bugs where an index and a segment are inconsistent. Any time we read back something that doesn't make sense.

[emaxerrno]

Got it. Was just confused by what you wrote on that we don’t have checksums on data on disk (we do). But you are thinking a concurrency style bug where the checksums in memory at time of writing are correct but we’d only detect corruption at time of read.

Are there any databases that do this background healing?

[jcsp]

The on-disk checksum thing was a note from a call, if we already do that then 👍 (but we still need a way to cleanly handle a bad checksum)

Background scrubbing/healing is common in filesystems (e.g. ZFS zpool scrub), I'm not sure how much DBs do in the background vs manual processes like cassandra's anti-entropy feature.

[emaxerrno]

https://github.com/vectorizedio/redpanda/blob/dev/src/v/model/record.h#L354-L386 - we have header crc and payload crc (kafka) - in fact we are safer than kafka in that refard in that we can detect corruption of additional header fields.

[twmb]

Is the thought around recovery to use replicas?