[Proposal] Add crc to detect errors in wal entry

[lsytj0413]

Summary

When user writes data to oxia, the corresponding request (such as WriteRequest) is processed by the Leader. The Leader first writes the request information to the wal, then asynchronously replicate the data to the Follower through ReplicationStream, waits for the number of nodes in the Quorum to persist the data to the wal, and then applies the log to the state machine through CommitOffset. And the Follower will wait for the CommitOffset update from the Leader before applying the logs to the state machine.

When a new Leader starts, it will also read out all data from the wal and apply it to the state machine, and then start accepting user requests (here, a lock is used to ensure that only one operation is being executed).

The accuracy and consistency of the wal information are crucial in oxia system. This proposal verifies the accuracy of the data by adding crc to the wal entry.

Motivation

The main goal is to avoid Data corruption in wal, which may occur at multiple levels, such as when only a portion of the data is written, or when unknown errors occur during data storage, resulting in certain bytes in the data being scrambled.

Goals

• Add crc to the wal entry

Non-Goals

Proposal

User Stories

• When writing data, only a portion was written, which can be verified through CRC information during reading
• The data is correct when written, but it can be verified during reading after underlying hardware failure
• The CRC information can be used to verify data consistency between multiple replicas

Risks and Mitigations

• Increased the amount of data on the disk

Design Details

• Segment File
  • Each file will have an CRC Entry at the first offset，and it's value was previous segment file's last Log Entry's Data crc，If current Segment File is first segment，it CRC Entry's value is 0
  • wal data is persistent as Log Entry
• Entry Format
  • CRC Entry：8-bytes Entry Length，8-bytes Previous Segment CRC
  • Log Entry：8-bytes Entry Length，8x-bytes Entry Data，8-bytes Previous Segment CRC
• Entry Length Format
  • The first 1-bytes is Entry Type, current we support Crc Entry and Log Entry
  • The second 1-bytes is Entry Padding, it's value is 0-7，because we will padding Entry Data to 8x-bytes length, for example, if Entry Data is 9-bytes, we will padding it to 16-bytes，and set Entry Padding to 7
  • The last 4-bytes is Entry Data length as bytes

Test Plan

Add ALICE tests.

Upgrade / Downgrade Strategy

Because oxia didn't release stable version，so we plan didn't support previous version to upgrade.

Alternatives

Future possibilities

[merlimat]

@lsytj0413 Sorry for long time to get feedback here.

A couple of thoughts:

1. Since we're just looking at data corruption detection (rather than cryptographically prove of integrity), even 4 bytes CRC would be able to do a decent job there, with less overhead
2. I'm not sure I understand why do we need the CRC at the segment level. Couldn't just read the entry at the previous logical offset and get the CRC from there?

We should also describe few examples on how the CRC will be used:

1. When doing a Truncate() to followers, the leader should include the CRC and the follower should compare it with its own local entry. This would help double-checking the integrity of the replicas.
2. What to do when we're replaying the log and we detect CRC inconsistencies
3. The follower should double check the CRC before applying on the local WAL. This will protect with on-the-wire data corruption.

[lsytj0413]

Answer for thoughts：

• 4-bytes CRC is enough，but if we change this，we should reform the whole Log Entry to ensure Entry Data + Data CRC is 8-byte align，so Entry Length is write in one section（to avoid torn write）
• We cann't read previous CRC if previous segment file is truncated, so there is an backup

Answer for examples：

• Yep, we should add this as an advance feature
• To my opinion，we should just exit and wait the administrator to fix it，because the data is corrupt，we can not skip it or apply it
• Yep

[lsytj0413]

close because #538 already implement it.