[sys-4815] treat inconsistent epoch number as poison instead of corruption (#284)

The consistency check compares the replicated epoch number from leader against epoch number calculated based on existing files on followers. It's actually possible that the consistency check fails in following rare case:

D as leader running on old rocksdb, E as follower running on new rocksdb. Then D is re-opened as leader on new rocksdb, during which we recover the epoch number on db open. During recovery, the epoch number calculated on D might be different from the epoch number on E. But we won't detect the inconsistency until the mismatched files are compacted, which may only happen a few hours/days later.

This is fine since E will resync manifest files when re-opened. And very likely shard reopen has already happened during latest leaf roll. But out of precaution, let's treat this as poison so that E will reopen immediately instead of corruption (which will cause permanently errored shards) before enabling epoch replication for all orgs.

Author: seckcoder

db/db_impl/db_impl.cc

@@ -1448,12 +1448,13 @@ Status DBImpl::ApplyReplicationLogRecord(ReplicationLogRecord record,
                 cfd->SetNextEpochNumber(newFiles.begin()->second.epoch_number);
               }
 
-              // do consistency check by comparing the replicated epoch number against
-              // inferred epoch number
+              // do consistency check by comparing the replicated epoch number
+              // against inferred epoch number No need to
+              // `reset_next_epoch_number` here since we have already done it
               s = InferEpochNumber(&e, cfd, info,
                                    false /* reset_next_epoch_number */);
               if (s.ok() && info->mismatched_epoch_num > 0) {
-                s = Status::Corruption("epoch number consistency check fails");
+                s = Status::Poison("epoch number consistency check fails");
               }
               if (!s.ok()) {
                 break;