Skip to content

Commit 06bd95f

Browse files
committed
Fix the data loss issue that caused by the wrong entry log header
--- # Motivation We observed numerous errors in the broker that failed to read the ledger from the bookkeeper; although the ledger metadata still exists, it was unable to read from the bookkeeper. After checking the data, we found the ledger located entry log was deleted by the bookkeeper. We have a data loss issue with the bookkeeper. The entry log file was deleted by the Garbage collector because the entry log file wrote a wrong file header. And there is an example that the shows the header is wrong: ``` Failed to get ledgers map index from: 82.log : Not all ledgers were found in ledgers map index. expected: -1932430239 -- found: 0 -- entryLogId: 82 ```
1 parent 37b63c9 commit 06bd95f

File tree

3 files changed

+7
-1
lines changed

3 files changed

+7
-1
lines changed

bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/DefaultEntryLogger.java

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1151,6 +1151,10 @@ EntryLogMetadata extractEntryLogMetadataFromIndex(long entryLogId) throws IOExce
11511151
+ " -- found: " + meta.getLedgersMap().size() + " -- entryLogId: " + entryLogId);
11521152
}
11531153

1154+
if (header.ledgersCount == 0) {
1155+
throw new IOException("No ledgers map found in entryLogId " + entryLogId + ", do scan to double confirm");
1156+
}
1157+
11541158
return meta;
11551159
}
11561160

bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLoggerAllocator.java

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,7 @@ class EntryLoggerAllocator {
8585
// within the same JVM. All of these Bookie instances access this header
8686
// so there can be race conditions when entry logs are rolled over and
8787
// this header buffer is cleared before writing it into the new logChannel.
88+
logfileHeader.setZero(0, DefaultEntryLogger.LOGFILE_HEADER_SIZE);
8889
logfileHeader.writeBytes("BKLO".getBytes(UTF_8));
8990
logfileHeader.writeInt(DefaultEntryLogger.HEADER_CURRENT_VERSION);
9091
logfileHeader.writerIndex(DefaultEntryLogger.LOGFILE_HEADER_SIZE);

bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -842,11 +842,12 @@ protected void extractMetaFromEntryLogs() throws EntryLogMetadataMapException {
842842
continue;
843843
}
844844

845-
LOG.info("Extracting entry log meta from entryLogId: {}", entryLogId);
846845

847846
try {
848847
// Read through the entry log file and extract the entry log meta
849848
EntryLogMetadata entryLogMeta = entryLogger.getEntryLogMetadata(entryLogId, throttler);
849+
LOG.info("Extracted entry log meta from entryLogId: {}, ledgers {}",
850+
entryLogId, entryLogMeta.getLedgersMap().keys());
850851
removeIfLedgerNotExists(entryLogMeta);
851852
if (entryLogMeta.isEmpty()) {
852853
// This means the entry log is not associated with any active

0 commit comments

Comments
 (0)