Skip to content

Commit 71e9a1a

Browse files
committed
Hack to load OPTIONS file for read_amp_bytes_per_bit (facebook#7659)
Summary: A temporary hack to work around a bug in 6.10, 6.11, 6.12, 6.13 and 6.14. The bug will write out 8 bytes to OPTIONS file from the starting address of BlockBasedTableOptions.read_amp_bytes_per_bit which is actually a uint32. Consequently, the value of read_amp_bytes_per_bit written in the OPTIONS file is wrong. From 6.15, RocksDB will try to parse the read_amp_bytes_per_bit from OPTIONS file as a uint32. To be able to load OPTIONS file generated by affected releases before the fix, we need to manually parse read_amp_bytes_per_bit with this hack. Pull Request resolved: facebook#7659 Test Plan: Generate a db with current 6.14.fb (head at facebook@b6db05d). Maybe use db_stress. Checkout this PR, run ``` ~/rocksdb/ldb --db=. --try_load_options --ignore_unknown_options idump --count_only ``` Expect success, and should not see ``` Failed: Invalid argument: Error parsing read_amp_bytes_per_bit:17179869184 ``` Also make check Reviewed By: anand1976 Differential Revision: D24954752 Pulled By: riversand963 fbshipit-source-id: c7b802fc3e52acd050a4fc1cd475016122234394
1 parent ebaf09d commit 71e9a1a

File tree

2 files changed

+25
-2
lines changed

2 files changed

+25
-2
lines changed

options/options_test.cc

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -838,7 +838,12 @@ TEST_F(OptionsTest, GetBlockBasedTableOptionsFromString) {
838838
"block_cache=1M;block_cache_compressed=1k;block_size=1024;"
839839
"block_size_deviation=8;block_restart_interval=4;"
840840
"format_version=5;whole_key_filtering=1;"
841-
"filter_policy=bloomfilter:4.567:false;",
841+
"filter_policy=bloomfilter:4.567:false;"
842+
// A bug caused read_amp_bytes_per_bit to be a large integer in OPTIONS
843+
// file generated by 6.10 to 6.14. Though bug is fixed in these releases,
844+
// we need to handle the case of loading OPTIONS file generated before the
845+
// fix.
846+
"read_amp_bytes_per_bit=17179869185;",
842847
&new_opt));
843848
ASSERT_TRUE(new_opt.cache_index_and_filter_blocks);
844849
ASSERT_EQ(new_opt.index_type, BlockBasedTableOptions::kHashSearch);
@@ -858,6 +863,9 @@ TEST_F(OptionsTest, GetBlockBasedTableOptionsFromString) {
858863
dynamic_cast<const BloomFilterPolicy&>(*new_opt.filter_policy);
859864
EXPECT_EQ(bfp.GetMillibitsPerKey(), 4567);
860865
EXPECT_EQ(bfp.GetWholeBitsPerKey(), 5);
866+
// Verify that only the lower 32bits are stored in
867+
// new_opt.read_amp_bytes_per_bit.
868+
EXPECT_EQ(1U, new_opt.read_amp_bytes_per_bit);
861869

862870
// unknown option
863871
Status s = GetBlockBasedTableOptionsFromString(

table/block_based/block_based_table_factory.cc

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -334,7 +334,22 @@ static std::unordered_map<std::string, OptionTypeInfo>
334334
{"read_amp_bytes_per_bit",
335335
{offsetof(struct BlockBasedTableOptions, read_amp_bytes_per_bit),
336336
OptionType::kUInt32T, OptionVerificationType::kNormal,
337-
OptionTypeFlags::kNone}},
337+
OptionTypeFlags::kNone,
338+
[](const ConfigOptions& /*opts*/, const std::string& /*name*/,
339+
const std::string& value, char* addr) {
340+
// A workaround to fix a bug in 6.10, 6.11, 6.12, 6.13
341+
// and 6.14. The bug will write out 8 bytes to OPTIONS file from the
342+
// starting address of BlockBasedTableOptions.read_amp_bytes_per_bit
343+
// which is actually a uint32. Consequently, the value of
344+
// read_amp_bytes_per_bit written in the OPTIONS file is wrong.
345+
// From 6.15, RocksDB will try to parse the read_amp_bytes_per_bit
346+
// from OPTIONS file as a uint32. To be able to load OPTIONS file
347+
// generated by affected releases before the fix, we need to
348+
// manually parse read_amp_bytes_per_bit with this special hack.
349+
uint64_t read_amp_bytes_per_bit = ParseUint64(value);
350+
EncodeFixed32(addr, static_cast<uint32_t>(read_amp_bytes_per_bit));
351+
return Status::OK();
352+
}}},
338353
{"enable_index_compression",
339354
{offsetof(struct BlockBasedTableOptions, enable_index_compression),
340355
OptionType::kBoolean, OptionVerificationType::kNormal,

0 commit comments

Comments
 (0)