Skip to content

Commit 5878f35

Browse files
committed
Merge bitcoin/bitcoin#31144: [IBD] multi-byte block obfuscation
248b6a2 optimization: peel align-head and unroll body to 64 bytes (Lőrinc) e7114fc optimization: migrate fixed-size obfuscation from `std::vector<std::byte>` to `uint64_t` (Lőrinc) 478d40a refactor: encapsulate `vector`/`array` keys into `Obfuscation` (Lőrinc) 377aab8 refactor: move `util::Xor` to `Obfuscation().Xor` (Lőrinc) fa5d296 refactor: prepare mempool_persist for obfuscation key change (Lőrinc) 6bbf2d9 refactor: prepare `DBWrapper` for obfuscation key change (Lőrinc) 0b8bec8 scripted-diff: unify xor-vs-obfuscation nomenclature (Lőrinc) 9726979 bench: make ObfuscationBench more representative (Lőrinc) 618a30e test: compare util::Xor with randomized inputs against simple impl (Lőrinc) a5141cd test: make sure dbwrapper obfuscation key is never obfuscated (Lőrinc) 54ab0bd refactor: commit to 8 byte obfuscation keys (Lőrinc) 7aa557a random: add fixed-size `std::array` generation (Lőrinc) Pull request description: This change is part of [[IBD] - Tracking PR for speeding up Initial Block Download](bitcoin/bitcoin#32043) ### Summary Current block obfuscations are done byte-by-byte, this PR batches them to 64 bit primitives to speed up obfuscating bigger memory batches. This is especially relevant now that bitcoin/bitcoin#31551 was merged, having bigger obfuscatable chunks. Since this obfuscation is optional, the speedup measured here depends on whether it's a [random value](bitcoin/bitcoin#31144 (comment)) or [completely turned off](bitcoin/bitcoin#31144 (comment)) (i.e. XOR-ing with 0). ### Changes in testing, benchmarking and implementation * Added new tests comparing randomized inputs against a trivial implementation and performing roundtrip checks with random chunks. * Migrated `std::vector<std::byte>(8)` keys to plain `uint64_t`; * Process unaligned bytes separately and unroll body to 64 bytes. ### Assembly Memory alignment is enforced by a small peel-loop (`std::memcpy` is optimized out on tested platform), with an `std::assume_aligned<8>` check, see the Godbolt listing at https://godbolt.org/z/59EMv7h6Y for details <details> <summary>Details</summary> Target & Compiler | Stride (per hot-loop iter) | Main operation(s) in loop | Effective XORs / iter -- | -- | -- | -- Clang x86-64 (trunk) | 64 bytes | 4 × movdqu → pxor → store | 8 × 64-bit GCC x86-64 (trunk) | 64 bytes | 4 × movdqu/pxor sequence, enabled by 8-way unroll | 8 × 64-bit GCC RV32 (trunk) | 8 bytes | copy 8 B to temp → 2 × 32-bit XOR → copy back | 1 × 64-bit (as 2 × 32-bit) GCC s390x (big-endian 14.2) | 64 bytes | 8 × XC (mem-mem 8-B XOR) with key cached on stack | 8 × 64-bit </details> ### Endianness The only endianness issue was with bit rotation, intended to realign the key if obfuscation halted before full key consumption. Elsewhere, memory is read, processed, and written back in the same endianness, preserving byte order. Since CI lacks a big-endian machine, testing was done locally via Docker. <details> <summary>Details</summary> ```bash brew install podman pigz softwareupdate --install-rosetta podman machine init podman machine start docker run --platform linux/s390x -it ubuntu:latest /bin/bash apt update && apt install -y git build-essential cmake ccache pkg-config libevent-dev libboost-dev libssl-dev libsqlite3-dev python3 && \ cd /mnt && git clone --depth=1 https://github.com/bitcoin/bitcoin.git && cd bitcoin && git remote add l0rinc https://github.com/l0rinc/bitcoin.git && git fetch --all && git checkout l0rinc/optimize-xor && \ cmake -B build && cmake --build build --target test_bitcoin -j$(nproc) && \ ./build/bin/test_bitcoin --run_test=streams_tests ``` </details> ### Measurements (micro benchmarks and full IBDs) > cmake -B build -DBUILD_BENCH=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=gcc/clang -DCMAKE_CXX_COMPILER=g++/clang++ && \ cmake --build build -j$(nproc) && \ build/bin/bench_bitcoin -filter='ObfuscationBench' -min-time=5000 <details> <summary>GNU 14.2.0</summary> > Before: | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.84 | 1,184,138,235.64 | 0.0% | 9.01 | 3.03 | 2.971 | 1.00 | 0.1% | 5.50 | `ObfuscationBench` > After (first optimizing commit): | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.04 | 28,365,698,819.44 | 0.0% | 0.34 | 0.13 | 2.714 | 0.07 | 0.0% | 5.33 | `ObfuscationBench` > and (second optimizing commit): | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.03 | 32,464,658,919.11 | 0.0% | 0.50 | 0.11 | 4.474 | 0.08 | 0.0% | 5.29 | `ObfuscationBench` </details> <details> <summary>Clang 20.1.7</summary> > Before: | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.89 | 1,124,087,330.23 | 0.1% | 6.52 | 3.20 | 2.041 | 0.50 | 0.2% | 5.50 | `ObfuscationBench` > After (first optimizing commit): | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.08 | 13,012,464,203.00 | 0.0% | 0.65 | 0.28 | 2.338 | 0.13 | 0.8% | 5.50 | `ObfuscationBench` > and (second optimizing commit): | ns/byte | byte/s | err% | ins/byte | cyc/byte | IPC | bra/byte | miss% | total | benchmark |--------------------:|--------------------:|--------:|----------------:|----------------:|-------:|---------------:|--------:|----------:|:---------- | 0.02 | 41,231,547,045.17 | 0.0% | 0.30 | 0.09 | 3.463 | 0.02 | 0.0% | 5.47 | `ObfuscationBench` </details> i.e. 27.4x faster obfuscation with GCC, 36.7x faster with Clang For other benchmark speedups see https://corecheck.dev/bitcoin/bitcoin/pulls/31144 ------ Running an IBD until 888888 blocks reveals a 4% speedup. <details> <summary>Details</summary> SSD: ```bash COMMITS="8324a00bd4a6a5291c841f2d01162d8a014ddb02 5ddfd31b4158a89b0007cfb2be970c03d9278525"; \ STOP_HEIGHT=888888; DBCACHE=1000; \ CC=gcc; CXX=g++; \ BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \ (for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \ hyperfine \ --sort 'command' \ --runs 1 \ --export-json "$BASE_DIR/ibd-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \ --parameter-list COMMIT ${COMMITS// /,} \ --prepare "killall bitcoind; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard; \ cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && \ cmake --build build -j$(nproc) --target bitcoind && \ ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 100" \ --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \ "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -printtoconsole=0" ``` > 8324a00bd4 test: Compare util::Xor with randomized inputs against simple impl > 5ddfd31b41 optimization: Xor 64 bits together instead of byte-by-byte ```python Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 8324a00bd4a6a5291c841f2d01162d8a014ddb02) Time (abs ≡): 25033.413 s [User: 33953.984 s, System: 2613.604 s] Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 5ddfd31b4158a89b0007cfb2be970c03d9278525) Time (abs ≡): 24110.710 s [User: 33389.536 s, System: 2660.292 s] Relative speed comparison 1.04 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 8324a00bd4a6a5291c841f2d01162d8a014ddb02) 1.00 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=1000 -blocksonly -printtoconsole=0 (COMMIT = 5ddfd31b4158a89b0007cfb2be970c03d9278525) ``` > HDD: ```bash COMMITS="71eb6eaa740ad0b28737e90e59b89a8e951d90d9 46854038e7984b599d25640de26d4680e62caba7"; \ STOP_HEIGHT=888888; DBCACHE=4500; \ CC=gcc; CXX=g++; \ BASE_DIR="/mnt/my_storage"; DATA_DIR="$BASE_DIR/BitcoinData"; LOG_DIR="$BASE_DIR/logs"; \ (for c in $COMMITS; do git fetch origin $c -q && git log -1 --pretty=format:'%h %s' $c || exit 1; done) && \ hyperfine \ --sort 'command' \ --runs 2 \ --export-json "$BASE_DIR/ibd-${COMMITS// /-}-$STOP_HEIGHT-$DBCACHE-$CC.json" \ --parameter-list COMMIT ${COMMITS// /,} \ --prepare "killall bitcoind; rm -rf $DATA_DIR/*; git checkout {COMMIT}; git clean -fxd; git reset --hard; \ cmake -B build -DCMAKE_BUILD_TYPE=Release -DENABLE_WALLET=OFF && cmake --build build -j$(nproc) --target bitcoind && \ ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=1 -printtoconsole=0; sleep 100" \ --cleanup "cp $DATA_DIR/debug.log $LOG_DIR/debug-{COMMIT}-$(date +%s).log" \ "COMPILER=$CC ./build/bin/bitcoind -datadir=$DATA_DIR -stopatheight=$STOP_HEIGHT -dbcache=$DBCACHE -blocksonly -printtoconsole=0" ``` > 71eb6eaa74 test: compare util::Xor with randomized inputs against simple impl > 46854038e7 optimization: migrate fixed-size obfuscation from `std::vector<std::byte>` to `uint64_t` ```python Benchmark 1: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 71eb6eaa740ad0b28737e90e59b89a8e951d90d9) Time (mean ± σ): 37676.293 s ± 83.100 s [User: 36900.535 s, System: 2220.382 s] Range (min … max): 37617.533 s … 37735.053 s 2 runs Benchmark 2: COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 46854038e7984b599d25640de26d4680e62caba7) Time (mean ± σ): 36181.287 s ± 195.248 s [User: 34962.822 s, System: 1988.614 s] Range (min … max): 36043.226 s … 36319.349 s 2 runs Relative speed comparison 1.04 ± 0.01 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 71eb6eaa740ad0b28737e90e59b89a8e951d90d9) 1.00 COMPILER=gcc ./build/bin/bitcoind -datadir=/mnt/my_storage/BitcoinData -stopatheight=888888 -dbcache=4500 -blocksonly -printtoconsole=0 (COMMIT = 46854038e7984b599d25640de26d4680e62caba7) ``` </details> ACKs for top commit: achow101: ACK 248b6a2 maflcko: review ACK 248b6a2 🎻 ryanofsky: Code review ACK 248b6a2. Looks good! Thanks for adapting this and considering all the suggestions. I did leave more comments below but non are important and this looks good as-is Tree-SHA512: ef541cd8a1f1dc504613c4eaa708202e32ae5ac86f9c875e03bcdd6357121f6af0860ef83d513c473efa5445b701e59439d416effae1085a559716b0fd45ecd6
2 parents e9edd43 + 248b6a2 commit 5878f35

16 files changed

+363
-187
lines changed

src/bench/CMakeLists.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ add_executable(bench_bitcoin
3535
mempool_eviction.cpp
3636
mempool_stress.cpp
3737
merkle_root.cpp
38+
obfuscation.cpp
3839
parse_hex.cpp
3940
peer_eviction.cpp
4041
poly1305.cpp
@@ -52,7 +53,6 @@ add_executable(bench_bitcoin
5253
txorphanage.cpp
5354
util_time.cpp
5455
verify_script.cpp
55-
xor.cpp
5656
)
5757

5858
include(TargetDataSources)

src/bench/xor.cpp renamed to src/bench/obfuscation.cpp

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,21 +4,22 @@
44

55
#include <bench/bench.h>
66
#include <random.h>
7-
#include <span.h>
8-
#include <streams.h>
7+
#include <util/obfuscation.h>
98

109
#include <cstddef>
1110
#include <vector>
1211

13-
static void Xor(benchmark::Bench& bench)
12+
static void ObfuscationBench(benchmark::Bench& bench)
1413
{
1514
FastRandomContext frc{/*fDeterministic=*/true};
1615
auto data{frc.randbytes<std::byte>(1024)};
17-
auto key{frc.randbytes<std::byte>(31)};
16+
const Obfuscation obfuscation{frc.randbytes<Obfuscation::KEY_SIZE>()};
1817

18+
size_t offset{0};
1919
bench.batch(data.size()).unit("byte").run([&] {
20-
util::Xor(data, key);
20+
obfuscation(data, offset++); // mutated differently each time
21+
ankerl::nanobench::doNotOptimizeAway(data);
2122
});
2223
}
2324

24-
BENCHMARK(Xor, benchmark::PriorityLevel::HIGH);
25+
BENCHMARK(ObfuscationBench, benchmark::PriorityLevel::HIGH);

src/dbwrapper.cpp

Lines changed: 11 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
#include <streams.h>
1212
#include <util/fs.h>
1313
#include <util/fs_helpers.h>
14+
#include <util/obfuscation.h>
1415
#include <util/strencodings.h>
1516

1617
#include <algorithm>
@@ -173,7 +174,7 @@ void CDBBatch::Clear()
173174
void CDBBatch::WriteImpl(std::span<const std::byte> key, DataStream& ssValue)
174175
{
175176
leveldb::Slice slKey(CharCast(key.data()), key.size());
176-
ssValue.Xor(dbwrapper_private::GetObfuscateKey(parent));
177+
dbwrapper_private::GetObfuscation(parent)(ssValue);
177178
leveldb::Slice slValue(CharCast(ssValue.data()), ssValue.size());
178179
m_impl_batch->batch.Put(slKey, slValue);
179180
}
@@ -248,24 +249,14 @@ CDBWrapper::CDBWrapper(const DBParams& params)
248249
LogPrintf("Finished database compaction of %s\n", fs::PathToString(params.path));
249250
}
250251

251-
// The base-case obfuscation key, which is a noop.
252-
obfuscate_key = std::vector<unsigned char>(OBFUSCATE_KEY_NUM_BYTES, '\000');
253-
254-
bool key_exists = Read(OBFUSCATE_KEY_KEY, obfuscate_key);
255-
256-
if (!key_exists && params.obfuscate && IsEmpty()) {
257-
// Initialize non-degenerate obfuscation if it won't upset
258-
// existing, non-obfuscated data.
259-
std::vector<unsigned char> new_key = CreateObfuscateKey();
260-
261-
// Write `new_key` so we don't obfuscate the key with itself
262-
Write(OBFUSCATE_KEY_KEY, new_key);
263-
obfuscate_key = new_key;
264-
265-
LogPrintf("Wrote new obfuscate key for %s: %s\n", fs::PathToString(params.path), HexStr(obfuscate_key));
252+
assert(!m_obfuscation); // Needed for unobfuscated Read()/Write() below
253+
if (!Read(OBFUSCATION_KEY_KEY, m_obfuscation) && params.obfuscate && IsEmpty()) {
254+
// Generate, write and read back the new obfuscation key, making sure we don't obfuscate the key itself
255+
Write(OBFUSCATION_KEY_KEY, FastRandomContext{}.randbytes(Obfuscation::KEY_SIZE));
256+
Read(OBFUSCATION_KEY_KEY, m_obfuscation);
257+
LogInfo("Wrote new obfuscation key for %s: %s", fs::PathToString(params.path), m_obfuscation.HexKey());
266258
}
267-
268-
LogPrintf("Using obfuscation key for %s: %s\n", fs::PathToString(params.path), HexStr(obfuscate_key));
259+
LogInfo("Using obfuscation key for %s: %s", fs::PathToString(params.path), m_obfuscation.HexKey());
269260
}
270261

271262
CDBWrapper::~CDBWrapper()
@@ -310,25 +301,6 @@ size_t CDBWrapper::DynamicMemoryUsage() const
310301
return parsed.value();
311302
}
312303

313-
// Prefixed with null character to avoid collisions with other keys
314-
//
315-
// We must use a string constructor which specifies length so that we copy
316-
// past the null-terminator.
317-
const std::string CDBWrapper::OBFUSCATE_KEY_KEY("\000obfuscate_key", 14);
318-
319-
const unsigned int CDBWrapper::OBFUSCATE_KEY_NUM_BYTES = 8;
320-
321-
/**
322-
* Returns a string (consisting of 8 random bytes) suitable for use as an
323-
* obfuscating XOR key.
324-
*/
325-
std::vector<unsigned char> CDBWrapper::CreateObfuscateKey() const
326-
{
327-
std::vector<uint8_t> ret(OBFUSCATE_KEY_NUM_BYTES);
328-
GetRandBytes(ret);
329-
return ret;
330-
}
331-
332304
std::optional<std::string> CDBWrapper::ReadImpl(std::span<const std::byte> key) const
333305
{
334306
leveldb::Slice slKey(CharCast(key.data()), key.size());
@@ -412,9 +384,9 @@ void CDBIterator::Next() { m_impl_iter->iter->Next(); }
412384

413385
namespace dbwrapper_private {
414386

415-
const std::vector<unsigned char>& GetObfuscateKey(const CDBWrapper &w)
387+
const Obfuscation& GetObfuscation(const CDBWrapper& w)
416388
{
417-
return w.obfuscate_key;
389+
return w.m_obfuscation;
418390
}
419391

420392
} // namespace dbwrapper_private

src/dbwrapper.h

Lines changed: 8 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,6 @@
1818
#include <optional>
1919
#include <stdexcept>
2020
#include <string>
21-
#include <vector>
2221

2322
static const size_t DBWRAPPER_PREALLOC_KEY_SIZE = 64;
2423
static const size_t DBWRAPPER_PREALLOC_VALUE_SIZE = 1024;
@@ -63,8 +62,7 @@ namespace dbwrapper_private {
6362
* Database obfuscation should be considered an implementation detail of the
6463
* specific database.
6564
*/
66-
const std::vector<unsigned char>& GetObfuscateKey(const CDBWrapper &w);
67-
65+
const Obfuscation& GetObfuscation(const CDBWrapper&);
6866
}; // namespace dbwrapper_private
6967

7068
bool DestroyDB(const std::string& path_str);
@@ -166,7 +164,7 @@ class CDBIterator
166164
template<typename V> bool GetValue(V& value) {
167165
try {
168166
DataStream ssValue{GetValueImpl()};
169-
ssValue.Xor(dbwrapper_private::GetObfuscateKey(parent));
167+
dbwrapper_private::GetObfuscation(parent)(ssValue);
170168
ssValue >> value;
171169
} catch (const std::exception&) {
172170
return false;
@@ -179,24 +177,19 @@ struct LevelDBContext;
179177

180178
class CDBWrapper
181179
{
182-
friend const std::vector<unsigned char>& dbwrapper_private::GetObfuscateKey(const CDBWrapper &w);
180+
friend const Obfuscation& dbwrapper_private::GetObfuscation(const CDBWrapper&);
183181
private:
184182
//! holds all leveldb-specific fields of this class
185183
std::unique_ptr<LevelDBContext> m_db_context;
186184

187185
//! the name of this database
188186
std::string m_name;
189187

190-
//! a key used for optional XOR-obfuscation of the database
191-
std::vector<unsigned char> obfuscate_key;
192-
193-
//! the key under which the obfuscation key is stored
194-
static const std::string OBFUSCATE_KEY_KEY;
195-
196-
//! the length of the obfuscate key in number of bytes
197-
static const unsigned int OBFUSCATE_KEY_NUM_BYTES;
188+
//! optional XOR-obfuscation of the database
189+
Obfuscation m_obfuscation;
198190

199-
std::vector<unsigned char> CreateObfuscateKey() const;
191+
//! obfuscation key storage key, null-prefixed to avoid collisions
192+
inline static const std::string OBFUSCATION_KEY_KEY{"\000obfuscate_key", 14}; // explicit size to avoid truncation at leading \0
200193

201194
//! path to filesystem storage
202195
const fs::path m_path;
@@ -228,7 +221,7 @@ class CDBWrapper
228221
}
229222
try {
230223
DataStream ssValue{MakeByteSpan(*strValue)};
231-
ssValue.Xor(obfuscate_key);
224+
m_obfuscation(ssValue);
232225
ssValue >> value;
233226
} catch (const std::exception&) {
234227
return false;

src/node/blockstorage.cpp

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@
3131
#include <util/batchpriority.h>
3232
#include <util/check.h>
3333
#include <util/fs.h>
34+
#include <util/obfuscation.h>
3435
#include <util/signalinterrupt.h>
3536
#include <util/strencodings.h>
3637
#include <util/syserror.h>
@@ -779,13 +780,13 @@ void BlockManager::UnlinkPrunedFiles(const std::set<int>& setFilesToPrune) const
779780

780781
AutoFile BlockManager::OpenBlockFile(const FlatFilePos& pos, bool fReadOnly) const
781782
{
782-
return AutoFile{m_block_file_seq.Open(pos, fReadOnly), m_xor_key};
783+
return AutoFile{m_block_file_seq.Open(pos, fReadOnly), m_obfuscation};
783784
}
784785

785786
/** Open an undo file (rev?????.dat) */
786787
AutoFile BlockManager::OpenUndoFile(const FlatFilePos& pos, bool fReadOnly) const
787788
{
788-
return AutoFile{m_undo_file_seq.Open(pos, fReadOnly), m_xor_key};
789+
return AutoFile{m_undo_file_seq.Open(pos, fReadOnly), m_obfuscation};
789790
}
790791

791792
fs::path BlockManager::GetBlockPosFilename(const FlatFilePos& pos) const
@@ -1123,7 +1124,7 @@ static auto InitBlocksdirXorKey(const BlockManager::Options& opts)
11231124
{
11241125
// Bytes are serialized without length indicator, so this is also the exact
11251126
// size of the XOR-key file.
1126-
std::array<std::byte, 8> xor_key{};
1127+
std::array<std::byte, Obfuscation::KEY_SIZE> obfuscation{};
11271128

11281129
// Consider this to be the first run if the blocksdir contains only hidden
11291130
// files (those which start with a .). Checking for a fully-empty dir would
@@ -1140,14 +1141,14 @@ static auto InitBlocksdirXorKey(const BlockManager::Options& opts)
11401141
if (opts.use_xor && first_run) {
11411142
// Only use random fresh key when the boolean option is set and on the
11421143
// very first start of the program.
1143-
FastRandomContext{}.fillrand(xor_key);
1144+
FastRandomContext{}.fillrand(obfuscation);
11441145
}
11451146

11461147
const fs::path xor_key_path{opts.blocks_dir / "xor.dat"};
11471148
if (fs::exists(xor_key_path)) {
11481149
// A pre-existing xor key file has priority.
11491150
AutoFile xor_key_file{fsbridge::fopen(xor_key_path, "rb")};
1150-
xor_key_file >> xor_key;
1151+
xor_key_file >> obfuscation;
11511152
} else {
11521153
// Create initial or missing xor key file
11531154
AutoFile xor_key_file{fsbridge::fopen(xor_key_path,
@@ -1157,28 +1158,28 @@ static auto InitBlocksdirXorKey(const BlockManager::Options& opts)
11571158
"wbx"
11581159
#endif
11591160
)};
1160-
xor_key_file << xor_key;
1161+
xor_key_file << obfuscation;
11611162
if (xor_key_file.fclose() != 0) {
11621163
throw std::runtime_error{strprintf("Error closing XOR key file %s: %s",
11631164
fs::PathToString(xor_key_path),
11641165
SysErrorString(errno))};
11651166
}
11661167
}
11671168
// If the user disabled the key, it must be zero.
1168-
if (!opts.use_xor && xor_key != decltype(xor_key){}) {
1169+
if (!opts.use_xor && obfuscation != decltype(obfuscation){}) {
11691170
throw std::runtime_error{
11701171
strprintf("The blocksdir XOR-key can not be disabled when a random key was already stored! "
11711172
"Stored key: '%s', stored path: '%s'.",
1172-
HexStr(xor_key), fs::PathToString(xor_key_path)),
1173+
HexStr(obfuscation), fs::PathToString(xor_key_path)),
11731174
};
11741175
}
1175-
LogInfo("Using obfuscation key for blocksdir *.dat files (%s): '%s'\n", fs::PathToString(opts.blocks_dir), HexStr(xor_key));
1176-
return std::vector<std::byte>{xor_key.begin(), xor_key.end()};
1176+
LogInfo("Using obfuscation key for blocksdir *.dat files (%s): '%s'\n", fs::PathToString(opts.blocks_dir), HexStr(obfuscation));
1177+
return Obfuscation{obfuscation};
11771178
}
11781179

11791180
BlockManager::BlockManager(const util::SignalInterrupt& interrupt, Options opts)
11801181
: m_prune_mode{opts.prune_target > 0},
1181-
m_xor_key{InitBlocksdirXorKey(opts)},
1182+
m_obfuscation{InitBlocksdirXorKey(opts)},
11821183
m_opts{std::move(opts)},
11831184
m_block_file_seq{FlatFileSeq{m_opts.blocks_dir, "blk", m_opts.fast_prune ? 0x4000 /* 16kB */ : BLOCKFILE_CHUNK_SIZE}},
11841185
m_undo_file_seq{FlatFileSeq{m_opts.blocks_dir, "rev", UNDOFILE_CHUNK_SIZE}},

src/node/blockstorage.h

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -235,7 +235,7 @@ class BlockManager
235235

236236
const bool m_prune_mode;
237237

238-
const std::vector<std::byte> m_xor_key;
238+
const Obfuscation m_obfuscation;
239239

240240
/** Dirty block index entries. */
241241
std::set<CBlockIndex*> m_dirty_blockindex;

src/node/mempool_persist.cpp

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
#include <uint256.h>
1717
#include <util/fs.h>
1818
#include <util/fs_helpers.h>
19+
#include <util/obfuscation.h>
1920
#include <util/signalinterrupt.h>
2021
#include <util/syserror.h>
2122
#include <util/time.h>
@@ -59,15 +60,17 @@ bool LoadMempool(CTxMemPool& pool, const fs::path& load_path, Chainstate& active
5960
try {
6061
uint64_t version;
6162
file >> version;
62-
std::vector<std::byte> xor_key;
63+
6364
if (version == MEMPOOL_DUMP_VERSION_NO_XOR_KEY) {
64-
// Leave XOR-key empty
65+
file.SetObfuscation({});
6566
} else if (version == MEMPOOL_DUMP_VERSION) {
66-
file >> xor_key;
67+
Obfuscation obfuscation;
68+
file >> obfuscation;
69+
file.SetObfuscation(obfuscation);
6770
} else {
6871
return false;
6972
}
70-
file.SetXor(xor_key);
73+
7174
uint64_t total_txns_to_load;
7275
file >> total_txns_to_load;
7376
uint64_t txns_tried = 0;
@@ -179,12 +182,13 @@ bool DumpMempool(const CTxMemPool& pool, const fs::path& dump_path, FopenFn mock
179182
const uint64_t version{pool.m_opts.persist_v1_dat ? MEMPOOL_DUMP_VERSION_NO_XOR_KEY : MEMPOOL_DUMP_VERSION};
180183
file << version;
181184

182-
std::vector<std::byte> xor_key(8);
183185
if (!pool.m_opts.persist_v1_dat) {
184-
FastRandomContext{}.fillrand(xor_key);
185-
file << xor_key;
186+
const Obfuscation obfuscation{FastRandomContext{}.randbytes<Obfuscation::KEY_SIZE>()};
187+
file << obfuscation;
188+
file.SetObfuscation(obfuscation);
189+
} else {
190+
file.SetObfuscation({});
186191
}
187-
file.SetXor(xor_key);
188192

189193
uint64_t mempool_transactions_to_write(vinfo.size());
190194
file << mempool_transactions_to_write;

src/random.h

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -301,6 +301,15 @@ class RandomMixin
301301
return ret;
302302
}
303303

304+
/** Generate fixed-size random bytes. */
305+
template <size_t N, BasicByte B = std::byte>
306+
std::array<B, N> randbytes() noexcept
307+
{
308+
std::array<B, N> ret;
309+
Impl().fillrand(MakeWritableByteSpan(ret));
310+
return ret;
311+
}
312+
304313
/** Generate a random 32-bit integer. */
305314
uint32_t rand32() noexcept { return Impl().template randbits<32>(); }
306315

0 commit comments

Comments
 (0)