cryptonote_core: support RandomX V2 and commitments in v17

See docs/BLOCK_HASHING.md

This commit does required consensus changes only.

Author: jeffro256

docs/BLOCK_HASHING.md

@@ -0,0 +1,52 @@
+# Block hashing
+
+## Background
+
+In Bitcoin, block IDs and Proof-of-Work (PoW) hashes are calculated the same way, which is why on-chain block IDs always start
+with many 0s. However, in Monero, block IDs and PoW hashes are calcuated using different hash functions and slightly different
+inputs. This is because the hash functions used for PoW (CryptoNight and RandomX) are much slower to run than other
+cryptographic hash functions, specifically Keccak256, which Monero uses for block IDs. By contrast, Bitcoin uses SHA-256 for
+both block IDs and PoW. The reason that CryptoNight and RandomX were chosen for PoW in Monero is to protect against
+application-specific integrated circuits (ASICs) from dominanting the network hashrate like in Bitcoin. In theory, this
+would restore the principle of "1 CPU, 1 vote" outlined in the Satoshi whitepaper, and bring greater decentralization to the
+Monero exosystem. This document aims to describe exactly how block IDs and PoW hashes are calculated in the Monero reference
+codebase.
+
+## CryptoNight epoch
+
+To be reductive, the underlying design of Cryptonight is to 1) fill a 2MB buffer of memory using a memory-hard function, 2)
+perform over a millon random reads on that buffer to permutate some state, and then 3) do a final hash on the state using
+a random choice between four independent hash fuctions. There are 4 variants of CryptoNight: v0, v1, v2, and v4. This is the
+data flow for how block hashing in the CryptoNight epoch (Monero fork versions v1-v11) is performed:
+
+![CryptoNight data flow](resources/cryptonight-data-flow.png)
+
+## RandomX epoch
+
+Like, CryptoNight, RandomX also uses a memory hard function to fill a large space of memory called a "cache". However, this
+space is 256MB, not 2MB. It can be further expanded to 2GB to trade memory usage for compute speed. Unlike CryptoNight's cache,
+the RandomX cache is computed only once every 2048 blocks, not once per block. Monero uses a "seed" block ID of a previous block
+in the chain to fill the memory space. And to make ASICs even harder, RandomX uses random instruction execution as well. This
+is the data flow for how block hashing in the RandomX epoch (Monero fork versions v12-present) is performed:
+
+![RandomX data flow](resources/randomx-data-flow.png)
+
+## RandomX, with commitments, epoch
+
+Because RandomX can be slow to run, especially while building the cache (nearly 1/2 a second on my machine), it inadvertantly
+can introduce a Denial-of-Service (DoS) vector. For this reason, RandomX hash commitments can be added, so the PoW can be
+partially verified using only Blake2B, a much lighter hash function. For simplicity's sake, we can also remove the number of
+transactions in the block from the block hashing blob. This is the data flow for how block hashing works with these changes:
+
+![RandomX commitment data flow](resources/randomx-commitment-data-flow.png)
+
+The way that the hashes are calculated allows us to do partial PoW verification like so:
+
+![Partial RandomX verification data flow](resources/partial-pow-verify-data-flow.png)
+
+A node can be given simply the block header, block content hash, and intermediate PoW hash, and verify that some PoW was done
+using only `randomx_calculate_commitment()` (Blake2B undernath). Passing this partial verification requires PoW to done using
+Blake2B up to the relevant difficulty. While doing so is much easier than doing the full RandomX PoW, it still requires
+significant work. It is very important that we can calculate the block ID from the same information (or a subset thereof)
+that we calculate PoW from, since each block header contains the previous block ID. Binding the block headers as such makes
+faking intermediate PoW on chains of blocks exponentially harder the longer the chain is.

docs/resources/cryptonight-data-flow.png

@@ -98,13 +98,16 @@ bool tree_branch_hash(const char hash[HASH_SIZE], const char (*branch)[HASH_SIZE
 bool is_branch_in_tree(const char hash[HASH_SIZE], const char root[HASH_SIZE], const char (*branch)[HASH_SIZE], size_t depth, uint32_t path);
 
 #define RX_BLOCK_VERSION	12
+#define RX_VARIANT_1       1
+#define RX_VARIANT_2       2
 void rx_slow_hash_allocate_state(void);
 void rx_slow_hash_free_state(void);
 uint64_t rx_seedheight(const uint64_t height);
 void rx_seedheights(const uint64_t height, uint64_t *seed_height, uint64_t *next_height);
 
 void rx_set_main_seedhash(const char *seedhash, size_t max_dataset_init_threads);
-void rx_slow_hash(const char *seedhash, const void *data, size_t length, char *result_hash);
+void rx_slow_hash(const char *seedhash, const int variant, const void *data, size_t length, char *result_hash);
+void rx_commitment(const void *data, size_t length, const void *rx_hash, char *result_commitment);
 
 void rx_set_miner_thread(uint32_t value, size_t max_dataset_init_threads);
 uint32_t rx_get_miner_thread(void);

docs/resources/partial-pow-verify-data-flow.png

@@ -66,8 +66,11 @@ static int secondary_seedhash_set = 0;
 #define THREADV __thread
 #endif
 
+static THREADV int main_vm_full_variant = 0;
 static THREADV randomx_vm *main_vm_full = NULL;
+static THREADV int main_vm_light_variant = 0;
 static THREADV randomx_vm *main_vm_light = NULL;
+static THREADV int secondary_vm_light_variant = 0;
 static THREADV randomx_vm *secondary_vm_light = NULL;
 
 static THREADV uint32_t miner_thread = 0;
@@ -86,7 +89,7 @@ static void local_abort(const char *msg)
 #endif
 }
 
-static void hash2hex(const char* hash, char* hex) {
+static void hash2hex(const char hash[HASH_SIZE], char hex[HASH_SIZE * 2 + 1]) {
   const char* d = "0123456789abcdef";
   for (int i = 0; i < HASH_SIZE; ++i) {
     const uint8_t b = hash[i];
@@ -189,7 +192,7 @@ void rx_seedheights(const uint64_t height, uint64_t *seedheight, uint64_t *nexth
   *nextheight = rx_seedheight(height + get_seedhash_epoch_lag());
 }
 
-static void rx_alloc_dataset(randomx_flags flags, randomx_dataset** dataset, int ignore_env)
+static void rx_alloc_dataset(randomx_dataset** dataset, int ignore_env)
 {
   if (*dataset) {
     return;
@@ -213,6 +216,8 @@ static void rx_alloc_dataset(randomx_flags flags, randomx_dataset** dataset, int
     return;
   }
 
+  const randomx_flags flags = enabled_flags();
+
   *dataset = randomx_alloc_dataset((flags | RANDOMX_FLAG_LARGE_PAGES) & ~disabled_flags());
   if (!*dataset) {
     alloc_err_msg("Couldn't allocate RandomX dataset using large pages");
@@ -223,12 +228,14 @@ static void rx_alloc_dataset(randomx_flags flags, randomx_dataset** dataset, int
   }
 }
 
-static void rx_alloc_cache(randomx_flags flags, randomx_cache** cache)
+static void rx_alloc_cache(randomx_cache** cache)
 {
   if (*cache) {
     return;
   }
 
+  const randomx_flags flags = enabled_flags();
+
   *cache = randomx_alloc_cache((flags | RANDOMX_FLAG_LARGE_PAGES) & ~disabled_flags());
   if (!*cache) {
     alloc_err_msg("Couldn't allocate RandomX cache using large pages");
@@ -237,17 +244,22 @@ static void rx_alloc_cache(randomx_flags flags, randomx_cache** cache)
   }
 }
 
-static void rx_init_full_vm(randomx_flags flags, randomx_vm** vm)
+static void rx_init_full_vm(const int variant, randomx_vm** vm, int* vm_variant)
 {
-  if (*vm || !main_dataset || (disabled_flags() & RANDOMX_FLAG_FULL_MEM)) {
+  if (!main_dataset || (disabled_flags() & RANDOMX_FLAG_FULL_MEM)) {
+    return;
+  }
+  else if (*vm && variant == *vm_variant) {
     return;
   }
 
+  randomx_flags flags = enabled_flags();
   if ((flags & RANDOMX_FLAG_JIT) && !miner_thread) {
     flags |= RANDOMX_FLAG_SECURE;
   }
 
   *vm = randomx_create_vm((flags | RANDOMX_FLAG_LARGE_PAGES | RANDOMX_FLAG_FULL_MEM) & ~disabled_flags(), NULL, main_dataset);
+  *vm_variant = variant;
   if (!*vm) {
     static int shown = 0;
     if (!shown) {
@@ -261,20 +273,22 @@ static void rx_init_full_vm(randomx_flags flags, randomx_vm** vm)
   }
 }
 
-static void rx_init_light_vm(randomx_flags flags, randomx_vm** vm, randomx_cache* cache)
+static void rx_init_light_vm(const int variant, randomx_vm** vm, int* vm_variant, randomx_cache* cache)
 {
-  if (*vm) {
+  if (*vm && variant == *vm_variant) {
     randomx_vm_set_cache(*vm, cache);
     return;
   }
 
+  randomx_flags flags = enabled_flags();
   if ((flags & RANDOMX_FLAG_JIT) && !miner_thread) {
     flags |= RANDOMX_FLAG_SECURE;
   }
 
   flags &= ~RANDOMX_FLAG_FULL_MEM;
 
   *vm = randomx_create_vm((flags | RANDOMX_FLAG_LARGE_PAGES) & ~disabled_flags(), cache, NULL);
+  *vm_variant = variant;
   if (!*vm) {
     static int shown = 0;
     if (!shown) {
@@ -367,9 +381,8 @@ static CTHR_THREAD_RTYPE rx_set_main_seedhash_thread(void *arg) {
   hash2hex(main_seedhash, buf);
   minfo(RX_LOGCAT, "RandomX new main seed hash is %s", buf);
 
-  const randomx_flags flags = enabled_flags() & ~disabled_flags();
-  rx_alloc_dataset(flags, &main_dataset, 0);
-  rx_alloc_cache(flags, &main_cache);
+  rx_alloc_dataset(&main_dataset, 0);
+  rx_alloc_cache(&main_cache);
 
   randomx_init_cache(main_cache, info->seedhash, HASH_SIZE);
   minfo(RX_LOGCAT, "RandomX main cache initialized");
@@ -405,8 +418,7 @@ void rx_set_main_seedhash(const char *seedhash, size_t max_dataset_init_threads)
   CTHR_THREAD_CLOSE(t);
 }
 
-void rx_slow_hash(const char *seedhash, const void *data, size_t length, char *result_hash) {
-  const randomx_flags flags = enabled_flags() & ~disabled_flags();
+void rx_slow_hash(const char *seedhash, const int variant, const void *data, size_t length, char *result_hash) {
   int success = 0;
 
   // Fast path (seedhash == main_seedhash)
@@ -416,7 +428,7 @@ void rx_slow_hash(const char *seedhash, const void *data, size_t length, char *r
     if (main_dataset && CTHR_RWLOCK_TRYLOCK_READ(main_dataset_lock)) {
       // Double check that main_seedhash didn't change
       if (is_main(seedhash)) {
-        rx_init_full_vm(flags, &main_vm_full);
+        rx_init_full_vm(variant, &main_vm_full, &main_vm_full_variant);
         if (main_vm_full) {
           randomx_calculate_hash(main_vm_full, data, length, result_hash);
           success = 1;
@@ -427,7 +439,7 @@ void rx_slow_hash(const char *seedhash, const void *data, size_t length, char *r
       CTHR_RWLOCK_LOCK_READ(main_cache_lock);
       // Double check that main_seedhash didn't change
       if (is_main(seedhash)) {
-        rx_init_light_vm(flags, &main_vm_light, main_cache);
+        rx_init_light_vm(variant, &main_vm_light, &main_vm_light_variant, main_cache);
         randomx_calculate_hash(main_vm_light, data, length, result_hash);
         success = 1;
       }
@@ -449,7 +461,7 @@ void rx_slow_hash(const char *seedhash, const void *data, size_t length, char *r
       hash2hex(seedhash, buf);
       minfo(RX_LOGCAT, "RandomX new secondary seed hash is %s", buf);
 
-      rx_alloc_cache(flags, &secondary_cache);
+      rx_alloc_cache(&secondary_cache);
       randomx_init_cache(secondary_cache, seedhash, HASH_SIZE);
       minfo(RX_LOGCAT, "RandomX secondary cache updated");
       memcpy(secondary_seedhash, seedhash, HASH_SIZE);
@@ -460,7 +472,7 @@ void rx_slow_hash(const char *seedhash, const void *data, size_t length, char *r
 
   CTHR_RWLOCK_LOCK_READ(secondary_cache_lock);
   if (is_secondary(seedhash)) {
-    rx_init_light_vm(flags, &secondary_vm_light, secondary_cache);
+    rx_init_light_vm(variant, &secondary_vm_light, &secondary_vm_light_variant, secondary_cache);
     randomx_calculate_hash(secondary_vm_light, data, length, result_hash);
     success = 1;
   }
@@ -482,11 +494,15 @@ void rx_slow_hash(const char *seedhash, const void *data, size_t length, char *r
     memcpy(secondary_seedhash, seedhash, HASH_SIZE);
     secondary_seedhash_set = 1;
   }
-  rx_init_light_vm(flags, &secondary_vm_light, secondary_cache);
+  rx_init_light_vm(variant, &secondary_vm_light, &secondary_vm_light_variant, secondary_cache);
   randomx_calculate_hash(secondary_vm_light, data, length, result_hash);
   CTHR_RWLOCK_UNLOCK_WRITE(secondary_cache_lock);
 }
 
+void rx_commitment(const void *data, size_t length, const void *rx_hash, char *result_commitment) {
+  randomx_calculate_commitment(data, length, rx_hash, result_commitment);
+}
+
 void rx_set_miner_thread(uint32_t value, size_t max_dataset_init_threads) {
   miner_thread = value;
 
@@ -497,8 +513,7 @@ void rx_set_miner_thread(uint32_t value, size_t max_dataset_init_threads) {
     return;
   }
 
-  const randomx_flags flags = enabled_flags() & ~disabled_flags();
-  rx_alloc_dataset(flags, &main_dataset, 1);
+  rx_alloc_dataset(&main_dataset, 1);
   rx_init_dataset(max_dataset_init_threads);
 
   CTHR_RWLOCK_UNLOCK_WRITE(main_dataset_lock);

docs/resources/randomx-commitment-data-flow.png

@@ -1438,7 +1438,8 @@ namespace cryptonote
     blobdata blob = t_serializable_object_to_blob(static_cast<block_header>(b));
     crypto::hash tree_root_hash = get_tx_tree_hash(b);
     blob.append(reinterpret_cast<const char*>(&tree_root_hash), sizeof(tree_root_hash));
-    blob.append(tools::get_varint_data(b.tx_hashes.size()+1));
+    if (b.major_version < HF_VERSION_POW_COMMITMENT)
+      blob.append(tools::get_varint_data(b.tx_hashes.size()+1));
     return blob;
   }
   //---------------------------------------------------------------
@@ -1604,10 +1605,16 @@ namespace cryptonote
     return get_tx_tree_hash(txs_ids);
   }
   //---------------------------------------------------------------
+  int get_randomx_variant_for_hf_version(const std::uint8_t hf_version)
+  {
+    return (hf_version >= HF_VERSION_RANDOMX_V2) ? RX_VARIANT_2 : RX_VARIANT_1;
+  }
+  //---------------------------------------------------------------
   crypto::hash get_block_longhash(const blobdata_ref block_hashing_blob,
     const uint64_t height,
     const uint8_t major_version,
-    const crypto::hash &seed_hash)
+    const crypto::hash &seed_hash,
+    crypto::hash &intermediate_hash_out)
   {
     crypto::hash res;
 
@@ -1618,7 +1625,12 @@ namespace cryptonote
     }
     else if (major_version >= RX_BLOCK_VERSION) // RandomX
     {
-      crypto::rx_slow_hash(seed_hash.data, block_hashing_blob.data(), block_hashing_blob.size(), res.data);
+      const int rx_variant = get_randomx_variant_for_hf_version(major_version);
+      crypto::rx_slow_hash(seed_hash.data, rx_variant,
+        block_hashing_blob.data(), block_hashing_blob.size(), res.data);
+      intermediate_hash_out = res;
+      if (major_version >= HF_VERSION_POW_COMMITMENT)
+        crypto::rx_commitment(block_hashing_blob.data(), block_hashing_blob.size(), res.data, res.data);
     }
     else // CryptoNight
     {

docs/resources/randomx-data-flow.png

@@ -261,10 +261,12 @@ namespace cryptonote
   void get_tx_tree_hash(const std::vector<crypto::hash>& tx_hashes, crypto::hash& h);
   crypto::hash get_tx_tree_hash(const std::vector<crypto::hash>& tx_hashes);
   crypto::hash get_tx_tree_hash(const block& b);
+  int get_randomx_variant_for_hf_version(const std::uint8_t hf_version);
   crypto::hash get_block_longhash(const blobdata_ref block_hashing_blob,
     const uint64_t height,
     const uint8_t major_version,
-    const crypto::hash &seed_hash);
+    const crypto::hash &seed_hash,
+    crypto::hash &intermediate_hash_out);
   bool is_valid_decomposed_amount(uint64_t amount);
   void get_hash_stats(uint64_t &tx_hashes_calculated, uint64_t &tx_hashes_cached, uint64_t &block_hashes_calculated, uint64_t & block_hashes_cached);