Merge dashpay#6017: backport: bitcoin#18836, bitcoin#19046, bitcoin#19490, bitcoin#20403, bitcoin#21127, bitcoin#21238, bitcoin#21329 - descriptor wallets part V & sethdseed

d3ad11d chore: add release notes for sethdseed RPC (Konstantin Akimov)
ad25d54 Merge bitcoin#21329: descriptor wallet: Cache last hardened xpub and use in normalized descriptors (Samuel Dobson)
24b1f6b Merge bitcoin#21238: A few descriptor improvements to prepare for Taproot support (W. J. van der Laan)
14ac2b7 Merge bitcoin#21127: wallet: load flags before everything else (Wladimir J. van der Laan)
19b2b27 Merge bitcoin#20403: wallet: upgradewallet fixes, improvements, test coverage (MarcoFalke)
708586c Merge bitcoin#18836: wallet: upgradewallet fixes and additional tests (Andrew Chow)
752e4ca Merge bitcoin#19490: wallet: Fix typo in comments; Simplify assert (Samuel Dobson)
63895fd Merge bitcoin#19046: Replace CWallet::Set* functions that use memonly with Add/Load variants (Andrew Chow)
2c0d5b7 refactor: rename hdChain to m_hd_chain (Konstantin Akimov)
266aefc feat: sethdseed rpc added. Based on bitcoin#12560 and the newest related changes (Konstantin Akimov)

Pull request description:

  ## Issue being fixed or feature implemented
  Next batch of backports related to descriptor wallets.
  See related issue to track a progress: dashpay/dash-issues#59
  Changes in this PR also used in "hardware signing" feature (coming later)

  ## What was done?
  1. It implements a new rpc `sethdseed` that is based on bitcoin#12560 and newer related changes.
  2. refactoring to rename `hdChain` to `m_hd_chain` (see bitcoin#17681 which is DNM).
  3. Bitcoin backports (some of them uses sethdseed, and requires m_hd_chain to reduce conflicts):
   - bitcoin#19046
   - bitcoin#19490
   - bitcoin#18836
   - bitcoin#20403
   - bitcoin#21127
   - bitcoin#21238
   - bitcoin#21329

  ## How Has This Been Tested?
  Run unit/functional tests.
  The backports bitcoin#18836 and bitcoin#20403 are heavily modified to adopt `wallet_upgradewallet.py` to our codebase.

  ## Breaking Changes
  N/A
  Though, `sethdseed` implementation is not a final version as it is now, can be removed (superseeded by `upgradetohd` or got mnemonic-feature and super-seed `upgradetohd`).

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone

ACKs for top commit:
  PastaPastaPasta:
    utACK d3ad11d

Tree-SHA512: 02182182ec7a5f89eb7d3bc34072d894a86cc89c5eea124e702cc5ed527f76863469b1fd9313b3ea643a8774a358031be927d7b78ec7cd39df0a9ca77559d66d

Author: PastaPastaPasta

doc/release-notes-6017.md

@@ -0,0 +1,3 @@
+RPC changes
+-----------
+- A new `sethdseed` RPC allows users to initialize their blank HD wallets with an HD seed. **A new backup must be made when a new HD seed is set.** This command cannot replace an existing HD seed if one is already set. `sethdseed` uses WIF private key as a seed. If you have a mnemonic, use the `upgradetohd` RPC.

src/qt/rpcconsole.cpp

@@ -70,6 +70,7 @@ namespace {
 const QStringList historyFilter = QStringList()
     << "importprivkey"
     << "importmulti"
+    << "sethdseed"
     << "signmessagewithprivkey"
     << "signrawtransactionwithkey"
     << "upgradetohd"

src/rpc/client.cpp

@@ -45,6 +45,7 @@ static const CRPCConvertParam vRPCConvertParams[] =
     { "sendtoaddress", 7, "conf_target" },
     { "sendtoaddress", 9, "avoid_reuse" },
     { "settxfee", 0, "amount" },
+    { "sethdseed", 0, "newkeypool" },
     { "getreceivedbyaddress", 1, "minconf" },
     { "getreceivedbyaddress", 2, "addlocked" },
     { "getreceivedbylabel", 1, "minconf" },

src/script/descriptor.cpp

@@ -22,6 +22,8 @@ class DescriptorCache {
     std::unordered_map<uint32_t, ExtPubKeyMap> m_derived_xpubs;
     /** Map key expression index -> parent xpub */
     ExtPubKeyMap m_parent_xpubs;
+    /** Map key expression index -> last hardened xpub */
+    ExtPubKeyMap m_last_hardened_xpubs;
 
 public:
     /** Cache a parent xpub
@@ -50,11 +52,30 @@ class DescriptorCache {
      * @param[in] xpub The CExtPubKey to get from cache
      */
     bool GetCachedDerivedExtPubKey(uint32_t key_exp_pos, uint32_t der_index, CExtPubKey& xpub) const;
+    /** Cache a last hardened xpub
+     *
+     * @param[in] key_exp_pos Position of the key expression within the descriptor
+     * @param[in] xpub The CExtPubKey to cache
+     */
+    void CacheLastHardenedExtPubKey(uint32_t key_exp_pos, const CExtPubKey& xpub);
+    /** Retrieve a cached last hardened xpub
+     *
+     * @param[in] key_exp_pos Position of the key expression within the descriptor
+     * @param[in] xpub The CExtPubKey to get from cache
+     */
+    bool GetCachedLastHardenedExtPubKey(uint32_t key_exp_pos, CExtPubKey& xpub) const;
 
     /** Retrieve all cached parent xpubs */
     const ExtPubKeyMap GetCachedParentExtPubKeys() const;
     /** Retrieve all cached derived xpubs */
     const std::unordered_map<uint32_t, ExtPubKeyMap> GetCachedDerivedExtPubKeys() const;
+    /** Retrieve all cached last hardened xpubs */
+    const ExtPubKeyMap GetCachedLastHardenedExtPubKeys() const;
+
+    /** Combine another DescriptorCache into this one.
+     * Returns a cache containing the items from the other cache unknown to current cache
+     */
+    DescriptorCache MergeAndDiff(const DescriptorCache& other);
 };
 
 /** \brief Interface for parsed descriptor objects.
@@ -94,7 +115,7 @@ struct Descriptor {
     virtual bool ToPrivateString(const SigningProvider& provider, std::string& out) const = 0;
 
     /** Convert the descriptor to a normalized string. Normalized descriptors have the xpub at the last hardened step. This fails if the provided provider does not have the private keys to derive that xpub. */
-    virtual bool ToNormalizedString(const SigningProvider& provider, std::string& out, bool priv) const = 0;
+    virtual bool ToNormalizedString(const SigningProvider& provider, std::string& out, const DescriptorCache* cache = nullptr) const = 0;
 
     /** Expand a descriptor at a specified position.
      *

src/script/descriptor.h

@@ -24,7 +24,7 @@ void CheckUnparsable(const std::string& prv, const std::string& pub, const std::
     auto parse_pub = Parse(pub, keys_pub, error);
     BOOST_CHECK_MESSAGE(!parse_priv, prv);
     BOOST_CHECK_MESSAGE(!parse_pub, pub);
-    BOOST_CHECK(error == expected_error);
+    BOOST_CHECK_EQUAL(error, expected_error);
 }
 
 constexpr int DEFAULT = 0;
@@ -115,14 +115,10 @@ void DoCheck(const std::string& prv, const std::string& pub, const std::string&
 
     // Check that private can produce the normalized descriptors
     std::string norm1;
-    BOOST_CHECK(parse_priv->ToNormalizedString(keys_priv, norm1, false));
+    BOOST_CHECK(parse_priv->ToNormalizedString(keys_priv, norm1));
     BOOST_CHECK(EqualDescriptor(norm1, norm_pub));
-    BOOST_CHECK(parse_pub->ToNormalizedString(keys_priv, norm1, false));
+    BOOST_CHECK(parse_pub->ToNormalizedString(keys_priv, norm1));
     BOOST_CHECK(EqualDescriptor(norm1, norm_pub));
-    BOOST_CHECK(parse_priv->ToNormalizedString(keys_priv, norm1, true));
-    BOOST_CHECK(EqualDescriptor(norm1, norm_prv));
-    BOOST_CHECK(parse_pub->ToNormalizedString(keys_priv, norm1, true));
-    BOOST_CHECK(EqualDescriptor(norm1, norm_prv));
 
     // Check whether IsRange on both returns the expected result
     BOOST_CHECK_EQUAL(parse_pub->IsRange(), (flags & RANGE) != 0);
@@ -335,9 +331,8 @@ BOOST_AUTO_TEST_CASE(descriptor_test)
 
     // Check for invalid nesting of structures
     CheckUnparsable("sh(XJvEUEcFWCHCyruc8ZX5exPZaGe4UR7gC5FHrhwPnQGDs1uWCsT2)", "sh(03a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd)", "A function is needed within P2SH"); // P2SH needs a script, not a key
-    CheckUnparsable("sh(sh(pk(XJvEUEcFWCHCyruc8ZX5exPZaGe4UR7gC5FHrhwPnQGDs1uWCsT2)))", "sh(sh(pk(03a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd)))", "Cannot have sh in non-top level"); // Cannot embed P2SH inside P2SH
-    CheckUnparsable("sh(combo(XJvEUEcFWCHCyruc8ZX5exPZaGe4UR7gC5FHrhwPnQGDs1uWCsT2))", "sh(combo(03a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd))", "Cannot have combo in non-top level"); // Old must be top level
-
+    CheckUnparsable("sh(sh(pk(XJvEUEcFWCHCyruc8ZX5exPZaGe4UR7gC5FHrhwPnQGDs1uWCsT2)))", "sh(sh(pk(03a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd)))", "Can only have sh() at top level"); // Cannot embed P2SH inside P2SH
+    CheckUnparsable("sh(combo(XJvEUEcFWCHCyruc8ZX5exPZaGe4UR7gC5FHrhwPnQGDs1uWCsT2))", "sh(combo(03a34b99f22c790c4e36b2b3c2c35a36db06226e41c692fc82b8b56ac1c540c5bd))", "Can only have combo() at top level"); // Old must be top level
     // Checksums
     Check("sh(multi(2,[00000000/111'/222]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc,xprv9uPDJpEQgRQfDcW7BkF7eTya6RPxXeJCqCJGHuCJ4GiRVLzkTXBAJMu2qaMWPrS7AANYqdq6vcBcBUdJCVVFceUvJFjaPdGZ2y9WACViL4L/0))#ggrsrxfy", "sh(multi(2,[00000000/111'/222]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL,xpub68NZiKmJWnxxS6aaHmn81bvJeTESw724CRDs6HbuccFQN9Ku14VQrADWgqbhhTHBaohPX4CjNLf9fq9MYo6oDaPPLPxSb7gwQN3ih19Zm4Y/0))#tjg09x5t", "sh(multi(2,[00000000/111'/222]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc,xprv9uPDJpEQgRQfDcW7BkF7eTya6RPxXeJCqCJGHuCJ4GiRVLzkTXBAJMu2qaMWPrS7AANYqdq6vcBcBUdJCVVFceUvJFjaPdGZ2y9WACViL4L/0))#ggrsrxfy", "sh(multi(2,[00000000/111'/222]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL,xpub68NZiKmJWnxxS6aaHmn81bvJeTESw724CRDs6HbuccFQN9Ku14VQrADWgqbhhTHBaohPX4CjNLf9fq9MYo6oDaPPLPxSb7gwQN3ih19Zm4Y/0))#tjg09x5t", DEFAULT, {{"a91445a9a622a8b0a1269944be477640eedc447bbd8487"}}, OutputType::LEGACY, {{0x8000006FUL,222},{0}});
     Check("sh(multi(2,[00000000/111'/222]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc,xprv9uPDJpEQgRQfDcW7BkF7eTya6RPxXeJCqCJGHuCJ4GiRVLzkTXBAJMu2qaMWPrS7AANYqdq6vcBcBUdJCVVFceUvJFjaPdGZ2y9WACViL4L/0))", "sh(multi(2,[00000000/111'/222]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL,xpub68NZiKmJWnxxS6aaHmn81bvJeTESw724CRDs6HbuccFQN9Ku14VQrADWgqbhhTHBaohPX4CjNLf9fq9MYo6oDaPPLPxSb7gwQN3ih19Zm4Y/0))", "sh(multi(2,[00000000/111'/222]xprvA1RpRA33e1JQ7ifknakTFpgNXPmW2YvmhqLQYMmrj4xJXXWYpDPS3xz7iAxn8L39njGVyuoseXzU6rcxFLJ8HFsTjSyQbLYnMpCqE2VbFWc,xprv9uPDJpEQgRQfDcW7BkF7eTya6RPxXeJCqCJGHuCJ4GiRVLzkTXBAJMu2qaMWPrS7AANYqdq6vcBcBUdJCVVFceUvJFjaPdGZ2y9WACViL4L/0))", "sh(multi(2,[00000000/111'/222]xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL,xpub68NZiKmJWnxxS6aaHmn81bvJeTESw724CRDs6HbuccFQN9Ku14VQrADWgqbhhTHBaohPX4CjNLf9fq9MYo6oDaPPLPxSb7gwQN3ih19Zm4Y/0))", DEFAULT, {{"a91445a9a622a8b0a1269944be477640eedc447bbd8487"}}, OutputType::LEGACY, {{0x8000006FUL,222},{0}});

src/test/descriptor_tests.cpp

@@ -1766,7 +1766,7 @@ static UniValue ProcessDescriptorImport(CWallet * const pwallet, const UniValue&
             if (!w_desc.descriptor->GetOutputType()) {
                 warnings.push_back("Unknown output type, cannot set descriptor to active.");
             } else {
-                pwallet->SetActiveScriptPubKeyMan(spk_manager->GetID(), internal);
+                pwallet->AddActiveScriptPubKeyMan(spk_manager->GetID(), internal);
             }
         }
 
@@ -1972,8 +1972,6 @@ RPCHelpMan listdescriptors()
         throw JSONRPCError(RPC_WALLET_ERROR, "listdescriptors is not available for non-descriptor wallets");
     }
 
-    EnsureWalletIsUnlocked(wallet.get());
-
     LOCK(wallet->cs_wallet);
 
     UniValue descriptors(UniValue::VARR);
@@ -1987,7 +1985,7 @@ RPCHelpMan listdescriptors()
         LOCK(desc_spk_man->cs_desc_man);
         const auto& wallet_descriptor = desc_spk_man->GetWalletDescriptor();
         std::string descriptor;
-        if (!desc_spk_man->GetDescriptorString(descriptor, false)) {
+        if (!desc_spk_man->GetDescriptorString(descriptor)) {
             throw JSONRPCError(RPC_WALLET_ERROR, "Can't get normalized descriptor string.");
         }
         spk.pushKV("desc", descriptor);