Merge dashpay#5981: backport: bitcoin#19136, bitcoin#21063, bitcoin#21277, bitcoin#21302, partial bitcoin#20267 - descriptor wallets part IV

ceefab5 fix: feature_backwards compatible works now with as expected if no bdb compiled (Konstantin Akimov)
b20f812 fix: follow-up fixes for functional tests used protx (Konstantin Akimov)
655146d Merge bitcoin#21302: wallet: createwallet examples for descriptor wallets (W. J. van der Laan)
99a8b60 Merge bitcoin#21063: wallet, rpc: update listdescriptors response format (fanquake)
6ee2c7c Merge bitcoin#21277: wallet: listdescriptors uses normalized descriptor form (Wladimir J. van der Laan)
8bacdbf Merge bitcoin#19136: wallet: add parent_desc to getaddressinfo (Samuel Dobson)
f567de0 chore: release notes for 5965 with wallet tool improvements (Konstantin Akimov)
0daf360 chore: add TODO to implement mnemonic for descriptor wallets (Konstantin Akimov)
5016294 chore: move functional test wallet_multiwallet from category "slow 5 minutes" to "fast test" (Konstantin Akimov)
ef7ce87 fix: remove workarounds introduced due to missing bitcoin#20267 (bdb is not compiled) (Konstantin Akimov)
06b2d85 partial Merge bitcoin#20267: Disable and fix tests for when BDB is not compiled (Wladimir J. van der Laan)

Pull request description:

  ## Issue being fixed or feature implemented
  dashpay/dash-issues#59

  ## Extra notes
  This commit `chore: move functional test wallet_multiwallet from category "slow 5 minutes" to "fast test"` is not directly connected to descriptor wallets, but added to this PR due to conflicts with 20267

  ## What was done?
  It steadily improves support of descriptor wallets in Dash core.

  Done backports and related fixes:
   - partial bitcoin#20267
   - bitcoin#19136
   - bitcoin#21277
   - bitcoin#21063
   - bitcoin#21302

  Beside backports and related fixes, this PR includes release notes for previous batch of backports for descriptor wallets support dashpay#5965

  ## How Has This Been Tested?
  Run unit functional tests

  ## Breaking Changes
  N/A

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone

Top commit has no ACKs.

Tree-SHA512: f4b2033f8c4fa1d0f72cfc31378909703b3ae8f44748989ff00c3e71311ac80ac37837137133c7e4a166823a941ed7df10efa09c89f5b213f3c8ede7d3d6e8f4

Author: PastaPastaPasta

doc/release-notes-5965.md

@@ -0,0 +1,12 @@
+## Wallet Tool Enhancements
+
+This release introduces several improvements and new features to the `dash-wallet` tool, making it more versatile and user-friendly for managing Dash wallets.
+
+### Wallet Version Bump
+
+Wallets created with the `dash-wallet` tool will now utilize the `FEATURE_LATEST` version of wallet which is the HD (Hierarchical Deterministic) wallets with HD chain inside.
+
+### New functionality
+- new command line argument `-descriptors` to enable _experimental_ support of Descriptor wallets. It lets users to create descriptor wallets directly from the command line. This change aims to align the command-line interface with the `createwallet` RPC, promoting the use of descriptor wallets which offer a more robust and flexible way to manage wallet addresses and keys.
+- new command line argument `-usehd` which let to create non-Hierarchical Deterministic (non-HD) wallets with the `wallettool` for compatibility reasons since default version is bumped to HD version
+- new commands `dump` and `createfromdump` have been added, enhancing the wallet's storage migration capabilities. The `dump` command allows for exporting every key-value pair from the wallet as comma-separated hex values, facilitating a storage agnostic dump. Meanwhile, the `createfromdump` command enables the creation of a new wallet file using the records specified in a dump file. These commands are similar to BDB's `db_dump` and `db_load` tools and are crucial for manual wallet file construction for testing or migration purposes.

src/rpc/evo.cpp

@@ -777,6 +777,7 @@ static UniValue protx_register_common_wrapper(const JSONRPCRequest& request,
                 return ret;
             } else {
                 // lets prove we own the collateral
+                // TODO: make collateral works with Descriptor wallets too
                 const LegacyScriptPubKeyMan* spk_man = wallet->GetLegacyScriptPubKeyMan();
                 if (!spk_man) {
                     throw JSONRPCError(RPC_WALLET_ERROR, "This type of wallet does not support this command");

src/rpc/util.cpp

@@ -158,18 +158,81 @@ bool ParseBoolV(const UniValue& v, const std::string &strName)
     throw JSONRPCError(RPC_INVALID_PARAMETER, strName+" must be true, false, yes, no, 1 or 0 (not '"+strBool+"')");
 }
 
+namespace {
+
+/**
+ * Quote an argument for shell.
+ *
+ * @note This is intended for help, not for security-sensitive purposes.
+ */
+std::string ShellQuote(const std::string& s)
+{
+    std::string result;
+    result.reserve(s.size() * 2);
+    for (const char ch: s) {
+        if (ch == '\'') {
+            result += "'\''";
+        } else {
+            result += ch;
+        }
+    }
+    return "'" + result + "'";
+}
+
+/**
+ * Shell-quotes the argument if it needs quoting, else returns it literally, to save typing.
+ *
+ * @note This is intended for help, not for security-sensitive purposes.
+ */
+std::string ShellQuoteIfNeeded(const std::string& s)
+{
+    for (const char ch: s) {
+        if (ch == ' ' || ch == '\'' || ch == '"') {
+            return ShellQuote(s);
+        }
+    }
+
+    return s;
+}
+
+}
+
 std::string HelpExampleCli(const std::string& methodname, const std::string& args)
 {
     return "> dash-cli " + methodname + " " + args + "\n";
 }
 
+std::string HelpExampleCliNamed(const std::string& methodname, const RPCArgList& args)
+{
+    std::string result = "> dash-cli -named " + methodname;
+    for (const auto& argpair: args) {
+        const auto& value = argpair.second.isStr()
+                ? argpair.second.get_str()
+                : argpair.second.write();
+        result += " " + argpair.first + "=" + ShellQuoteIfNeeded(value);
+    }
+    result += "\n";
+    return result;
+}
+
 std::string HelpExampleRpc(const std::string& methodname, const std::string& args)
 {
     return "> curl --user myusername --data-binary '{\"jsonrpc\": \"1.0\", \"id\": \"curltest\", "
         "\"method\": \"" + methodname + "\", \"params\": [" + args + "]}' -H 'content-type: text/plain;'"
         " http://127.0.0.1:9998/\n";
 }
 
+std::string HelpExampleRpcNamed(const std::string& methodname, const RPCArgList& args)
+{
+    UniValue params(UniValue::VOBJ);
+    for (const auto& param: args) {
+        params.pushKV(param.first, param.second);
+    }
+
+    return "> curl --user myusername --data-binary '{\"jsonrpc\": \"1.0\", \"id\": \"curltest\", "
+           "\"method\": \"" + methodname + "\", \"params\": " + params.write() + "}' -H 'content-type: text/plain;' http://127.0.0.1:8332/\n";
+}
+
 // Converts a hex string to a public key if possible
 CPubKey HexToPubKey(const std::string& hex_in)
 {

src/rpc/util.h

@@ -85,8 +85,12 @@ extern double ParseDoubleV(const UniValue& v, const std::string &strName);
 extern bool ParseBoolV(const UniValue& v, const std::string &strName);
 
 extern CAmount AmountFromValue(const UniValue& value);
+
+using RPCArgList = std::vector<std::pair<std::string, UniValue>>;
 extern std::string HelpExampleCli(const std::string& methodname, const std::string& args);
+extern std::string HelpExampleCliNamed(const std::string& methodname, const RPCArgList& args);
 extern std::string HelpExampleRpc(const std::string& methodname, const std::string& args);
+extern std::string HelpExampleRpcNamed(const std::string& methodname, const RPCArgList& args);
 
 CPubKey HexToPubKey(const std::string& hex_in);
 CPubKey AddrToPubKey(const FillableSigningProvider& keystore, const std::string& addr_in);

src/script/descriptor.cpp

@@ -180,6 +180,9 @@ struct PubkeyProvider
     /** Get the descriptor string form including private data (if available in arg). */
     virtual bool ToPrivateString(const SigningProvider& arg, std::string& out) const = 0;
 
+    /** Get the descriptor string form with the xpub at the last hardened derivation */
+    virtual bool ToNormalizedString(const SigningProvider& arg, std::string& out, bool priv) const = 0;
+
     /** Derive a private key, if private data is available in arg. */
     virtual bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const = 0;
 };
@@ -213,6 +216,21 @@ class OriginPubkeyProvider final : public PubkeyProvider
         ret = "[" + OriginString() + "]" + std::move(sub);
         return true;
     }
+    bool ToNormalizedString(const SigningProvider& arg, std::string& ret, bool priv) const override
+    {
+        std::string sub;
+        if (!m_provider->ToNormalizedString(arg, sub, priv)) return false;
+        // If m_provider is a BIP32PubkeyProvider, we may get a string formatted like a OriginPubkeyProvider
+        // In that case, we need to strip out the leading square bracket and fingerprint from the substring,
+        // and append that to our own origin string.
+        if (sub[0] == '[') {
+            sub = sub.substr(9);
+            ret = "[" + OriginString() + std::move(sub);
+        } else {
+            ret = "[" + OriginString() + "]" + std::move(sub);
+        }
+        return true;
+    }
     bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const override
     {
         return m_provider->GetPrivKey(pos, arg, key);
@@ -244,6 +262,12 @@ class ConstPubkeyProvider final : public PubkeyProvider
         ret = EncodeSecret(key);
         return true;
     }
+    bool ToNormalizedString(const SigningProvider& arg, std::string& ret, bool priv) const override
+    {
+        if (priv) return ToPrivateString(arg, ret);
+        ret = ToString();
+        return true;
+    }
     bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const override
     {
         return arg.GetKey(m_pubkey.GetID(), key);
@@ -387,6 +411,56 @@ class BIP32PubkeyProvider final : public PubkeyProvider
         }
         return true;
     }
+    bool ToNormalizedString(const SigningProvider& arg, std::string& out, bool priv) const override
+    {
+        // For hardened derivation type, just return the typical string, nothing to normalize
+        if (m_derive == DeriveType::HARDENED) {
+            if (priv) return ToPrivateString(arg, out);
+            out = ToString();
+            return true;
+        }
+        // Step backwards to find the last hardened step in the path
+        int i = (int)m_path.size() - 1;
+        for (; i >= 0; --i) {
+            if (m_path.at(i) >> 31) {
+                break;
+            }
+        }
+        // Either no derivation or all unhardened derivation
+        if (i == -1) {
+            if (priv) return ToPrivateString(arg, out);
+            out = ToString();
+            return true;
+        }
+        // Derive the xpub at the last hardened step
+        CExtKey xprv;
+        if (!GetExtKey(arg, xprv)) return false;
+        KeyOriginInfo origin;
+        int k = 0;
+        for (; k <= i; ++k) {
+            // Derive
+            xprv.Derive(xprv, m_path.at(k));
+            // Add to the path
+            origin.path.push_back(m_path.at(k));
+            // First derivation element, get the fingerprint for origin
+            if (k == 0) {
+                std::copy(xprv.vchFingerprint, xprv.vchFingerprint + 4, origin.fingerprint);
+            }
+        }
+        // Build the remaining path
+        KeyPath end_path;
+        for (; k < (int)m_path.size(); ++k) {
+            end_path.push_back(m_path.at(k));
+        }
+        // Build the string
+        std::string origin_str = HexStr(origin.fingerprint) + FormatHDKeypath(origin.path);
+        out = "[" + origin_str + "]" + (priv ? EncodeExtKey(xprv) : EncodeExtPubKey(xprv.Neuter())) + FormatHDKeypath(end_path);
+        if (IsRange()) {
+            out += "/*";
+            assert(m_derive == DeriveType::UNHARDENED);
+        }
+        return true;
+    }
     bool GetPrivKey(int pos, const SigningProvider& arg, CKey& key) const override
     {
         CExtKey extkey;
@@ -450,15 +524,17 @@ class DescriptorImpl : public Descriptor
         return false;
     }
 
-    bool ToStringHelper(const SigningProvider* arg, std::string& out, bool priv) const
+    bool ToStringHelper(const SigningProvider* arg, std::string& out, bool priv, bool normalized) const
     {
         std::string extra = ToStringExtra();
         size_t pos = extra.size() > 0 ? 1 : 0;
         std::string ret = m_name + "(" + extra;
         for (const auto& pubkey : m_pubkey_args) {
             if (pos++) ret += ",";
             std::string tmp;
-            if (priv) {
+            if (normalized) {
+                if (!pubkey->ToNormalizedString(*arg, tmp, priv)) return false;
+            } else if (priv) {
                 if (!pubkey->ToPrivateString(*arg, tmp)) return false;
             } else {
                 tmp = pubkey->ToString();
@@ -468,7 +544,7 @@ class DescriptorImpl : public Descriptor
         if (m_subdescriptor_arg) {
             if (pos++) ret += ",";
             std::string tmp;
-            if (!m_subdescriptor_arg->ToStringHelper(arg, tmp, priv)) return false;
+            if (!m_subdescriptor_arg->ToStringHelper(arg, tmp, priv, normalized)) return false;
             ret += std::move(tmp);
         }
         out = std::move(ret) + ")";
@@ -478,13 +554,20 @@ class DescriptorImpl : public Descriptor
     std::string ToString() const final
     {
         std::string ret;
-        ToStringHelper(nullptr, ret, false);
+        ToStringHelper(nullptr, ret, false, false);
         return AddChecksum(ret);
     }
 
     bool ToPrivateString(const SigningProvider& arg, std::string& out) const override final
     {
-        bool ret = ToStringHelper(&arg, out, true);
+        bool ret = ToStringHelper(&arg, out, true, false);
+        out = AddChecksum(out);
+        return ret;
+    }
+
+    bool ToNormalizedString(const SigningProvider& arg, std::string& out, bool priv) const override final
+    {
+        bool ret = ToStringHelper(&arg, out, priv, true);
         out = AddChecksum(out);
         return ret;
     }

src/script/descriptor.h

@@ -93,6 +93,9 @@ struct Descriptor {
     /** Convert the descriptor to a private string. This fails if the provided provider does not have the relevant private keys. */
     virtual bool ToPrivateString(const SigningProvider& provider, std::string& out) const = 0;
 
+    /** Convert the descriptor to a normalized string. Normalized descriptors have the xpub at the last hardened step. This fails if the provided provider does not have the private keys to derive that xpub. */
+    virtual bool ToNormalizedString(const SigningProvider& provider, std::string& out, bool priv) const = 0;
+
     /** Expand a descriptor at a specified position.
      *
      * @param[in] pos The position at which to expand the descriptor. If IsRange() is false, this is ignored.