Initial batch of remote-signer RPC commands. (squashed cpp-hsmd)

Implemented PassClientHSMFd
Added proxy_handle_get_per_commitment_point.
Added proxy_handle_sign_remote_htlc_tx (server side needed for testing)
Modified proxy_handle_sign_remote_htlc_tx to use original code until server is implemented.
Added proxy_handle_sign_invoice, needs final wiring when server implemented.
Added proxy_handle_channel_update_sig.

Hack to log un-proxied handlers.

Added proxy_handle_get_channel_basepoints.
Added proxy_handle_sign_mutual_close_tx.
Fixed witscript present test wrt NULL witscript instead of NULL ptr.
Improved the PROXY_IMPL warning code.
Added SignCommitmentTx
Reverted handle_sign_withdrawal_tx to PROXY_IMPL_MARSHALED due to problem signing P2SH
Disabled num_output==2 assert in proxy_handle_sign_withdrawal_tx
Made handle_sign_withdrawal_tx remote signing dependent at runtime on P2SH inputs.
Updated NOTES and TODO.
Don't attempt to sign withdraw_tx if close_info present.
Added proxy_handle_cannouncement_sig
Added proxy_handle_sign_node_announcement
Added proxy_handle_sign_penalty_to_us
Added proxy_handle_sign_delayed_payment_to_us
Added proxy_handle_sign_local_htlc_tx
Reconciled similar methods; added SignRemoteHTLCToUs
Remove loops from single input cases..
Added proxy_handle_check_future_secret

Rebased cpp-hsmd-3 onto master after prior PRs merged.

Merged reconcile-api branch

Added new command-line for simple test.

Updated NOTES build instructions.

Added handling for grpc::Status CANCELLED.

Added more running notes.

Factored common transaction fields into Transaction submessage

Log grpc to stderr

Factored into setup_single_input_tx

Added Makefile dependency for the proxy.o on the generated grpc headers.

Wrapped signatures w/ typed messages.

Wrapped API method parameters w/ typed messages.

Added missing SignMessage API call.

Renamed methods to have consistent Sign prefix

Implemented SignChannelUpdate using single channel_update argument.

Made proxy marshal/unmarshal naming consistent.

Adapted to remoteserver API cleanup.

Implemented sign-node-announcement.

API updates.

Added proxy_handle_sign_invoice and proxy_handle_sign_remote_htlc_tx.

Added get-channel-basepoints.

Added get-per-commitment-point.

Added sign-local-htlc-tx.

Added sign-remote-htlc-to-us.

Added sign-delayed-payment-to-us.

Added sign-penalty-to-us.

Added sign-commitment-tx and sign-mutual-close-tx.

Added check-future-secret.

Added sign-message.

Added sign-channel-announcement.

API mods: SpendType, Descriptors, Funding return

Added p2sh-p2wpkh support in funding transactions.

Added unilateral-close-info.

Transposed hsmd changes from c-lightning master.

Removed all private keys/secrets from remote_hsmd.

Amended the GetExtPubKey interface, removed derivation_path.

Added scripts/{run-all-tests,run-one-test}.

Updated dependency instructions and test running instructions.

Skip proxy_handle_pass_client_hsmfd for zero dbid (master, gossipd, connectd).

Made contrib/remote_hsmd/hsm_wire.csv a symbolic link to hsmd/hsm_wire.csv.

Author: ksedgwic

.dir-locals.el

@@ -4,4 +4,10 @@
             (c-basic-offset . 8)
             (tab-width . 8)
             ))
-)
+
+ (c++-mode . ((c-file-style . "linux")
+            (indent-tabs-mode . t)
+            (show-trailing-whitespace . t)
+            (c-basic-offset . 8)
+            (tab-width . 8)
+            )))

Makefile

@@ -222,7 +222,8 @@ ALL_C_SOURCES :=
 ALL_C_HEADERS := header_versions_gen.h version_gen.h
 
 CPPFLAGS += -DBINTOPKGLIBEXECDIR="\"$(shell sh tools/rel.sh $(bindir) $(pkglibexecdir))\""
-CFLAGS = $(CPPFLAGS) $(CWARNFLAGS) $(CDEBUGFLAGS) $(COPTFLAGS) -I $(CCANDIR) $(EXTERNAL_INCLUDE_FLAGS) -I . -I/usr/local/include $(SQLITE3_CFLAGS) $(POSTGRES_INCLUDE) $(FEATURES) $(COVFLAGS) $(DEV_CFLAGS) -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS $(PIE_CFLAGS) $(COMPAT_CFLAGS) -DBUILD_ELEMENTS=1
+CMNFLAGS = $(CDEBUGFLAGS) $(COPTFLAGS) -I $(CCANDIR) $(EXTERNAL_INCLUDE_FLAGS) -I . -I/usr/local/include $(SQLITE3_CFLAGS) $(POSTGRES_INCLUDE) $(FEATURES) $(COVFLAGS) $(DEV_CFLAGS) -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS $(PIE_CFLAGS) $(COMPAT_CFLAGS) -DBUILD_ELEMENTS=1
+CFLAGS = $(CPPFLAGS) $(CWARNFLAGS) $(CMNFLAGS)
 # If CFLAGS is already set in the environment of make (to whatever value, it
 # does not matter) then it would export it to subprocesses with the above value
 # we set, including CWARNFLAGS which by default contains -Wall -Werror. This
@@ -320,6 +321,7 @@ include devtools/Makefile
 include tools/Makefile
 include plugins/Makefile
 include tests/plugins/Makefile
+include contrib/remote_hsmd/Makefile
 ifneq ($(FUZZING),0)
 	include tests/fuzz/Makefile
 endif
@@ -346,7 +348,8 @@ PKGLIBEXEC_PROGRAMS = \
 	       lightningd/lightning_gossipd \
 	       lightningd/lightning_hsmd \
 	       lightningd/lightning_onchaind \
-	       lightningd/lightning_openingd
+	       lightningd/lightning_openingd \
+	       lightningd/remote_hsmd
 
 # Don't delete these intermediaries.
 .PRECIOUS: $(ALL_GEN_HEADERS) $(ALL_GEN_SOURCES)

contrib/pyln-testing/pyln/testing/utils.py

@@ -79,7 +79,7 @@ def env(name, default=None):
 SLOW_MACHINE = env("SLOW_MACHINE", "0") == "1"
 DEPRECATED_APIS = env("DEPRECATED_APIS", "0") == "1"
 TIMEOUT = int(env("TIMEOUT", 180 if SLOW_MACHINE else 60))
-
+SUBDAEMON = env("SUBDAEMON", "")
 
 def wait_for(success, timeout=TIMEOUT):
     start_time = time.time()
@@ -538,6 +538,9 @@ def __init__(self, lightning_dir, bitcoindproxy, port=9735, random_hsm=False, no
             'bitcoin-datadir': lightning_dir,
         }
 
+        if SUBDAEMON:
+            opts['subdaemon'] = SUBDAEMON
+
         for k, v in opts.items():
             self.opts[k] = v
 

contrib/remote_hsmd/.gitignore

@@ -0,0 +1,5 @@
+/remotesigner.grpc.pb.cc
+/remotesigner.grpc.pb.h
+/remotesigner.pb.cc
+/remotesigner.pb.h
+/remotesigner.proto

contrib/remote_hsmd/Makefile

@@ -0,0 +1,111 @@
+#! /usr/bin/make
+
+# Designed to be run at the top
+rmthsmd-wrongdir:
+	$(MAKE) -C ../.. lightningd/hsm-all
+
+default: rmthsmd-all
+
+LIGHTNINGD_RMTHSM_SRC := contrib/remote_hsmd/hsmd.c	\
+	contrib/remote_hsmd/gen_hsm_wire.c \
+	contrib/remote_hsmd/remotesigner.pb.cc \
+	contrib/remote_hsmd/remotesigner.grpc.pb.cc \
+	contrib/remote_hsmd/dump.cc \
+	contrib/remote_hsmd/proxy.cc
+LIGHTNINGD_RMTHSM_HEADERS := contrib/remote_hsmd/gen_hsm_wire.h \
+	contrib/remote_hsmd/remotesigner.pb.h \
+	contrib/remote_hsmd/remotesigner.grpc.pb.h
+LIGHTNINGD_RMTHSM_CCOBJS := $(LIGHTNINGD_RMTHSM_SRC:.cc=.o)
+LIGHTNINGD_RMTHSM_OBJS := $(LIGHTNINGD_RMTHSM_CCOBJS:.c=.o)
+
+HOST_SYSTEM = $(shell uname | cut -f 1 -d_)
+SYSTEM ?= $(HOST_SYSTEM)
+CXX = g++
+CXXFLAGS += -std=c++11 \
+	-I. \
+	-Iccan \
+	-Iexternal/libwally-core/include \
+	-Iexternal/libwally-core/src/secp256k1/include \
+	-g
+PROTOC = protoc
+GRPC_CPP_PLUGIN = grpc_cpp_plugin
+GRPC_CPP_PLUGIN_PATH ?= `which $(GRPC_CPP_PLUGIN)`
+PROTOS_PATH = contrib/remote_hsmd
+
+# Common source we use.
+RMTHSMD_COMMON_OBJS :=				\
+	common/amount.o				\
+	common/bigsize.o			\
+	common/bip32.o				\
+	common/daemon.o				\
+	common/daemon_conn.o			\
+	common/derive_basepoints.o		\
+	common/funding_tx.o			\
+	common/gen_status_wire.o		\
+	common/hash_u5.o			\
+	common/key_derive.o			\
+	common/memleak.o			\
+	common/msg_queue.o			\
+	common/node_id.o			\
+	common/permute_tx.o			\
+	common/status.o				\
+	common/status_wire.o			\
+	common/subdaemon.o			\
+	common/type_to_string.o			\
+	common/utils.o				\
+	common/utxo.o				\
+	common/version.o			\
+	common/withdraw_tx.o
+
+# For checking
+LIGHTNINGD_RMTHSM_ALLSRC_NOGEN := $(filter-out contrib/remote_hsmd/gen_%, $(LIGHTNINGD_RMTHSM_SRC) $(LIGHTNINGD_RMTHSM_SRC))
+LIGHTNINGD_RMTHSM_ALLHEADERS_NOGEN := $(filter-out contrib/remote_hsmd/gen_%, $(LIGHTNINGD_RMTHSM_HEADERS))
+
+$(LIGHTNINGD_RMTHSM_OBJS): $(LIGHTNINGD_HEADERS)
+
+# Make sure these depend on everything.
+ALL_OBJS += $(LIGHTNINGD_RMTHSM_OBJS)
+ALL_PROGRAMS += lightningd/remote_hsmd
+ALL_GEN_HEADERS += contrib/remote_hsmd/gen_hsm_wire.h
+ALL_GEN_HEADERS += contrib/remote_hsmd/remotesigner.pb.h contrib/remote_hsmd/remotesigner.grpc.pb.h
+
+rmthsmd-all: lightningd/remote_hsmd
+
+lightningd/remote_hsmd: $(LIGHTNINGD_RMTHSM_OBJS) $(LIGHTNINGD_LIB_OBJS) $(RMTHSMD_COMMON_OBJS) $(BITCOIN_OBJS) $(WIRE_OBJS)
+
+contrib/remote_hsmd/remotesigner.pb.o: $(ALL_GEN_HEADERS)
+
+contrib/remote_hsmd/remotesigner.grpc.pb.o contrib/remote_hsmd/remotesigner.pb.o contrib/remote_hsmd/proxy.o contrib/remote_hsmd/dump.o : CPPFLAGS +=  $(CMNFLAGS) `pkg-config --cflags protobuf grpc`
+lightningd/remote_hsmd: LDLIBS += -L/usr/local/lib \
+	`pkg-config --libs protobuf grpc++`\
+	-pthread\
+	-Wl,--no-as-needed -lgrpc++_reflection -Wl,--as-needed\
+	-ldl
+lightningd/remote_hsmd: LDLIBS += -lstdc++
+
+contrib/remote_hsmd/gen_hsm_wire.h: $(WIRE_GEN) contrib/remote_hsmd/hsm_wire.csv
+	$(WIRE_GEN) --page header $@ hsm_wire_type < contrib/remote_hsmd/hsm_wire.csv > $@
+
+contrib/remote_hsmd/gen_hsm_wire.c: $(WIRE_GEN) contrib/remote_hsmd/hsm_wire.csv
+	$(WIRE_GEN) --page impl ${@:.c=.h} hsm_wire_type < contrib/remote_hsmd/hsm_wire.csv > $@
+
+check-source: $(LIGHTNINGD_RMTHSM_ALLSRC_NOGEN:%=check-src-include-order/%) $(LIGHTNINGD_RMTHSM_ALLHEADERS_NOGEN:%=check-hdr-include-order/%)
+check-source-bolt: $(LIGHTNINGD_RMTHSM_SRC:%=bolt-check/%)
+
+check-whitespace: $(LIGHTNINGD_RMTHSM_ALLSRC_NOGEN:%=check-whitespace/%) $(LIGHTNINGD_RMTHSM_ALLHEADERS_NOGEN:%=check-whitespace/%)
+
+clean: lightningd/rmthsm-clean
+
+lightningd/rmthsm-clean:
+	$(RM) $(LIGHTNINGD_RMTHSM_OBJS) contrib/remote_hsmd/gen_*
+	$(RM) contrib/remote_hsmd/*.pb.cc contrib/remote_hsmd/*.pb.h
+
+.PRECIOUS: %.grpc.pb.cc
+%.grpc.pb.cc: %.proto
+	$(PROTOC) -I $(PROTOS_PATH) --grpc_out=contrib/remote_hsmd --plugin=protoc-gen-grpc=$(GRPC_CPP_PLUGIN_PATH) $<
+
+.PRECIOUS: %.pb.cc
+%.pb.cc: %.proto
+	$(PROTOC) -I $(PROTOS_PATH) --cpp_out=contrib/remote_hsmd $<
+
+-include contrib/remote_hsmd/test/Makefile

contrib/remote_hsmd/NOTES.md

@@ -0,0 +1,106 @@
+c-lightning
+----------------------------------------------------------------
+
+Setup
+
+    cd contrib/remote_hsmd && \
+    ln -s /path/to/rust-lightning-signer/src/server/remotesigner.proto
+
+Building
+
+    ./configure --enable-developer
+    make
+
+Dependencies
+
+Build libsecp256k1 with `./configure --enable-module-recovery`, see
+https://github.com/golemfactory/golem/issues/2168 for background.
+
+    pip3 install --user base58
+    pip3 install --user bitstring
+    pip3 install --user secp256k1
+
+Run all of the integration tests:
+
+    ./contrib/remote_hsmd/scripts/run-all-tests |& tee log
+    
+Run a single test:
+
+    ./contrib/remote_hsmd/scripts/run-one-test $THETEST |& tee log
+
+Some popular tests:
+
+    # sign-invoice, handle-sign-remote-htlc-tx
+    export THETEST=tests/test_connection.py::test_balance
+    export THETEST=tests/test_pay.py::test_sendpay
+    export THETEST=tests/test_pay.py::test_pay
+
+    # sign-local-htlc-tx
+    export THETEST=tests/test_closing.py::test_onchain_different_fees
+
+    # sign-remote-htlc-to-us
+    export THETEST=tests/test_closing.py::test_onchain_feechange
+    export THETEST=tests/test_closing.py::test_onchain_all_dust
+    export THETEST=tests/test_closing.py::test_permfail_new_commit
+
+    # sign-delayed-payment-to-us
+    export THETEST=tests/test_closing.py::test_onchain_multihtlc_our_unilateral
+    export THETEST=tests/test_closing.py::test_onchain_multihtlc_their_unilateral
+    export THETEST=tests/test_closing.py::test_permfail_htlc_in
+    export THETEST=tests/test_closing.py::test_permfail_htlc_out
+
+    # sign-penalty-to-us
+    export THETEST=tests/test_closing.py::test_penalty_inhtlc
+    export THETEST=tests/test_closing.py::test_penalty_outhtlc
+    export THETEST=tests/test_closing.py::test_closing
+
+    # sign-mutual-close
+    export THETEST=tests/test_closing.py::test_closing
+    
+    # check-future-secret
+    export THETEST=tests/test_connection.py::test_dataloss_protection
+    
+    # sign-message
+    export THETEST=tests/test_misc.py::test_signmessage
+
+    # sign-channel-announcement
+    export THETEST=tests/test_closing.py::test_closing_different_fees
+
+    # P2SH_P2WPKH
+    export THETEST=tests/test_closing.py::test_onchain_first_commit 
+    export THETEST=tests/test_connection.py::test_disconnect_funder 
+    export THETEST=tests/test_connection.py::test_disconnect_fundee 
+    export THETEST=tests/test_connection.py::test_reconnect_signed 
+    export THETEST=tests/test_connection.py::test_reconnect_openingd 
+    export THETEST=tests/test_connection.py::test_shutdown_awaiting_lockin
+    
+    # unilateral_close_info option_static_remotekey
+    export THETEST=tests/test_connection.py::test_fee_limits
+    export THETEST=tests/test_closing.py::test_option_upfront_shutdown_script
+
+rust-lightning-signer
+----------------------------------------------------------------
+
+    cargo run --bin server |& tee log3
+
+
+Signing Formats
+```
+rust-lightning  c-lightning     rust-lightning-signer
+p2pkh                           P2PKH
+p2sh
+p2wpkh          p2wpkh          P2WPKH
+p2shwpkh        p2sh-p2wpkh     P2SH_P2WPKH
+p2wsh
+p2shwsh
+```
+
+
+Failing tests after removing seed from hsmd:
+```
+    export THETEST=tests/test_misc.py::test_blockchaintrack
+    export THETEST=tests/test_misc.py::test_new_node_is_mainnet
+    export THETEST=tests/test_misc.py::test_getsharedsecret
+    export THETEST=tests/test_wallet.py::test_hsm_secret_encryption
+    export THETEST=tests/test_wallet.py::test_hsmtool_secret_decryption
+```

contrib/remote_hsmd/TODO.md

@@ -0,0 +1,44 @@
+
+API Coverage
+----------------------------------------------------------------
+
+## Failing Tests
+
+# Frequently Intermittent
+tests/test_connection.py::test_funding_cancel_race
+tests/test_misc.py::test_bad_onion_immediate_peer
+
+## Proxy Scoreboard
+
+```
+COMPLETE		proxy_stat proxy_handle_ecdh
+COMPLETE		proxy_stat proxy_handle_pass_client_hsmfd
+COMPLETE		proxy_stat proxy_handle_sign_remote_commitment_tx
+COMPLETE		proxy_stat proxy_handle_channel_update_sig
+COMPLETE		proxy_stat proxy_handle_sign_node_announcement
+COMPLETE		proxy_stat proxy_handle_sign_remote_htlc_tx
+COMPLETE		proxy_stat proxy_handle_sign_invoice
+COMPLETE		proxy_stat proxy_handle_get_channel_basepoints
+COMPLETE		proxy_stat proxy_handle_get_per_commitment_point
+COMPLETE		proxy_stat proxy_handle_sign_local_htlc_tx
+COMPLETE		proxy_stat proxy_handle_sign_remote_htlc_to_us
+COMPLETE		proxy_stat proxy_handle_sign_delayed_payment_to_us
+COMPLETE		proxy_stat proxy_handle_sign_penalty_to_us
+COMPLETE		proxy_stat proxy_handle_sign_commitment_tx
+COMPLETE		proxy_stat proxy_handle_sign_mutual_close_tx
+COMPLETE		proxy_stat proxy_handle_check_future_secret
+COMPLETE		proxy_stat proxy_handle_cannouncement_sig
+COMPLETE		proxy_stat proxy_handle_sign_message
+
+PARTIAL (-P2SH)	proxy_stat proxy_handle_sign_withdrawal_tx
+
+MARSHALED		proxy_stat proxy_init_hsm
+```
+
+Improvements
+----------------------------------------------------------------
+
+#### Remove contrib/remote_signer/hsm_wire.csv
+
+Generate gen_hsm_wire.{h,c} from c-lightning/hsmd/hsm_wire.csv instead.
+  

contrib/remote_hsmd/capabilities.h

@@ -0,0 +1,12 @@
+#ifndef LIGHTNING_HSMD_CAPABILITIES_H
+#define LIGHTNING_HSMD_CAPABILITIES_H
+
+#define HSM_CAP_ECDH 1
+#define HSM_CAP_SIGN_GOSSIP 2
+#define HSM_CAP_SIGN_ONCHAIN_TX 4
+#define HSM_CAP_COMMITMENT_POINT 8
+#define HSM_CAP_SIGN_REMOTE_TX 16
+#define HSM_CAP_SIGN_CLOSING_TX 32
+
+#define HSM_CAP_MASTER 1024
+#endif /* LIGHTNING_HSMD_CAPABILITIES_H */