changed configuration setup to blindbit.example.toml style

* fixed bug related to non-eligible taproot spent taproot outputs causing crashes at count_tweak != 1 check
* look out for computational overhead due to constant checks on existing utxos (alternative is to store all spent and periodically prune; and for initial sync after full sync is done)

Author: setavenger

.gitignore

@@ -15,3 +15,6 @@ docker-compose.yaml
 /mainnet-test-1/
 /_prod_/
 /regtest-2/
+/bin/
+/signet-1/
+/dev/

README.md

@@ -2,7 +2,8 @@
 
 A GO implementation for a BIP0352 Silent Payments Indexing Server.
 This backend was focused on serving the BlindBit light clients with tweak data and other simplified data to spend and
-receive.
+receive. The produced index matches that
+of [other implementations](https://github.com/bitcoin/bitcoin/pull/28241#issuecomment-2079270744).
 
 ## Setup
 
@@ -55,7 +56,11 @@ Once the ENV variables are set you can just run the binary.
 
 ## Todos
 
-- [ ] Convert env vars into command line args
+- [ ] Add flags to control setup
+  - reindex
+  - headers only
+  - tweaks only
+  - move most env controls to config file or cli flags/args
 - [ ] Include [gobip352 module](https://github.com/setavenger/gobip352)
 - [ ] Refactor a bunch of stuff to use byte arrays or slices instead of strings for internal uses
     - Could potentially reduce the serialisation overhead
@@ -64,28 +69,11 @@ Once the ENV variables are set you can just run the binary.
 - [ ] Clean up code (bytes.Equal, parity on big.int with .Bit(), etc.)
 - [ ] Update to new test vectors
 - [ ] Write operation tests to ensure data integrity
-- [ ] Benchmark btcec vs libsecp C library wrapper/binding
-    - probably need to create my own wrapper
-    - https://github.com/renproject/secp256k1
-    - https://github.com/ethereum/go-ethereum/tree/master/crypto/secp256k1/libsecp256k1 (~8 years without update)
 - [ ] Periodically recompute filters
     - One could implement a periodic re-computation every 144 blocks of filters with the current UTXO set
-- [ ] Investigate whether we should change the compound keys to use the height instead of the hash. As keys are sorted
-  this could potentially give a performance boost due to better order across blocks.
-    - ON_HOLD: Not reorg resistant unless some extra work and checks are made
 - [ ] Document EVERYTHING: especially serialization patterns to easily look them up later.
     - Serialisation
     - tweak computation methods
-    - ...
-- [x] Redo the storage system. After syncing approximately 5,500 blocks, the estimated storage at 100,000 blocks for
-  tweaks alone will be somewhere around 40Gb. Additionally, performance is getting worse.
-    - Done: Switched to LevelDB see here
-      for [current numbers](https://github.com/setavenger/BIP0352-light-client-specification)
-- [x] Investigate whether RPC parallel calls can speed up syncing. Caution: currently the flow is synchronous and hence
-  there is less complexity. Making parallel calls will change that.
-    - note: This was mainly limited by a slow home node. First tests a more performant node show that this is not as big
-      as a problem. Also using parallel calls on a weak node just increases the latency for every individual call
-      reducing most of the gains from parallel calls.
 - [ ] Include redundancy for when RPC calls are failing (probably due to networking issues in a testing home
   environment).
 - [ ] Review all duplicate key error exemptions and raise to error/warn from debug.
@@ -94,6 +82,10 @@ Once the ENV variables are set you can just run the binary.
 - [ ] Convert hardcoded serialisation assertions into constants (?)
 - [ ] Use x-only 32 byte public keys instead of scriptPubKey
 
+### Low Priority
+- [ ] Index the next couple blocks in mempool
+  - Every 1-3 minute or so?
+
 ## Endpoints
 
 ```text

blindbit.example.toml

@@ -0,0 +1,27 @@
+# default "./.blindbit-oracle"
+base_directory="./blindbit-oracle"
+
+# 0.0.0.0:8000 to expose outside of localhost
+host="127.0.0.1:8000"
+
+# (defaults to http://127.0.0.1:8332)
+rpc_endpoint="http://127.0.0.1:18443"
+
+rpc_pass="your-rpc-password"
+
+rpc_user="your-rpc-user"
+
+# (has to be >= 1)
+sync_start_height=1
+
+# the default for this is 1, but should be set to a higher value to increase performance, 
+# one should set this in accordance to how many cores one wants to use
+max_parallel_tweak_computations=4
+
+# (depends on max-rpc-workers of the underlying full node)
+max_parallel_requests=4
+
+# 
+# optional - will only generate tweaks
+# default: 0 set to 1 to activate
+tweaks_only=1

go.mod

@@ -1,6 +1,6 @@
 module SilentPaymentAppBackend
 
-go 1.18
+go 1.20
 
 require (
 	github.com/btcsuite/btcd/btcec/v2 v2.2.0
@@ -9,6 +9,7 @@ require (
 	github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d
 	github.com/gin-gonic/gin v1.9.1
 	github.com/shopspring/decimal v1.3.1
+	github.com/spf13/viper v1.18.2
 	github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7
 	golang.org/x/crypto v0.17.0
 )
@@ -21,31 +22,43 @@ require (
 	github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
 	github.com/decred/dcrd/crypto/blake256 v1.0.0 // indirect
 	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.0.1 // indirect
-	github.com/fsnotify/fsnotify v1.6.0 // indirect
+	github.com/fsnotify/fsnotify v1.7.0 // indirect
 	github.com/gabriel-vasile/mimetype v1.4.2 // indirect
 	github.com/gin-contrib/sse v0.1.0 // indirect
 	github.com/go-playground/locales v0.14.1 // indirect
 	github.com/go-playground/universal-translator v0.18.1 // indirect
 	github.com/go-playground/validator/v10 v10.14.0 // indirect
 	github.com/goccy/go-json v0.10.2 // indirect
 	github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb // indirect
-	github.com/google/go-cmp v0.5.9 // indirect
+	github.com/hashicorp/hcl v1.0.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23 // indirect
 	github.com/klauspost/cpuid/v2 v2.2.4 // indirect
 	github.com/leodido/go-urn v1.2.4 // indirect
+	github.com/magiconair/properties v1.8.7 // indirect
 	github.com/mattn/go-isatty v0.0.19 // indirect
+	github.com/mitchellh/mapstructure v1.5.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
-	github.com/pelletier/go-toml/v2 v2.0.8 // indirect
-	github.com/stretchr/testify v1.8.4 // indirect
+	github.com/pelletier/go-toml/v2 v2.1.0 // indirect
+	github.com/sagikazarmark/locafero v0.4.0 // indirect
+	github.com/sagikazarmark/slog-shim v0.1.0 // indirect
+	github.com/sourcegraph/conc v0.3.0 // indirect
+	github.com/spf13/afero v1.11.0 // indirect
+	github.com/spf13/cast v1.6.0 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/subosito/gotenv v1.6.0 // indirect
 	github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
 	github.com/ugorji/go/codec v1.2.11 // indirect
+	go.uber.org/atomic v1.9.0 // indirect
+	go.uber.org/multierr v1.9.0 // indirect
 	golang.org/x/arch v0.3.0 // indirect
-	golang.org/x/net v0.18.0 // indirect
+	golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
+	golang.org/x/net v0.19.0 // indirect
 	golang.org/x/sys v0.16.0 // indirect
 	golang.org/x/text v0.14.0 // indirect
-	google.golang.org/protobuf v1.30.0 // indirect
+	google.golang.org/protobuf v1.31.0 // indirect
+	gopkg.in/ini.v1 v1.67.0 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )