Add ZMQ RPC function 'GetTreePaths' for LWS wallets

[vtnerd]

This adds (controversially) a ZeroMQ RPC for LWS wallets to build/construct a proper fcmp++ tree proof. The RPC takes a list of {amount, amount_index} and returns information to init + build a tree cache for those outputs. I have verified locally that the transactions constructed from this RPC are accepted by a daemon.

This new RPC leaks outputs from a wallet to the daemon, which is roughly the same privacy of existing LWS wallets. The only difference is that currently the decoy ring outputs are leaked to the daemon, whereas this leaks the spent outputs directly.

@j-berman will the {amount, amount_index} DB calls be deprecated at some point? I can modify the RPC to accept either the old or "new" style output indexing, using the old style means that LWS databases don't require a re-scan to work.

[jeffro256]

You might be interested in #89 which allows for a much more compact response when dealing with multiple paths at once.

[vtnerd]

I'll wait until that finalizes before incorporating it into this patch. Although its been sitting a while ...

[vtnerd]

@j-berman in #89 showed desired to slowly deprecate {amount, amount_index} format for indexing outputs. I'll have to think about this for that review, as LWS will need a transition phase. The lwsf <-> lws communication will likely support both formats ({amount, amount_index} and {global_index}) whereas the lws <-> monerod will only support {global_index}.

[j-berman]

@vtnerd personally I think it would be nice to eventually deprecate the amount-style indexes for FCMP++ / Carrot outputs, since they're definitely no longer relevant at that point and would just be debt. But we will need them into perpetuity for pre-FCMP++/Carrot outputs in order to validate legacy ring signatures.

That might lead to a frankenstein impl that supports the amount-style indexes for just legacy outputs over an RPC, and global for future. But we can implement an "optimal" solution from a space-saving perspective where lws still wouldn't need a rescan.

I'm open to whatever.

[vtnerd]

@j-berman my current strategy is to change zmq/get_tree_paths to take only the "new" global indexes. A second rpc will provide remappings from old to new, so legacy outputs will require an extra round trip.

Then, the zmq/get_blocks_fast will switch it's output indexing after the next hardfork. Lws will therefore only store new indexes after the fork, but will continue to store legacy indexes up until the fork because they are required for ring signatures.

Lws servers that don't update for the hardfork will eventually see an empty array for output indexes, and will error out until the user upgrades lws. And they'll probably trigger an unknown output type (is_carrot) before this anyway.

The frontend project will have a new boolean sent from server to indicate whether it's using new indexing. If the user doesn't upgrade their wallet they can't send any transactions anyway so this part is easy.

[j-berman]

That sounds good to me

[vtnerd]

Ran into a snag, just need to confirm something. There doesn't appear to be way to directly get "unified" ids from a tx hash, only "legacy" ids. I really don't want to make a DB change just for LWS/ZMQ-RPC, but I think I've found a suitably quick way to get the unified IDs.

@j-berman what I need is confirmation that for post-ringct blocks, every unified id will be offset by the same value from legacy id values. In other words, legacy->unified mappings are only complicated for pre-ringct outputs, whereas post-ringct its just a fixed adjustment.

[jeffro256]

@vtnerd this assumption isn't true at the moment since pre-RingCT txs are still allowed on the chain in rare scenarios (unmixable sweep txs). The FCMP++ v17 fork should disallow them forever, at which point, I think that your assumption will be true.

[vtnerd]

Good to know. That's all I really needed to make the protocol work, pre fcmp++ blocks must use the legacy IDs anyway.

[jeffro256]

That's a clever trick, I hadn't thought of that ;)

[vtnerd]

Force pushed a change:

• A ZMQ-RPC get_tree_paths was added, that takes unified IDs and returns full paths. It also returns info to init a new tree at the current top height
• A ZMQ-RPC get_unified_ids that can be used to convert legacy ids to unified ids.
• The existing ZMQ-RPC for get_blocks_fast now has a field unified_ids that is used automatically when the first block in the response is a fcmp++ block. Downstream clients will be forced to upgrade shortly after the fork.

These are somewhat specific for LWS usage, however no DB changes were needed to make it happen.