Maintain an auxillary state locally to reduce the bandwidth

[adithyaov]

Closes: #122

• Cleanup
• Test

[axman6]

Looks interesting, I hadn't considered this approach. Just making sure I understand, we're basically keeping the UTxO map in memory, but filtered to only scripts? Any idea what effect this has on memory? I imagine it's not too bad (and if we really need to bring memory usage down, we could specialise things).

Does this handle the case where a script is added to the ledger after the app starts? That will be the common case when starting from genesis. I couldn't quite follow how the scan was working.

I assume the big pay off here is that we now know when a script uses a script that we care about so only need to request UTxO state when we see that, and not for every block?

[adithyaov]

we're basically keeping the UTxO map in memory, but filtered to only scripts? Any idea what effect this has on memory? I imagine it's not too bad (and if we really need to bring memory usage down, we could specialise things).

Yes, we start with some valid utxo map and incrementally build it while going through blocks.

and if we really need to bring memory usage down, we could specialise things

I'm not sure what this means. Do you mean specialise a function?

Does this handle the case where a script is added to the ledger after the app starts? That will be the common case when starting from genesis. I couldn't quite follow how the scan was working.

Yes. TLDR:

Invariant on the local UTxO map: It only contains the UTxOs with script addressess that have an intersection with the configured scripts.

The PSR can start at any chainpoint. When it start we create the local UTxO map.
While consuming transactions in a block we modify the local UTxO map while maintaining these invariants.

I assume the big pay off here is that we now know when a script uses a script that we care about so only need to request UTxO state when we see that, and not for every block?

Yes. 2 big pay offs:

1. Skip blocks without interesting transactions.
2. The filter that queries the UTxO state is more agressive - we only query the inputs of transactions that are interesting.

[adithyaov]

Will we use this?
We can remove this now and maybe add this on demand,

[tweag-ev-ak]

I think we can keep it

[tweag-ev-ak]

LGTM 👍

[axman6]

LGTM! Great work, this should really improve how often we query the node.