Support custom filters for transactions to be indexed

[CDDelta]

Just bringing attention to this issue I filed in the old repo here, pasted below as well.

To reduce database storage requirements, I would like to be able to choose which transactions I want to index in the database based on various attributes such as tags, owner, and the recipient (for AR transfers).

Considering how complex the filtering logic could be I wonder what kind of format it should be defined in. Perhaps allow a custom JS function to be defined in some kind of config script? Assuming this will be a common use case for developers, it would be ideal not to need to fork the repo to implement our own filters.

This will make it much easier/faster to bootstrap application-specific gateways and also reduce the hardware specs required to do so.

[CDDelta]

I think the way to approach this would probably be a JS/TS file that can be mounted into the gateway source directory via Docker (or something else assuming vartex moves off of Docker)

[joshbenaron]

To further add - I'm not sure if this is a feature yet but also set a minimum block height to sync from. So if a new company starts now they can run a gateway that only syncs from now as they know that's when their data started going into the weave

[hlolli]

This could be some sort of add-on. It may be unlikely that a low-spec computer can keep itself in sync with arweave, but a cli package pushed to npm could be helpful to simplify. And potentially later on pick which parts to elide from the sync, I think some sort of pick and choose is important, for caching more data for better response time as well as storing less data or ignoring certain table columns. That's less of a priority right now as we need fully synced and fully featured gateways live asap.

[CDDelta]

@hlolli could you elaborate on why it would be hard for a low-spec computer to keep up with the head of the chain?

My organisation is looking to run gateways on many machines owned by clients. The expectation is that the gateway instances running on these machines will only serve txs from the client's specific wallet.

Is the performance issue with verifying each block of the chain or perhaps something else?