Query transactions from era files on a pruned node

[s1na]

Let's say I prune all premerge blocks from my node. We have this feature that if I simply drop an era1 file into the right corner of the datadir, then I can read the corresponding headers and bodies via rpc as if my node was never pruned.

However reading transactions which exist within that segment, by their hash, does not work. That is because once we prune the transaction index, we never re-create it from the era1 files. This affects both eth_getTransactionByHash and eth_getTransactionReceipt.
This is a complicated task because the transaction indexer cannot have gaps in its index right now.

The same applies to logs. eth_getLogs returns an error if query goes beyond the cutoff point regardless of the era files existing for it.

[MariusVanDerWijden]

Just for my understanding, it would be possible to recreate the index from the files though, right?
We would just need to go through all files once

[s1na]

Just for my understanding, it would be possible to recreate the index from the files though, right?
We would just need to go through all files once

In theory yes. But our setup is now as follows: on a pruned node, users can drop in whichever era files they need in a directory and geth will already be able to answer block-level queries for them. Without any import process.

We could potentially watch the directory explicitly and if a file is added then index the transactions. But it can mean that the era files are very sparse. It means that the txindexer should be able to support gaps in the beginning and middle of the chain. That is why I said the complexity can be a bit higher.

In general we could also say if you want to use the tx or log history you'd have to import it but that will be a bit inconsistent with block history which will be available simply after dropping in files.

[s1na]

Update: while pruned nodes can't serve transaction history which was pruned, a fresh node that uses import-history command to import blocks from era will eventually generate the tx index. I.e. after era files were imported, the node will start a snap sync to fetch the state. Once that is done, the tx indexer will become activated and generate the index.

So the biggest unsupported use-case is still the pruned node.

[0xjvn]

Is there any one era file remotely available for download or import ? , will be useful for testing

[s1na]

@the-shadow-coder Yes https://ethpandaops.io/data/history/

[0xjvn]

Ok I get it now , ImportHistory is strictly sequential and requires starting from genesis, so for a fresh node, as long as we have a sequence of Era files (0 to N), the indexer works fine as it sees a continuous chain , then snap sync starts from N+1

but for a running pruned node A user can place sparse files inbetween , so the gap problem ideally only arrives on this scenario , There are two options i see

a) Dropped ERA files can be sparse. From Geth’s perspective, what we should guarantee is consistency: if a query succeeds for a given block number, then all transaction-related queries for transactions included in that block must also succeed

(or)

b) assume when user place era files , we validate them to place the full archive in the era dir else we throw them error , So its either 0 or full history

IMO, Option b is easier as it removes the gap issues but a bad UX. Can we say the goal of this issue is to solve option A ?