"Exceed maximum block range: 5000" on contract.queryFilter

[khoanguyen-yang]

Hi all,

My purpose is to query events of a specific contract and store the transaction hashes of those events into my database.

However, I encountered the error exceed maximum block range: 5000 when using contract.queryFilter. The code is pretty simple and straightforward

const events = await contract.queryFilter(contract.filters.EventName());

I also tried to implement a "paging behavior" by getting the blockNumber first and did an implementation as following:

// inside a loop
const events = await contract.queryFilter(contract.filters.EventName(), blockNumber - 5000 * (i+1), blockNumber - 5000 * i);

The query succeeded for a few iterations and then always encountered an error as following, somewhere in the middle of the loop.

processing response error (body="{\"id\":0,\"jsonrpc\":\"2.0\",\"error\":{\"code\":-32603,\"message\":\"handle request error\"}}", error={"code":-32603}, requestBody="{\"method\":\"eth_getLogs\",\"params\":[{\"fromBlock\":\"0x922b5e\",\"toBlock\":\"0x923afe\",\"address\":\"0xd0d6bfce25876aebd50d884b35aae7b8b7f0193d\",\"topics\":[\"0x76264100b4b20aa06c294636c2e40aaf398c3490112488330659f4577010713f\"]}],\"id\":46,\"jsonrpc\":\"2.0\"}", requestMethod="POST", url="https://bsc-dataseed.binance.org/", code=SERVER_ERROR, version=web/5.4.0)

Wonder if there is any other strategy to achieve my purpose. Really appreciate any comments and helps.

[ricmoo]

It’s up to each node operator as to what window they want to expose. The BSC network is not very reliable for these sorts of things, but if you run your own node you might have better luck, but may still experience timeouts.

INFURA for example, on Ethereum has a completely custom solution where all logs are stored and indexed in a database.

You may need to do something similar, depending on your use case.

If you do find another solution though, please let me know. There are more people trying out BSC these days, so I’m sure future peeps would appreciate anything you find. :)

[khoanguyen-yang]

Hi @ricmoo,

My temporary solution is that I use the Logs APIs from BscScan (https://bscscan.com/apis#logs) instead of the contract.queryFilter.

https://api.bscscan.com/api?module=logs&action=getLogs&fromBlock=0&toBlock=latest&address=${address}&topic0=${contractTopic}

It succeeds without any error.

However, I am confused. For both approaches, I keep querying from block 0 to the latest block but the log API succeeds querying more than 5000 blocks while queryFilter does not, although both ways use the same method under the hood which is eth_getLogs.

[zemse]

What provider are you using when you do queryFilter?

You are asking a geth node to give you all the logs for an event having a particular topic contractTopic emitted from some txs in millions of blocks, it's not cheap to request to fulfill as the time complexity here is O(n) where n is the number of blocks in the range. Though there is logsBloom that makes it practical to scan all the blocks in your range by scaling down the time required to check if a log entry could exist in the block. But when you really have a huge range (like 0 to latest), it's a huge resource-consuming request. This is an easy way to DoS a public node, that's why providers have started to put guards that prevent you from making the node scan a lot of blocks.

However, it may be that api.bscscan.com gives you results from a database, which is very faster. If they are simply proxying the REST API to an RPC to a geth node, then it's a DoS vector and they should probably add restrictions too.

[khoanguyen-yang]

Thanks for the information @zemse,

I were using the JsonRPC provider (https://bsc-dataseed.binance.org/) when doing queryFilter, fyi.

The api.bscscan.com does add restrictions to prevent DoS which is the API rate limit, I think.
Currently, the bscscan api approach suits my purposes :) so I think if I encounter any restriction errors with the API, an implementation with paging behavior may help.

All in all, really thanks for the information, it is really helpful. Really appreciate your helps @ricmoo, @zemse

[khoanguyen-yang]

The temporary solution is to use BscScan APIs, which can be read more at https://bscscan.com/apis#logs.
This solution may not be the most ideal one, in my opinion.