torcache requests - rate limit

[Ivshti]

I'm trying out DHTcrawler2 and it performs really well - it discovers around 400 thousand unique torrents per day. However, it does not index all of them (merely around 400). Upon examining the logs, I discovered the issue is related to a timed out request to torcache, which I guess is because of a rate limit.

Are you doing anything special on your instance (bt.com) to overcome this limit?

[kevinlynx]

bt.cm uses the same codes hosted here. I'm nore sure whether torcache limit the request rate. The default setting config hash_reader 100 download processes per reader {max_download_per_reader,100} in hash_reader.config. And it's better to start hash_reader with 2 readers (win_start_hash.bat), higher than 2 does not really make things fast because of mongodb. BTW, dhtcrawler2 now is stable, i config it to use sphinx search on bt.cm, 50 dht nodes, 2 hash_readers, and enable index_cache, it's stable.