Hey! Collab?

[danieleperera]

Hey,
Nice project! I have been working on something similar. I was checking out how to make scrapy work with TOR and found your project! I was thinking if we could do a collab. https://github.com/danieleperera/OnionIngestor
It's very extendable and uses the logic of modules. There are source modules which collect onion links from the clear web. Then there are operators which scan and crawls onion links. Finally, the database engines where the collected information is index/stored! Currently, I'm working with onionscan as the main operator and elasticsearch and telegram as main database engines!

[dirtyfilthy]

Hey man, I'd be happy to contribute to your project. There's also some useful lessons I learnt around the crawling infrastructure that might be helpful. I'll take a look at your repo tonight after work and get in touch. Also onionscan, awesome. I got a lot of ideas from that project.