Gather ALL hidden services

[juhanurmi]

We could gather every hidden service serving some service and categories them.

HTTP servers
IRC servers
BitTorrent tarckers
etc.

Moreover, we could show the actual connection status of the hidden service. Did the circuit fail? Which ports answerd? Does this hidden service even exists?

[juhanurmi]

https://lists.torproject.org/pipermail/tor-talk/2013-September/029856.html

Scanning of hidden services is able via torified netcat or proxychains-ng with nmap. Works slow as expected. (but to optimize this we could pick custom routes with fast relays, I will try this in some time) Hidden service scanning: https://www.crypto-rebels.de/scanhidden.html
For speeding up we can start multiple processes of nc parallel (we need a transparent proxy or similar to handle all the nc requests)
What remains an issue is how we want to gather addresses of hidden services. I will research on that. It would be really nice if we could gather really all of them, scan them and collect the data.
I would suggest that we use netcat for scanning a whole port range (way more fast than nmap) and then if we find a open port start a nmap service scan on that

[juhanurmi]

If we know a hidden service address, technically it should be possible:

1. Check if a hidden service exists by testing if you can find its HS descriptor in HSDirs
2. Check if an HS is online by checking if you can rendezvous with it
3. Which ports are open
4. What kind of services it is running (HTTP server, IRC server, BitTorrent tracker...)

Before that we need to find possible hidden services. We can use the known bug in the Tor for that:
www.ieee-security.org/TC/SP2013/papers/4977a080.pdf

After that we can show this list on Ahmia's website.

[ohmygodel]

Hey dude,

I don't think it's a good idea to use the "Trawling for Hidden Services" techniques to find .onions. First, you can no longer use their "shadowing" technique to pre-position relays with HSDir flags [0,1]. Second, Tor considers that it is a privacy vulnerability to be able to predict the HSDirs of a hidden service and use that to target your HSDir, and they have a plan to fix it (see Section 1.4 of [2]). In general, publishing information about what your Tor relays observe has the potential to harm user privacy and should be handled carefully.

Another idea for finding .onions is to use erroneous name resolution requests to root DNS servers [3]. Data seems to be available via the DNS-OARC system: https://www.dns-oarc.net/oarc/data/catalog. You need to be a “participant” to access the data. It’s not clear what is required (you must request an applications from the OARC admin), but Matthew Thomas (author of [3]) told me at HotPETS14 that there was a small fee (~$100), and your organization will need to sign some data use agreement.

Cheers,
Aaron
http://www.ohmygodel.com

[0] https://trac.torproject.org/projects/tor/ticket/8146
[1] https://trac.torproject.org/projects/tor/ticket/8147
[2] https://lists.torproject.org/pipermail/tor-dev/2013-November/005877.html
[3] https://www.petsymposium.org/2014/papers/Thomas.pdf

[juhanurmi]

This ticket is a part of a larger plan to build an observatory for HSs.

http://muistio.tieke.fi/p/hidden_service_observatory

[juhanurmi]

https://trac.torproject.org/projects/tor/wiki/org/sponsors/SponsorR

[gpestana]

Nice one! @juhanurmi what's this feature status?