running the spider on reddit.com/r/onions returns
https://reddit.com/r/onions Failed - Can't Spider - java.io.IOException: Server returned HTTP response code: 429 for URL: https://www.reddit.com/r/onions/
I'm assuming this is because the spider uses the default user agent