Crawl Full website with single python file and store to your favorite database - (Elastic search or MongoDB). Also crawl surface with threading or multiprocessing. Now web is more enjoyable.
Crawler.py is recursive crawler
crawler-2.0 is multiprocessing approach
crawler-3.0 is thread based crawler