This script is using a cache (in sqlalchemy, crawler.db) and crawl to a depth of 3 from the home page. The no_cache parameter prevent '/' to be cached, enforcing new pull of the homepage each time the crawler is launched.
anilskalyane/web-crawler
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|