Skip to content

February/March 2025 Crawl

Choose a tag to compare

View all tags
@Mattm27 Mattm27 released this 14 Mar 21:44
· 107 commits to main since this release
V3.0.0
092d17a

Introduced Docker containerization for the web crawler.

  • Dockerization: The crawler is now fully containerized, including the addition of Dockerfiles for MariaDB and the complete isolation of containers from the local machine
  • Improved Efficiency: Reduced wait times and added a retry policy for persistent services, enhancing crawl speed and reliability
  • Enhanced Error Handling and Cleanup: More robust error handling implemented, along with automatic container shutdown and cleanup of volumes to ensure a cleaner environment after crawls
Assets 2
Loading