Code Cleanup #2
Description
Just normal overall code clean up and optimization. Currently, it takes around 12-15 seconds to pull, write, parse, delete, and write the discovered URLs to file (basically, perform all the functionality) for https://www.tiktok.com/robots.txt, which is about 16 files and 5200 URLs so it can be much faster