Skip to content

Code Cleanup #2

@rwils83

Description

@rwils83

Just normal overall code clean up and optimization. Currently, it takes around 12-15 seconds to pull, write, parse, delete, and write the discovered URLs to file (basically, perform all the functionality) for https://www.tiktok.com/robots.txt, which is about 16 files and 5200 URLs so it can be much faster

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions