Use this library within crawler/scraper programs to parse robots.txt files and check if your crawler user-agent is allowed to load certain paths.
You can find the documentation at crwlr.software.
If you consider contributing something to this package, read the contribution guide (CONTRIBUTING.md).