Add a function to check robots.txt rules and page type #69

Open

Labels

opened

on Jan 10, 2024

Check the default RobotFileParser
is_not_crawlable(link) + can_fetch(crawler, link) → is_doable(link)

Metadata

Assignees

No one assigned

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests