Automatically respect robots.txt

[DemiMarie]

It should be very easy for those who are writing nice bots and want to respect robots.txt to do so. It would be even better if being mean (and not respecting it) was a bit harder.

[krichprollsch]

Hello @DemiMarie, Thanks for the issue.
Offer an easy way to respect robots.txt with Lightpanda is on our roadmap.

[DemiMarie]

Would it be possible to turn this on by default? I wonder if a lot of scrapers that don’t respect it do so out of lazyness rather than malice.

[krichprollsch]

FYI the work is in progress to support robots.txt into #1407
It will be optional first, but we will update our documentation and our README to recommand it.