Block AI crawlers #3508
Block AI crawlers #3508
Conversation
| Disallow: / | ||
|
|
||
| User-agent: Seekr | ||
| Disallow: / |
There was a problem hiding this comment.
FWIW, I've found a longer list at https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-Agent: FacebookBot
Disallow: /
User-Agent: Applebot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: YouBot
Disallow: /
Not sure how to best manage this going forward. I imagine there are templates that are being crowdsourced.
Also, I've created a draft rule to block them at the Cloudflare level, which I think we should enable:
There was a problem hiding this comment.
I'll go ahead and enable the CF block to see if this stop the crawl traffic.
There was a problem hiding this comment.
Actually, I'll give it until this afternoon. We updated robots.txt at noon yesterday and Google update their view of it every 24h. This might be true for the other crawlers as well so let's see if the initial robots.txt has any effect on its own after 24h.
2 participants
No description provided.