Merge pull request #6 from khnumdev/copilot/block-ai-crawlers-access

khnumdev · web-flow · commit e0189adaf688 · 2025-12-28T22:15:35.000+01:00
Block AI crawlers from training on blog content
diff --git a/README.md b/README.md
@@ -226,6 +226,20 @@ Example custom CSS file (`assets/main.scss`):
 
 This project is open source and available under the terms specified in the [LICENSE](LICENSE) file.
 
+## Terms of Use & AI Training Policy
+
+**All content on this blog is protected by copyright.** The use of any content from this site for training artificial intelligence or machine learning models is **strictly prohibited** without explicit written permission from the author.
+
+This includes, but is not limited to:
+- Training large language models (LLMs)
+- Fine-tuning AI systems
+- Creating datasets for machine learning
+- Any form of automated content extraction for AI purposes
+
+This website employs technical measures including `robots.txt` directives and meta tags to prevent AI crawlers from accessing and using the content. Any circumvention of these protections is a violation of these terms.
+
+For licensing inquiries or permission requests, please contact the author.
+
 ## Contact
 
 - **Author**: Andrés Pérez Gil
diff --git a/_includes/head.html b/_includes/head.html
@@ -0,0 +1,17 @@
+<head>
+  <meta charset="utf-8">
+  <meta http-equiv="X-UA-Compatible" content="IE=edge">
+  <meta name="viewport" content="width=device-width, initial-scale=1">
+  
+  {%- seo -%}
+  <link rel="stylesheet" href="{{ "/assets/main.css" | relative_url }}">
+  {%- feed_meta -%}
+  {%- if jekyll.environment == 'production' and site.google_analytics -%}
+    {%- include google-analytics.html -%}
+  {%- endif -%}
+
+  <!-- Block AI crawlers at page level -->
+  <meta name="robots" content="noai, noimageai">
+  <meta name="googlebot" content="noai, noimageai">
+  <meta name="AdsBot-Google" content="noai, noimageai">
+</head>
diff --git a/robots.txt b/robots.txt
@@ -0,0 +1,90 @@
+# Block AI crawlers from training on this content
+# See: https://darkvisitors.com/ for more AI bot user agents
+
+# OpenAI
+User-agent: GPTBot
+Disallow: /
+
+User-agent: ChatGPT-User
+Disallow: /
+
+# Google AI
+User-agent: Google-Extended
+Disallow: /
+
+# Common Crawl (used by many AI companies)
+User-agent: CCBot
+Disallow: /
+
+# Facebook/Meta
+User-agent: FacebookBot
+Disallow: /
+
+# Anthropic
+User-agent: anthropic-ai
+Disallow: /
+
+User-agent: Claude-Web
+Disallow: /
+
+# Amazon
+User-agent: Amazonbot
+Disallow: /
+
+# ByteDance/TikTok
+User-agent: Bytespider
+Disallow: /
+
+# Apple
+User-agent: Applebot-Extended
+Disallow: /
+
+# Perplexity AI
+User-agent: PerplexityBot
+Disallow: /
+
+# Cohere
+User-agent: cohere-ai
+Disallow: /
+
+# Other AI crawlers
+User-agent: Omgilibot
+Disallow: /
+
+User-agent: Omgili
+Disallow: /
+
+User-agent: YouBot
+Disallow: /
+
+User-agent: Diffbot
+Disallow: /
+
+User-agent: ImagesiftBot
+Disallow: /
+
+User-agent: ai2bot
+Disallow: /
+
+# Allow legitimate search engines for SEO
+User-agent: Googlebot
+Allow: /
+
+User-agent: Bingbot
+Allow: /
+
+User-agent: Slurp
+Allow: /
+
+User-agent: DuckDuckBot
+Allow: /
+
+User-agent: Baiduspider
+Allow: /
+
+User-agent: YandexBot
+Allow: /
+
+# Default: allow all other bots
+User-agent: *
+Allow: /
