Create robots.txt

Added a robots.txt file to avoid scraping

Signed-off-by: Jonathan Peters <jonathanpeters051@gmail.com>

Author: QMS85

robots.txt

@@ -0,0 +1,31 @@
+# Allow all major search engines
+User-agent: Googlebot
+Disallow:
+
+User-agent: Bingbot
+Disallow:
+
+User-agent: Slurp
+Disallow:
+
+# Block known bad bots and generic scrapers
+User-agent: AhrefsBot
+Disallow: /
+
+User-agent: SemrushBot
+Disallow: /
+
+User-agent: MJ12bot
+Disallow: /
+
+User-agent: DotBot
+Disallow: /
+
+User-agent: BlexBot
+Disallow: /
+
+# Block all other bots (catch-all)
+User-agent: *
+Disallow: /
+
+Sitemap: https://yourdomain.com/sitemap.xml