robots: block semver-based URLs

jsha · jsha · commit 0126aa71311a · 2024-12-19T12:35:30.000-08:00
diff --git a/static/robots.txt b/static/robots.txt
@@ -1 +1,6 @@
 Sitemap: https://docs.rs/sitemap.xml
+# Semver-based URL are always redirects, and sometimes
+# confuse Google's duplicate detection, so we block crawling them.
+# https://docs.rs/about/redirections
+Disallow: */^
+Disallow: */~
