fix: allow crawling of docs v1.0.0 and v2.0.0

Remove Disallow rules from robots.txt and noindex headers from
netlify.toml for old doc versions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Neha Gupta <gneha21@yahoo.in>

Author: nehagup

netlify.toml

@@ -9,17 +9,6 @@
 ## Note: if you are looking for Redirects
 # they have been moved to /static/_redirects to make it more manageable - swyx
 
-# noindex headers for deprecated doc versions
-[[headers]]
-  for = "/docs/1.0.0/*"
-  [headers.values]
-    X-Robots-Tag = "noindex, nofollow"
-
-[[headers]]
-  for = "/docs/2.0.0/*"
-  [headers.values]
-    X-Robots-Tag = "noindex, nofollow"
-
 # Security headers for all pages
 [[headers]]
   for = "/*"

static/robots.txt

@@ -7,8 +7,6 @@ User-agent: *
 Allow: /
 Crawl-delay: 5
 Disallow: /cgi-bin/
-Disallow: /docs/1.0.0/
-Disallow: /docs/2.0.0/
 
 # Sitemap
 Sitemap: https://keploy.io/docs/sitemap.xml