Critical Bug: Partial Authentication Blocks robots.txt and llms.txt, Preventing SEO Indexing and LLM Access

[TYRONEMICHAEL]

When partial authentication is enabled on a Mintlify documentation site, both robots.txt and llms.txt files are automatically blocked and redirect to the login page. This creates several critical issues:

1. SEO Impact: Search engines cannot access robots.txt, which likely prevents proper indexing of the documentation site
2. LLM Access: The llms.txt file becomes inaccessible, blocking LLMs from accessing documentation content as intended
3. No Granular Control: There's no way to exempt these specific files from authentication

Current Behavior:

• With partial authentication enabled:
  • https://[yourdomain]/robots.txt → Redirects to login page
  • https://[yourdomain]/llms.txt → Redirects to login page
  • Search engines and LLMs cannot access these critical files

Expected Behavior:

• robots.txt and llms.txt should remain publicly accessible even when partial authentication is enabled
• These files should be exempt from authentication requirements by default or through configuration

Steps to Reproduce:

1. Enable partial authentication on a Mintlify documentation site
2. Try to access /robots.txt or /llms.txt in an incognito browser window
3. Observe that both files redirect to the login page instead of displaying content

Impact:

• Documentation site may not be indexed by search engines
• AI assistants and LLMs cannot access documentation through llms.txt protocol
• Reduced discoverability and accessibility of documentation

Previous Support Response:
Support previously suggested removing authentication altogether, which is not a viable solution as we need to protect certain sections of our documentation while maintaining public access to SEO and LLM discovery files.

Proposed Solution:
Please add configuration options to exempt specific paths from authentication, or automatically exempt standard files like robots.txt and llms.txt that should always be publicly accessible.

Environment:

• Mintlify version: [Your version]
• Authentication method: Partial authentication
• Custom domain: [Yes/No]

Additional Context:
This issue directly conflicts with Mintlify's advertised support for "llms.txt, MCP, and whatever comes next" as mentioned on your homepage. The current implementation makes it impossible to use both authentication and LLM accessibility features simultaneously.