Plain markdown version for llms crawlers

[matiaslopezd]

It's becoming more natural every day for people to search through LLMs. Gitbook could position itself as the first documentation platform offering plain markdown responses to crawlers based on the User-Agent header. This way, we could leverage Gitbook much more by providing all our product information to be easily indexed by these specific crawlers.

flowchart TD
    A[User searches in LLM] --> B[LLM sends request to GitBook]
    B --> C{Check User-Agent Header}
    C -->|LLM Crawler| D[Return Plain Markdown Response]
    C -->|Regular Browser| E[Return Standard HTML/UI]
    
    D --> F[LLM processes markdown easily]
    F --> G[Better indexing & understanding]
    G --> H[More accurate responses to users]
    
    E --> I[Standard user experience]
    
    J[GitBook becomes first docs platform<br/>optimized for LLM crawlers] --> K[Competitive advantage]
    
    L[Product information] --> M[Easier indexing by LLM crawlers]
    M --> N[Better discoverability]
    
    style A fill:#1976d2,color:#fff
    style D fill:#388e3c,color:#fff
    style G fill:#f57c00,color:#fff
    style J fill:#7b1fa2,color:#fff
    style K fill:#d32f2f,color:#fff

Loading

[matiaslopezd]

This is similar to https://github.com/orgs/GitbookIO/discussions/936, but not the same because crawlers are not looking for the MD version, just the normal path.