You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: fern/products/docs/pages/seo/llms-txt.mdx
+18-3Lines changed: 18 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,17 +9,25 @@ description: Enable tools like Cursor, GitHub Copilot, ChatGPT, and Claude to un
9
9
10
10
Contains a lightweight summary of your documentation site with each page distilled into a one-sentence description and URL. The format is structured to help AI tools parse and understand your documentation's hierarchy and organization.
11
11
12
+
`llms.txt` is available at any level of your documentation hierarchy (e.g., `/llms.txt`, `/docs/llms.txt`, `/v1/api-reference/llms.txt`).
13
+
12
14
See an example: [docs.cohere.com/llms.txt](https://docs.cohere.com/llms.txt)
13
15
14
16
## `llms-full.txt`
15
17
16
18
Contains complete documentation content including the full text of all pages. For API documentation, this includes your complete API Reference with resolved OpenAPI specifications and SDK code examples for enabled languages. Each API endpoint page includes its full OpenAPI spec.
17
19
20
+
`llms-full.txt` is available at the root of your documentation site and summarizes all of your documentation.
21
+
18
22
When Fern detects an LLM bot accessing your documentation, it automatically serves Markdown instead of HTML, reducing token consumption by 90%+.
19
23
20
-
View the llms.txt files for this site:
21
-
-`https://buildwithfern.com/learn/llms.txt`
22
-
-`https://buildwithfern.com/learn/llms-full.txt`
24
+
See an example: [docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt)
@@ -34,6 +42,13 @@ View the llms.txt files for this site:
34
42
The [Fern Dashboard](https://dashboard.buildwithfern.com/) provides comprehensive analytics for llms.txt usage including:
35
43
- Traffic by LLM provider (Claude, ChatGPT, Cursor, etc.)
36
44
- Page-level breakdowns of bot vs. human visitors for `.md` and `llms.txt` files
45
+
- Performance metrics including load times, processing times, and content length
46
+
- User agent detection to identify bot traffic versus browser requests
37
47
38
48
This visibility helps you understand LLM traffic patterns and optimize your documentation for AI consumption.
39
49
50
+
## Technical Details
51
+
52
+
The llms.txt generation system handles complex schema types including self-referencing schemas, preventing infinite recursion during content generation. For large documentation sites, the system uses optimized timeouts to ensure reliable content delivery without timing out.
53
+
54
+
For REST API endpoints, the markdown generation includes OpenAPI specifications in YAML format, making it easy for LLMs to understand your API structure. The same markdown generation logic is used for both serving llms.txt files and indexing content for search, ensuring consistency across your documentation platform.
0 commit comments