Skip to content

Conversation

@devin-ai-integration
Copy link
Contributor

@devin-ai-integration devin-ai-integration bot commented Oct 22, 2025

Update llms.txt documentation with improved clarity and structure

Summary

This PR restructures the llms.txt documentation page to improve clarity and add information about recent platform capabilities. The changes convert bullet-point descriptions to flowing prose, clarify the availability of each file type, add analytics information, and address PR feedback about content placement.

Key changes:

  • Restructured content from bullet points to prose for better readability
  • Clarified that llms.txt is available at any level of documentation hierarchy (e.g., /llms.txt, /docs/llms.txt, /v1/api-reference/llms.txt)
  • Clarified that llms-full.txt is only available at the root of documentation sites
  • Added explanation of automatic markdown serving when LLM bots are detected (90%+ token reduction)
  • Moved image to appear after the llms.txt section (per PR feedback from @kgowru)
  • Made example links clickable (converted from code blocks to markdown links)
  • Added "Analytics and monitoring" section describing dashboard features
  • Added "LLM" to Vale acronyms exceptions to fix linting

Review & Testing Checklist for Human

  • Verify availability claims: Test that llms.txt is actually accessible at different documentation levels (root, sub-sections, API reference pages) and that llms-full.txt is only at the root. Try accessing both files on a live docs site.
  • Verify analytics accuracy: Confirm that the analytics features described (LLM provider tracking, bot vs. human traffic breakdown) actually exist in the Fern Dashboard and match the description.
  • Check content flow: Review whether the restructured prose reads better than the previous bullet-point format. Consider if any important context was lost (like "Small and fast", "Token efficient" descriptors).
  • Test example links: Click through the example links to docs.cohere.com and buildwithfern.com to verify they work and display the expected content.

Test Plan

  1. Navigate to a Fern docs site (e.g., docs.cohere.com) and try accessing /llms.txt, /docs/llms.txt (if it has sub-sections), and /api-reference/llms.txt to verify availability
  2. Try accessing /llms-full.txt at root and at a sub-path to confirm it's only available at root
  3. Check the Fern Dashboard analytics to verify the described features exist

Notes

  • This documentation update is based on recent platform improvements discussed with Kapil (@devalog)
  • Addresses PR feedback from @kgowru about moving the markdown serving statement to the llms.txt section and moving the image up
  • Session: https://app.devin.ai/sessions/83af156cfd684cb18bfc587005ed53bd
  • The 90%+ token reduction claim for markdown serving should be verified against actual metrics if available

@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@kennyderek
Copy link
Contributor

could we find a non-cohere example for llms-full, I think their site is actually a bit wonky on the llms front, I'll look into why. elevenlabs should be good https://elevenlabs.io/docs/llms-full.txt
Re: See an example: [docs.cohere.com/llms-full.txt](http://docs.cohere.com/llms-full.txt](https://docs.cohere.com/llms-full.txt))

@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@devalog devalog requested a review from kgowru October 23, 2025 15:53
@github-actions
Copy link
Contributor

@devalog devalog merged commit afecc5b into main Oct 23, 2025
2 checks passed
@devalog devalog deleted the devin/1761163974-update-llms-txt-docs branch October 23, 2025 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants