Support for llm.txt files

[gusfune]

Goals

Add llm.txt support for dynamic content in similar fashion of sitemap.xml and robots.txt is implemented

Non-Goals

Generate the content for the website based on the site, just stick to a similar format compared to the current similar implementations.

Background

From the proposal format website: https://llmstxt.org/

Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety. Converting complex HTML pages with navigation, ads, and JavaScript into LLM-friendly plain text is both difficult and imprecise.

While websites serve both human readers and LLMs, the latter benefit from more concise, expert-level information gathered in a single, accessible location. This is particularly important for use cases like development environments, where LLMs need quick access to programming documentation and APIs.

A few companies already adopting the format:

• Strapi
• Vultr
• Langbase
• Checkly
• CircleCI

Proposal

Implement a llm.(js|ts) file that would live in the app directory in similar way we utilise sitemap or robots.txt.

[icyJoseph]

Hi,

Justo echo this entry - #80692 (comment) - I understand your request is more in line with the functionality we have for sitemap, robots etc... but just so that people know there's a way to do it yourself.

Vercel also has https://vercel.com/docs/llms.txt

And since yesterday, so does Next.js https://nextjs.org/docs/llms.txt - I was just about to announce this -

[gusfune]

Absolutely. I believe the "official support" as a functionality is more as a show of support/endorse a standard to push for wider adoption than something that is a technical limitation. I'm also running that in similar fashion. Thanks for the link, did not see that discussion from when I searched the topic.

[icyJoseph]

No worries.

I did some quick searching around regarding llms.txt and found mixed opinions. Though, some big players do have it. In a way it is a low entry barrier. Perhaps it has to become more stablished.

This kind of discussion entry signals that the standard is growing though. At the moment is that some engagement. We'll see what the core team thinks about it.