Add llms.txt template and generator

[theletterf]

Fixes #1689

The approach here is to create a template based on https://llmstxt.org/ recommendations, and then append a number of generated H2 sections after the boilerplate. The sections should contain a list, ordered following the same navigation order in the site, of the first level sections of each category, with a Markdown link to their .md files and, after a colon, their description, extracted from the description: frontmatter of those pages. This process should happen after assembly, more or less around the time we create the sitemap.xml.

For clarity. If, for example, we have ## Solutions as the category (https://www.elastic.co/docs/solutions), then I would expect the list to contain the first three subsections you see when opening it, and no more. The goal is for an LLM to consume this file and get an overview of the first level of navigation within each section.

LLM disclosure: All commits in this PR have been coauthored by Claude using Cursor Enterprise

[Mpdreamz]

❤️ thank you for getting this done @theletterf!

[theletterf]

@Mpdreamz @reakaleek After careful consideration, I reverted the .md append logic in this commit, because it was proving too complex to handle. The current PR generates the LLMs.txt file as intended and doesn't affect LLM links generation.

I think we could follow up in another PR to see how to best handle converting the links to .md links. I still think it's a good idea, as it would make things easier for the LLM, but it requires some special handling of various cases, and that risks making this PR too complicated, IMHO.

Also related to this, we could consider opening a separate PR to raise warnings when absolute URLs to our docs are being used (they should not).

[Copilot]

Pull Request Overview

This PR implements an llms.txt template and generator to provide structured documentation overviews for LLM consumption, following the llmstxt.org recommendations. The implementation creates a boilerplate template for the root documentation and dynamically generates navigation sections listing first-level content within each documentation category.

Key changes include:

• Added a new LlmsNavigationEnhancer class to generate structured navigation sections from the documentation hierarchy
• Enhanced the assembler build service to append navigation content to existing llms.txt files
• Extended LLM markdown rendering utilities to handle URL conversion for both localhost and relative URLs
• Updated the LLM markdown exporter to use a predefined template for root index files

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
LlmsNavigationEnhancer.cs	New class that extracts first-level navigation items and generates structured markdown sections with titles, URLs, and descriptions
AssemblerBuildService.cs	Integration point that enhances llms.txt files by appending generated navigation sections after assembly
LlmInlineRenderers.cs	Minor formatting change adding extra whitespace
LlmBlockRenderers.cs	Enhanced URL handling with localhost-to-canonical conversion and extracted utility methods for reuse
LlmMarkdownExporter.cs	Added llms.txt template constant and logic to use template for root index files

[theletterf]

@Mpdreamz Do we have your blessing? 🙏🏻

[Mpdreamz]

Spot a few cases for improvements but not blocking this PR lets get v1 of llms.txt out the door :) its been too long.