feat: agent-friendly docs (llms.txt + markdown endpoints)

[marc0olo]

Summary

Implements the Agent-Friendly Documentation spec so AI agents can discover and consume our docs programmatically.

Deployment structure after all steps are complete

cli.internetcomputer.org/
├── llms.txt                        ← root index (copied from latest version + version nav prepended)
├── index.html                      ← redirect to /0.2/ (existing)
├── versions.json                   ← existing
├── 0.2/
│   ├── llms.txt                    ← version-local index (links to /0.2/*.md URLs)
│   ├── quickstart.md               ← clean markdown endpoint (frontmatter stripped, title heading)
│   ├── guides/
│   │   ├── installation.md
│   │   └── ...
│   ├── concepts/*.md
│   ├── reference/*.md
│   ├── migration/*.md
│   ├── quickstart/
│   │   └── index.html              ← existing Starlight HTML (now with agent-signaling directive)
│   └── ...
├── 0.1/
│   ├── llms.txt                    ← added after cherry-picking to docs/v0.1
│   ├── *.md
│   └── ...
└── main/
    ├── llms.txt
    ├── *.md
    └── ...

Components

1. astro-agent-docs.mjs — Astro integration (astro:build:done hook)

   • Copies each source .md file into build output (frontmatter stripped, title prepended as # heading, UTF-8 BOM)
   • Generates llms.txt index with pages grouped by sidebar section, using absolute URLs with the versioned base path

2. rehype-agent-signaling.mjs — Rehype plugin

   • Injects a visually-hidden <blockquote class="agent-signaling"> at the top of every HTML page
   • Points agents to /llms.txt for documentation discovery
   • Uses CSS clip: rect(0,0,0,0) (not display:none) so it survives HTML-to-markdown conversion

3. Workflow changes (docs.yml)

   • Upgraded to Node 22 (needed for fs.globSync)
   • publish-root-files now copies llms.txt from the latest version's folder on docs-deployment and prepends version navigation links from versions.json
   • Root llms.txt is always rebuilt when versions.json changes, so it stays in sync with version bumps

4. Documentation

   • Updated VERSIONED_DOCS.md with agent-friendly docs section, updated deployment structure, removed outdated beta versions section

Design decisions

• .md files live inside versioned folders — generated in the same Astro build as HTML, so zero content drift by construction
• Root /llms.txt is owned by publish-root-files — rebuilt on every main push from the latest version's llms.txt + versions.json. This avoids the timing glitch where a version tag is pushed before versions.json is bumped
• Version navigation only in root llms.txt — versioned copies are static and never go stale; root copy gets dynamic version links prepended from versions.json

Spec checks this addresses

Check	Status
llms-txt-exists	✅ Fixed — /llms.txt at root
llms-txt-valid	✅ Fixed — proper format with sections
llms-txt-links-resolve	✅ Fixed — links to versioned .md files
llms-txt-links-markdown	✅ Fixed — all links are .md URLs
markdown-url-support	✅ Fixed — .md files served alongside HTML
llms-txt-directive	✅ Fixed — agent signaling on every page
content-negotiation	⬚ Not fixable — IC asset canister doesn't support Accept header routing
http-status-codes	⬚ Separate issue — asset canister soft 404 behavior
redirect-behavior	⬚ Acceptable — root redirect is bypassed by llms.txt discovery

Post-merge steps

Merging to main will deploy /main/llms.txt and /main/*.md automatically. To get llms.txt on the existing released versions and at root:

1. Add llms.txt to v0.2 docs (latest)

Cherry-pick the relevant commits to the docs/v0.2 branch:

git checkout docs/v0.2
git cherry-pick 858ab57 b6c2f84   # agent-docs plugin + Node 22
git push origin docs/v0.2

This triggers publish-versioned-docs → deploys /0.2/llms.txt + /0.2/*.md.

2. Update root llms.txt

Push any docs change to main (or re-run the workflow) to trigger publish-root-files, which will now find /0.2/llms.txt on docs-deployment and copy it to root with version navigation.

If the merge commit itself touches docs/** or docs-site/**, this happens automatically.

3. Add llms.txt to v0.1 docs (optional)

git checkout docs/v0.1
git cherry-pick 858ab57 b6c2f84   # agent-docs plugin + Node 22
git push origin docs/v0.1

4. Verify

npx afdocs check https://cli.internetcomputer.org