docs: structured llms.txt index + AI docs improvements#379
docs: structured llms.txt index + AI docs improvements#379gregnazario merged 4 commits intoaptos-labs:mainfrom
Conversation
|
@tippi-fifestarr is attempting to deploy a commit to the AptosLabs Team on Vercel. A member of the Team first needs to authorize it. |
gregnazario
force-pushed
the
ai/structured-llms-txt
branch
from
192e7fe to
8145d76
Compare
gregnazario
requested review from
Copilot and
gregnazario
and removed request for
Copilot
There was a problem hiding this comment.
Pull request overview
This pull request enhances the AI documentation tooling by replacing the default starlight-llms-txt plugin's index with a structured, hierarchical index that organizes documentation pages by section. The structured index provides page titles, descriptions, and per-page .md URLs to help AI agents efficiently select relevant documentation without ingesting all content.
Changes:
- Added structured
llms.txtindex with hierarchical organization (##for sections,###for sub-sections) and per-page.mdURLs - Created custom integration (
llms-txt-index.ts) to override the plugin's default/llms.txtroute viaastro:route:setuphook - Updated AI tools documentation pages to include AskAptos chatbot section and Claude Code usage guide (previously missing)
- Fixed homepage AskAptos chatbot link from placeholder
(#)to/build/ai - Applied all English documentation changes to Spanish and Chinese translations
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
src/pages/llms-index.ts |
New API route that generates structured llms.txt index with sections, sub-sections, and per-page .md links |
src/integrations/llms-txt-index.ts |
Astro integration that overrides plugin's /llms.txt route via astro:route:setup hook |
astro.config.mjs |
Imports new integration and adds plugin configuration (description, optional links) |
src/content/docs/llms-txt.mdx |
Adds "Per-Page Markdown Access" and "Claude Code" usage sections |
src/content/docs/build/ai.mdx |
Adds AskAptos chatbot section, updates llms.txt feed table descriptions |
src/content/docs/index.mdx |
Fixes AskAptos chatbot link from # to /build/ai |
src/content/docs/es/*.mdx |
Spanish translations of all English documentation changes |
src/content/docs/zh/*.mdx |
Chinese translations of all English documentation changes |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if ( | ||
| route.component.includes("starlight-llms-txt") && | ||
| route.component.endsWith("/llms.txt.ts") | ||
| ) { |
There was a problem hiding this comment.
The route component override uses a type cast that bypasses TypeScript's type system. While this may be necessary given Astro's integration API, consider adding a comment explaining why the cast is needed and ensuring the route.component property is indeed writable at runtime. Additionally, verify that the component path is correct and relative to the project root.
| ) { | |
| ) { | |
| // NOTE: Astro's RouteOptions type treats `component` as effectively read-only, | |
| // but the `astro:route:setup` integration hook is explicitly designed to allow | |
| // reassigning `route.component` at runtime in order to swap out the route handler. | |
| // We assert a mutable `component` property here to satisfy TypeScript while relying | |
| // on Astro's documented behavior that this mutation is supported. | |
| // The replacement path is intended to be relative to the project root and must | |
| // point to a valid page/component file that will handle the `/llms.txt` route. |
| export const GET: APIRoute = async () => { | ||
| // eslint-disable-next-line @typescript-eslint/no-unsafe-assignment, @typescript-eslint/no-unsafe-call | ||
| const docs = await getCollection("docs"); | ||
|
|
||
| // Filter to English-only, exclude 404, locale roots, and excluded pages | ||
| const englishDocs = (docs as Doc[]) | ||
| .filter( | ||
| (doc) => | ||
| !doc.id.startsWith("es/") && | ||
| !doc.id.startsWith("zh/") && | ||
| doc.id !== "es" && | ||
| doc.id !== "zh" && | ||
| !doc.id.includes("404") && | ||
| !EXCLUDE_PAGES.has(doc.id), | ||
| ) | ||
| .sort((a, b) => a.id.localeCompare(b.id)); | ||
|
|
||
| // Group by section | ||
| const sections = new Map<string, Doc[]>(); | ||
| const topLevel: Doc[] = []; | ||
|
|
||
| for (const doc of englishDocs) { | ||
| const sectionKey = getSectionKey(doc.id); | ||
| if (sectionKey === "other") { | ||
| topLevel.push(doc); | ||
| } else { | ||
| if (!sections.has(sectionKey)) sections.set(sectionKey, []); | ||
| const sectionList = sections.get(sectionKey); | ||
| if (sectionList) sectionList.push(doc); | ||
| } | ||
| } | ||
|
|
||
| // Build a lookup of slug -> doc for sub-section title resolution | ||
| const docsBySlug = new Map<string, Doc>(); | ||
| for (const doc of englishDocs) { | ||
| docsBySlug.set(doc.id, doc); | ||
| } | ||
|
|
||
| const lines: string[] = [ | ||
| "# Aptos Developer Documentation", | ||
| "", | ||
| "> Developer documentation for the Aptos blockchain — Move smart contracts, SDKs, APIs, indexer, node operations, and AI tools.", | ||
| "", | ||
| "This file lists all documentation pages with descriptions. Each page is also available as raw Markdown by appending `.md` to its URL (e.g. `https://aptos.dev/build/guides/first-transaction.md`).", | ||
| "", | ||
| "## Full Documentation", | ||
| "", | ||
| `- [Complete documentation](${SITE_URL}/llms-full.txt): All pages concatenated into a single file`, | ||
| `- [Abridged documentation](${SITE_URL}/llms-small.txt): Condensed version for smaller context windows`, | ||
| "", | ||
| ]; | ||
|
|
||
| // Top-level pages | ||
| for (const doc of topLevel) { | ||
| lines.push(formatEntry(doc)); | ||
| } | ||
| if (topLevel.length > 0) lines.push(""); | ||
|
|
||
| // Grouped sections with auto-derived sub-sections | ||
| for (const [sectionKey, sectionLabel] of Object.entries(SECTION_CONFIG)) { | ||
| const sectionDocs = sections.get(sectionKey); | ||
| if (!sectionDocs || sectionDocs.length === 0) continue; | ||
|
|
||
| lines.push(`## ${sectionLabel}`); | ||
| lines.push(""); | ||
|
|
||
| // Split into direct children and sub-sectioned pages | ||
| const directChildren: Doc[] = []; | ||
| const subSections = new Map<string, Doc[]>(); | ||
| const subSectionOrder: string[] = []; | ||
|
|
||
| for (const doc of sectionDocs) { | ||
| const subKey = getSubSectionKey(doc.id, sectionKey); | ||
| if (subKey) { | ||
| if (!subSections.has(subKey)) { | ||
| subSections.set(subKey, []); | ||
| subSectionOrder.push(subKey); | ||
| } | ||
| const subList = subSections.get(subKey); | ||
| if (subList) subList.push(doc); | ||
| } else { | ||
| directChildren.push(doc); | ||
| } | ||
| } | ||
|
|
||
| // Print direct children first | ||
| for (const doc of directChildren) { | ||
| lines.push(formatEntry(doc)); | ||
| } | ||
|
|
||
| // Print sub-sections | ||
| for (const subKey of subSectionOrder) { | ||
| const subDocs = subSections.get(subKey); | ||
| if (!subDocs || subDocs.length === 0) continue; | ||
|
|
||
| // Use the index page's title as the sub-section header, or derive from path | ||
| const indexDoc = docsBySlug.get(subKey); | ||
| const fallback = subKey.split("/").pop() ?? subKey; | ||
| const subLabel = indexDoc ? indexDoc.data.title : fallback; | ||
|
|
||
| lines.push(""); | ||
| lines.push(`### ${subLabel}`); | ||
| lines.push(""); | ||
|
|
||
| for (const doc of subDocs) { | ||
| lines.push(formatEntry(doc)); | ||
| } | ||
| } | ||
|
|
||
| lines.push(""); | ||
| } | ||
|
|
||
| return new Response(lines.join("\n"), { | ||
| status: 200, | ||
| headers: { | ||
| "Content-Type": "text/plain; charset=utf-8", | ||
| "Cache-Control": "public, max-age=3600", | ||
| }, | ||
| }); | ||
| }; |
There was a problem hiding this comment.
The GET handler lacks error handling. If getCollection fails or if there are issues during content processing (e.g., malformed doc data), the route will return a generic 500 error without useful diagnostics. Wrap the main logic in a try-catch block and return a meaningful error response if something goes wrong.
There was a problem hiding this comment.
This endpoint is prerender = true so it runs at build time only. If getCollection fails the build itself fails, which is the behavior we want. Adding try-catch would mask build errors rather than surface them.
|
@claude is it possible for you to rebase and fix this PR? |
tippi-fifestarr
force-pushed
the
ai/structured-llms-txt
branch
2 times, most recently
from
5d0aac0 to
81ed16c
Compare
|
Hey @gregnazario — rebased onto latest To clarify the concerns from our earlier discussion: Static / SSR question: This is fully static. How the hook works: The integration ( Copilot review items:
All 12 files clean, no conflicts. Ready for review. — @tippi-fifestarr + Claude |
…nd .md URLs Replace the default starlight-llms-txt plugin index (which only linked to full/small blob files) with a structured index that lists every documentation page grouped by section and auto-derived sub-sections. Each entry includes the page title, description, and a direct .md URL for per-page AI ingestion. Also adds AskAptos chatbot section to AI overview, documents the per-page .md URL feature and Claude Code usage on the llms-txt page, fixes the homepage AskAptos link, and translates all changes to ES and ZH. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Tippi Fifestarr <62179036+tippi-fifestarr@users.noreply.github.com>
gregnazario
force-pushed
the
ai/structured-llms-txt
branch
from
81ed16c to
0f8f633
Compare
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The sidebar referenced kotlin-sdk/fetch-data/response-handling but the file was at kotlin-sdk/response-handling.mdx, causing the build to fail. Also updated the internal link in fetch-data-via-sdk.mdx.
Summary
llms.txtindex: Replaces the default plugin output (just 2 links to blob files) with a Decibel-style structured index listing every doc page with title, description, and per-page.mdURL — grouped by section (##) with auto-derived sub-sections (###) from path depthllms-txt-index.ts): Usesastro:route:setuphook to override the plugin's/llms.txtroute while leavingllms-full.txtandllms-small.txtuntouched.mdlinks.mdURL feature(#)placeholder to/build/aiBuilds on the foundation from #375. The key insight is that the default
llms.txtindex only links to blob files — agents can't tell which pages are relevant without ingesting everything. The structured index lets them pick specific pages via.mdURLs.Test plan
pnpm dev→ verifyhttp://localhost:4321/llms.txtshows structured index with#/##/###hierarchy/llms-full.txtand/llms-small.txtstill work (plugin routes untouched).mdURLs work (e.g./build/ai.md)/build/ai🤖 Generated with Claude Code