Broaden llms.txt and llms-full.txt to index all public sections (DOCS-98) (#3708)

## What and why

`layouts/index.txt` renders `llms.txt`, the index an AI crawler reads to
discover Chainguard docs. It ranged over only four hardcoded sections --
`chainguard`, `open-source`, `software-security`, `compliance` -- so
every page outside them was invisible to those crawlers. That silently
dropped the largest section on the site (`platform`, 300+ pages),
`get-started`, and the root-level MCP server and Developer Resources
pages. Adding a new section later would reopen the same gap without
anyone noticing.

Fixes DOCS-98.

## Changes

- **`layouts/index.txt`** -- range over every section with `GroupBy
"Section"` instead of naming four, so new sections are indexed
automatically. Root-level pages group under a "General Resources"
heading.
- **Exclusion filter** -- skip any page the site already hides, via
`seo.robots: noindex` or `sitemap_exclude: true`. This keeps `llms.txt`
consistent with the site's own indexing signals.
- **`layouts/index.fulltxt.txt`** (renders `llms-full.txt`) -- apply the
identical section loop and exclusion filter so the two templates stay in
lockstep for future edits. Also fixes a latent bug: a two-variable
`GroupBy` range printed numeric section headers instead of names.
- **`content/vulnerabilities/index.md`** -- mark the legacy 2020
"Vulnerability Information" page `sitemap_exclude: true`, dropping it
from the sitemap and both AI indexes.

## Verification

Built locally with the vendored Hugo and inspected the generated
`public/`:

- `llms.txt`: eight section headings including Platform, Get Started,
the MCP server, and Developer Resources; `noindex` and `sitemap_exclude`
pages absent.
- `llms-full.txt`: same coverage and exclusions; section headers now
show names, not numbers.
- `sitemap.xml`: the legacy vulnerabilities page absent.

## Notes

`contributors/` does not appear in either index -- those are
author-profile `_index.md` section pages, not `RegularPages`. DOCS-101
tracks removing that section entirely.

---
Created in collaboration with Claude Code running Claude Opus 4.8 on
2026-08-05.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: matthewhelmke

content/vulnerabilities/index.md

@@ -2,10 +2,10 @@
 title : "Vulnerability Information"
 type: "article"
 date: 2020-10-06T08:48:23+00:00
-lastmod: 2022-10-06T08:48:23+00:00
+lastmod: 2026-08-05T13:04:18+00:00
 draft: false
 images: []
 weight: 100
 toc: false
+sitemap_exclude: true
 ---
-

layouts/index.fulltxt.txt

@@ -8,11 +8,21 @@ Documentation: CC BY-SA 4.0 | Code Examples: Apache 2.0
 ---
 
 ## Full Documentation
-
-{{ range $section, $pages := .Site.RegularPages.GroupBy "Section" }}
-## Section: {{ $section | title }}
-
-{{ range $pages.Pages }}
+{{ range .Site.RegularPages.GroupBy "Section" -}}
+{{- $section := .Key -}}
+{{- $heading := cond (eq $section "") "General Resources" ($section | replaceRE "-" " " | title) -}}
+{{/* Collect only pages the site wants advertised: skip anything marked noindex or excluded from the sitemap. */}}
+{{- $visible := slice -}}
+{{- range .Pages -}}
+{{- $robots := "" -}}
+{{- with .Params.seo }}{{ $robots = .robots | default "" }}{{ end -}}
+{{- $excluded := or (in $robots "noindex") (eq .Params.sitemap_exclude true) -}}
+{{- if not $excluded }}{{ $visible = $visible | append . }}{{ end -}}
+{{- end -}}
+{{- if $visible }}
+## {{ $heading }}
+
+{{ range $visible }}
 ### {{ .Title }}
 
 URL: {{ .Permalink | replaceRE "/$" ".md" }}
@@ -26,4 +36,5 @@ Tags: {{ delimit .Params.tags ", " }}
 ---
 
 {{ end }}
-{{ end }}
+{{- end -}}
+{{- end }}

layouts/index.txt

@@ -1,31 +1,25 @@
 # Chainguard Academy
 
 > Documentation, tutorials, and resources for Chainguard products and software supply chain security.
-
-## Chainguard Products
-
-{{ range where .Site.RegularPages "Section" "chainguard" -}}
-- [{{ .Title }}]({{ .Permalink | replaceRE "/$" ".md" }}){{ with .Description }}: {{ . }}{{ end }}
-{{ end }}
-
-## Open Source
-
-{{ range where .Site.RegularPages "Section" "open-source" -}}
+{{ range .Site.RegularPages.GroupBy "Section" -}}
+{{- $section := .Key -}}
+{{- $heading := cond (eq $section "") "General Resources" ($section | replaceRE "-" " " | title) -}}
+{{/* Collect only pages the site wants advertised: skip anything marked noindex or excluded from the sitemap. */}}
+{{- $visible := slice -}}
+{{- range .Pages -}}
+{{- $robots := "" -}}
+{{- with .Params.seo }}{{ $robots = .robots | default "" }}{{ end -}}
+{{- $excluded := or (in $robots "noindex") (eq .Params.sitemap_exclude true) -}}
+{{- if not $excluded }}{{ $visible = $visible | append . }}{{ end -}}
+{{- end -}}
+{{- if $visible }}
+## {{ $heading }}
+
+{{ range $visible -}}
 - [{{ .Title }}]({{ .Permalink | replaceRE "/$" ".md" }}){{ with .Description }}: {{ . }}{{ end }}
 {{ end }}
-
-## Software Security
-
-{{ range where .Site.RegularPages "Section" "software-security" -}}
-- [{{ .Title }}]({{ .Permalink | replaceRE "/$" ".md" }}){{ with .Description }}: {{ . }}{{ end }}
-{{ end }}
-
-## Compliance
-
-{{ range where .Site.RegularPages "Section" "compliance" -}}
-- [{{ .Title }}]({{ .Permalink | replaceRE "/$" ".md" }}){{ with .Description }}: {{ . }}{{ end }}
-{{ end }}
-
+{{- end -}}
+{{- end }}
 ## API Reference
 
 - [Chainguard API (Combined)](https://edu.chainguard.dev/api.json): OpenAPI spec for all Chainguard API versions (JSON)