Filter directory pages from llms.txt

[mvvmm]

Summary

Per-product llms.txt files previously included section index pages whose only content is a <DirectoryListing /> component — pages that exist purely as sidebar navigation containers. These pages have no standalone value for an LLM (the llms.txt file itself already serves as a directory), and they were failing the content-start-position audit check because 100% of their rendered content is navigation links.

This adds an isDirectoryOnlyPage() helper that strips imports, component tags, and JSX comments from a page's raw MDX body, then returns true if the page contains a <DirectoryListing /> with ≤ 250 characters of surrounding prose. Any page matching that condition is excluded from the per-product llms.txt filter in getStaticPaths.

• ~415 pure directory index pages are removed from per-product llms.txt outputs
• Pages with real prose alongside <DirectoryListing /> (>250 chars) are preserved
• Pages that don't use <DirectoryListing /> at all are unaffected
• No build time impact — operates on e.body already loaded in memory by getCollection
• The global /llms.txt (product directory) is unchanged

[github-actions]

This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:

Pattern	Owners
*.ts	@cloudflare/content-engineering, @kodster28

[mvvmm]

/bonk please review

[ask-bonk]

LGTM. The implementation is clean and well-documented. The 250-character threshold is a reasonable heuristic for distinguishing navigation-only pages from content pages. All checks pass.

[ask-bonk]

Approved PR #29200. The changes are clean, well-documented, and pass all validation checks.

github run

[github-actions]

Preview URL: https://157db81c.preview.developers.cloudflare.com
Preview Branch URL: https://remove-directory-pages-from-llms-txt.preview.developers.cloudflare.com