Skip to content

Add unified sitemap and robots.txt#13

Merged
svekars merged 4 commits intomainfrom
add-unified-sitemap
Mar 3, 2026
Merged

Add unified sitemap and robots.txt#13
svekars merged 4 commits intomainfrom
add-unified-sitemap

Conversation

@svekars
Copy link
Contributor

@svekars svekars commented Feb 27, 2026

Adds sitemap generation to meta-pytorch.org, aggregating all project documentation URLs into a unified sitemap.

Why Sitemaps Matter

  • SEO: Search engines use sitemaps to discover and index pages faster. The dates help prioritize fresh content.
  • LLMO: AI assistants and RAG pipelines use sitemaps to discover and index documentation.

Changes

  • sitemap.xml - Index pointing to all project sitemaps
  • sitemap-unified.xml - All URLs merged into one file (for LLMs)
  • robots.txt - Points crawlers to sitemaps
  • scripts/generate_sitemap.py - Sitemap generation script
  • projects.yaml - Added sitemap field for custom paths

Test Plan

python scripts/generate_sitemap.py --validate
make html

@meta-cla meta-cla bot added the cla signed label Feb 27, 2026
@svekars svekars merged commit e045f8c into main Mar 3, 2026
4 checks passed
@svekars svekars deleted the add-unified-sitemap branch March 3, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants