Add llms.txt and llms-full.txt support for AI crawlers

[gabinfay]

Nota: This PR replaces an old one where I just shared the llms.txt and llms-full.txt without the script to generate them, now the script is here and could be easily integrated in the CI/CD to automatically update the txt files.

Description

Implements llms-txt.org support to expose Ethereum.org's English documentation to AI crawlers and LLMs.
Adds two statically served files:

• llms.txt: URL index of 256 English documentation pages (~32KB)
• llms-full.txt: Full content of all pages (~2.8MB), cleaned for AI consumption

Includes a script to generate/update both files and a new npm command for ease of use.

Changes:

• public/llms.txt – index of all English docs, categorized by topic
• public/llms-full.txt – full content in markdown format
• src/scripts/generate-llms-files.js – generation script
• package.json – adds npm run generate-llms command

Key features:

• English-only, translations excluded
• Proper llms.txt formatting
• Cleaned markdown (no frontmatter, HTML, or JSX)
• Static serving via Next.js public/ directory
• Categorized by topic (e.g., developers, staking, community)

Files are accessible at:

• https://ethereum.org/llms.txt
• https://ethereum.org/llms-full.txt

Related Issue

N/A

[netlify]

✅ Deploy Preview for ethereumorg ready!

Name	Link
🔨 Latest commit	ff5905e
🔍 Latest deploy log	https://app.netlify.com/projects/ethereumorg/deploys/68909252d1c136000885622c
😎 Deploy Preview	https://deploy-preview-16016--ethereumorg.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
Lighthouse	7 paths audited Performance: 41 (🔴 down 10 from production) Accessibility: 90 (🔴 down 4 from production) Best Practices: 92 (no change from production) SEO: 97 (🔴 down 2 from production) PWA: - View the detailed breakdown and full score reports

---

To edit notification comments on pull requests, go to your Netlify project configuration.