A GitHub Action that automatically crawls websites using their sitemap and generates a single llms.txt file containing clean, markdown-formatted content from every page. Designed to create AI-friendly content for Large Language Models (LLMs) following the proposed llms.txt standard.

llms.txt is a proposed web standard by Jeremy Howard (Answer.AI) that provides AI-friendly website content in a structured format. It helps Large Language Models understand and extract your website's most important information efficiently.

• AI Search Optimization: Optimizes your content for AI-powered search engines like Perplexity, ChatGPT, and Claude
• Better Attribution: Ensures proper citation when AI models reference your content
• Context Window Friendly: Overcomes LLM token limitations with curated, essential content
• Controlled AI Interaction: You decide what content AI models should prioritize
• Future-Proof: Positions your website for the AI-driven web

• Perplexity - AI search engine
• ElevenLabs - AI voice technology
• FastHTML - Web framework documentation
• Answer.AI - AI research company
• Mintlify - Documentation platform

• Jina AI (Default): Free content extraction via Jina AI Reader API
• Firecrawl: Advanced crawling with Firecrawl API for complex websites

• Sitemap Discovery: Auto-detects sitemaps from robots.txt, /sitemap.xml, and /sitemap_index.xml
• Asynchronous Processing: Parallel content extraction for improved performance
• Clean Markdown Output: Converts HTML content to markdown format
• Aggregated Output: Combines all pages into a single llms.txt file

• GitHub Actions Integration: Runs as a composite action
• Python 3.11: Built with modern Python and async/await
• Dependencies: Uses httpx, beautifulsoup4, lxml, and firecrawl-py
• Error Handling: Graceful handling of failed requests and missing content

Create .github/workflows/generate-llms-txt.yml in your repository:

name: Generate AI-Optimized llms.txt
permissions:
  contents: write
on:
  push:
    branches: [ main ]
  schedule:
    - cron: '0 2 * * 0' # Weekly updates

jobs:
  generate-llms-txt:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Generate llms.txt with Jina AI
        uses: kevinnkansah/[email protected]
        with:
          domain: https://your-website.com
          outputFile: public/llms.txt
          # backend: jina  # Default, free option

      - name: Commit and push llms.txt
        uses: EndBug/add-and-commit@v9
        with:
          author_name: 'AI Bot'
          author_email: '[email protected]'
          add: 'public/llms.txt'
          message: 'feat: update llms.txt for AI optimization'

For complex websites requiring advanced crawling:

- name: Generate llms.txt with Firecrawl
  uses: kevinnkansah/[email protected]
  with:
    domain: https://your-complex-website.com
    outputFile: docs/llms.txt
    backend: firecrawl
    firecrawl_api_key: ${{ secrets.FIRECRAWL_API_KEY }}

This action first attempts to find your sitemap by checking /robots.txt and common paths like /sitemap.xml. It then parses the sitemap(s) to get a list of all page URLs. For each URL, it uses the selected backend to fetch the content as clean markdown:

• Jina Backend (Default): Uses the free Jina AI Reader API (https://r.jina.ai/)
• Firecrawl Backend: Uses the Firecrawl API for more advanced crawling capabilities

Finally, it aggregates the content from all pages into the specified outputFile.

- name: Generate llms.txt with Jina
  uses: kevinnkansah/[email protected]
  with:
    domain: https://dewflow.xyz
    outputFile: public/llms.txt
    # backend: jina  # Optional, this is the default

- name: Generate llms.txt with Firecrawl
  uses: kevinnkansah/[email protected]
  with:
    domain: https://dewflow.xyz
    outputFile: public/llms.txt
    backend: firecrawl
    firecrawl_api_key: ${{ secrets.FIRECRAWL_API_KEY }}

• Documentation Sites: API docs, technical guides, knowledge bases
• E-commerce: Product catalogs, store policies, FAQ sections
• Content Publishers: Blogs, news sites, educational content
• SaaS Companies: Feature documentation, help centers
• Business Websites: Company info, services, contact details
• Educational Sites: Course materials, research papers

• Better AI Citations: Content gets properly attributed in AI responses
• Enhanced Discoverability: Improved visibility in AI-powered search
• Structured Content: Pre-formatted content for AI consumption
• Content Control: Curated content selection for AI models
• Standard Compliance: Follows the proposed llms.txt specification

- name: Generate Product Catalog for AI
  uses: kevinnkansah/[email protected]
  with:
    domain: https://mystore.com
    outputFile: ai/product-catalog.txt
    backend: firecrawl  # Better for complex product pages
    firecrawl_api_key: ${{ secrets.FIRECRAWL_API_KEY }}

- name: Generate API Docs for AI
  uses: kevinnkansah/[email protected]
  with:
    domain: https://docs.myapi.com
    outputFile: public/llms.txt

strategy:
  matrix:
    locale: [en, es, fr, de]
steps:
  - name: Generate llms.txt for ${{ matrix.locale }}
    uses: kevinnkansah/[email protected]
    with:
      domain: https://mysite.com/${{ matrix.locale }}
      outputFile: public/llms-${{ matrix.locale }}.txt

This project uses uv for package management and pre-commit with commitizen to enforce conventional commit messages.

1. Clone the repository
2. Install dependencies:

   uv pip install -e .[dev]

3. Activate pre-commit hooks:

   uv run pre-commit install --hook-type commit-msg

4. Make your changes
5. Commit your work using the guided prompt:

   uv run cz commit

Your contributions will be automatically versioned and released upon merging to main.

This project is licensed under the MIT License. See the LICENSE file for details.