Spider Sitemap Generator

Crawl websites and generate visual, interactive sitemaps with XML export using Spider Cloud — the fastest web crawling infrastructure.

Live Demo: https://sitemap-generator.spider.cloud

Features

• Crawl any website using Spider Cloud API
• Real-time JSONL streaming results
• Supabase authentication (GitHub & Discord)
• Local IndexedDB storage for caching results
• Dark theme matching spider.cloud branding
• Configurable crawl settings (limit, format, request type)
• Cross-app switcher to jump between all Spider Cloud tools with the current URL pre-filled

Getting Started

Prerequisites

• Node.js 18+
• A Spider Cloud account with API credits
• Supabase project for authentication

Setup

1. Clone the repository:

git clone https://github.com/spider-rs/spider-sitemap-generator.git
cd spider-sitemap-generator

2. Install dependencies:

npm install

3. Configure environment variables:

cp .env.local .env

Edit .env and add your Supabase and Spider Cloud credentials:

NEXT_PUBLIC_SUPABASE_URL=your_supabase_url
NEXT_PUBLIC_SUPABASE_ANON_KEY=your_supabase_anon_key
NEXT_PUBLIC_API_URL=https://api.spider.cloud

4. Run the development server:

npm run dev

Open http://localhost:3001 in your browser.

Tech Stack

• Next.js 14 — React framework
• Tailwind CSS — Styling
• shadcn/ui — UI components
• Supabase — Authentication
• Spider Cloud — Web crawling API

Related Spider Cloud Tools

Tool	URL
Archiver	archiver.spider.cloud
Dead Link Checker	dead-link-checker.spider.cloud
A11y Checker	a11y-checker.spider.cloud
Knowledge Base	knowledge-base.spider.cloud
Perf Runner	perf-runner.spider.cloud
Content Translator	content-translator.spider.cloud
Diff Monitor	diff-monitor.spider.cloud
Sitemap Generator	sitemap-generator.spider.cloud
Link Graph	link-graph.spider.cloud

License

MIT