A Reddit content crawler with CLI, web, and desktop interface for subreddits and user profiles.
Fork from easy-reddit-downloader - almost entirely rewritten in TypeScript, with dependency injection, and comprehensive testing.
Especially useful for gathering data for machine learning projects, with automatic thumbnail generation for images and videos.
- Media Download — Images (jpg, png, gif, webp), videos (mp4, webm), galleries, and text posts
- Smart Filtering — Content-Type validation prevents invalid downloads (e.g., HTML error pages saved as images)
- Deduplication — SQLite database tracks downloads, automatically skips existing files, and handles missing files
- Duplicate Detection — Perceptual hash (pHash) based detection with multi-frame video analysis
- Web Gallery — Real-time progress monitoring, thumbnail generation, file browsing, and management
- Desktop App — Electron-based desktop version with system tray integration
- Type Safe — Full TypeScript codebase with dependency injection and comprehensive unit tests
- Third-Party Support: Gfycat, Imgur, YouTube (experimental), RedGifs
- Web Interface: Real-time download progress, file browser with thumbnails, lightbox viewer, and log streaming
- Desktop Integration: Runs as a standalone application with auto-assigned ports
- Portable Paths: Stores relative paths for database portability
| Layer | Technology |
|---|---|
| Runtime | Node.js, Electron |
| Frontend | Vue 3, Vite, TypeScript, SCSS |
| Backend | TypeScript, SQLite |
| Build / Packaging | Electron Forge |
| Libraries | tsyringe (DI), sharp (Image processing), ffmpeg (Video processing) |
- Download the latest release from GitHub Releases
- Extract and run the executable
- Follow the interactive prompts
# Clone and install
git clone https://github.com/srad/raddit-downloader.git
cd raddit-downloader
npm install
# Build and run
npm run build
npm startnpm start # Interactive CLI promptsThe CLI will ask you:
- Subreddit or user profile to download from (e.g.,
picsoru/username) - Number of posts (or
allfor unlimited) - Sort method:
Top,New,Hot,Rising,Controversial - Time period:
All,Year,Month,Week,Day,Hour - Run on repeat (continuous monitoring)
npm run web # Web interface at http://localhost:3000Features:
- Real-time download progress per batch
- File browser with thumbnails
- Gallery lightbox viewer
- Log streaming
npm run desktop # Desktop app (Electron)Electron-based desktop version with auto-assigned ports to avoid conflicts.
RadditDownloader uses perceptual hashing (pHash) to detect duplicate and similar media files.
CLI Commands:
# Find duplicates with default threshold (85% similarity)
npm run web -- --find-duplicates
# Find duplicates with custom threshold (lower = less strict)
npm run web -- --find-duplicates=70
# Generate phashes for existing downloads
npm run web -- --generate-phashWeb API:
# Get duplicates via API
GET http://localhost:3000/api/duplicates?threshold=85
# Trigger background phash generation
POST http://localhost:3000/api/duplicates/generate
# Delete a duplicate file
DELETE http://localhost:3000/api/duplicates/:idFor Images:
- Generates a single 64-bit perceptual hash per image
- Hash is invariant to resizing, compression, and minor edits
- Hamming distance used to compare similarity
For Videos:
- Extracts 5 frames at 10%, 30%, 50%, 70%, and 90% of video duration
- Generates phash for each frame
- Uses voting system: duplicates if 3+ frames match (60% confidence)
- Node.js (v18 or higher recommended)
- npm (comes with Node.js)
git clone https://github.com/srad/raddit-downloader.git
cd raddit-downloaderInstall all project dependencies (including devDependencies for Electron and testing):
npm installYou can run the application in different modes during development:
CLI Mode (Interactive):
npm startWeb Interface:
Starts the backend server and web UI at http://localhost:3000.
npm run webDesktop App (Electron): Builds the frontend and launches the Electron desktop application.
npm run desktopThis project uses Electron Forge to package the application.
Create Distributables:
This command compiles the TypeScript code, builds the Vue frontend, and packages the app for your current platform (e.g., .exe for Windows, .deb/.rpm for Linux).
npm run makeThe output files will be located in the out/ directory:
- Windows:
out/make/squirrel.windows/x64/ - Linux:
out/make/deb/x64/orout/make/rpm/x64/
npm test # Run all tests
npm test -- --selectProjects=unit # Unit tests only
npm test -- --selectProjects=e2e # E2E tests only
npm run test:watch # Watch mode
npm run test:coverage # Coverage reportUse Electron Forge to create platform-specific distributables (exe, deb, rpm, zip):
npm run make- Implement the
Downloaderinterface insrc/services/download/ - Register in
src/services/download/index.ts - Add tests in
__tests__/
@injectable()
export class MyDownloader implements Downloader {
canHandle(post: RedditPost): boolean {
return post.domain.includes('example.com');
}
async download(post: RedditPost, targetDir: string, filenameBase: string): Promise<string> {
// Download logic
}
}- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Write tests for your changes
- Ensure all tests pass (
npm test) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
Original project by Joseph R. Cox This fork by srad