Rad Cult Store Scraper

A serverless application that monitors the Rad Cult BigCartel store for product updates and sends notifications via Telegram when changes are detected. Never miss a release again!

Features

• 🕒 Hourly monitoring of the Rad Cult store
• 🔍 Detects new products
• 📱 Real-time notifications via Telegram
• ☁️ Runs serverlessly on AWS Lambda
• 💾 Tracks product history in S3

Prerequisites

• AWS Account
• AWS SAM CLI installed
• Node.js 20.x
• Telegram Bot Token and Chat ID
• Docker (for local testing)

Setup

1. Clone the repository

   git clone <repository-url>
   cd rad-cult-scraper

2. Create your environment file

   cp example.env.json env.json

   Edit env.json with your actual values:

   • BUCKET_NAME: Your S3 bucket name
   • OBJECT_KEY: JSON file name for storing product data
   • TELEGRAM_CHAT_ID: Your Telegram chat ID
   • TELEGRAM_BOT_TOKEN: Your Telegram bot token

3. Deploy to AWS

   sam build
   sam deploy --guided

   When prompted, provide your Telegram credentials as parameter overrides.

Local Testing

To test the function locally:

sam local invoke ScraperFunction

Architecture

• AWS Lambda: Runs the scraper function every hour
• Amazon S3: Stores product data between runs
• Telegram Bot API: Sends notifications when changes are detected
• Puppeteer: Handles web scraping with headless Chrome
• Cheerio: Parses HTML content

Configuration

SAM Template (template.yaml)

• Defines the Lambda function, S3 bucket, and necessary IAM permissions
• Sets up CloudWatch Events for hourly execution
• Configures environment variables

Environment Variables

• BUCKET_NAME: S3 bucket for storing product data
• OBJECT_KEY: Name of the JSON file in S3
• TELEGRAM_CHAT_ID: Telegram chat/group ID for notifications
• TELEGRAM_BOT_TOKEN: Telegram bot authentication token

Security Notes

• Never commit env.json to version control
• Use AWS Parameter Store or Secrets Manager for production credentials
• Keep your Telegram bot token secure

Contributing

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a new Pull Request

License

MIT License - See LICENSE file for details