MongoDB Dumper

A robust Go application that automates MongoDB database backups to Backblaze B2 Cloud Storage (via S3 API) on a scheduled basis. Designed for reliability, flexibility, and seamless integration with modern cloud-native environments.

🚀 Features

• Automated Backups: Schedule backups hourly or at custom intervals
• Multi-Environment Support: Separate configurations for staging and production
• Cloud Storage Integration: Direct upload to Backblaze B2 via S3-compatible API
• Kubernetes Integration: Ready-to-use deployment manifests for cloud environments
• Native BSON Format: Backups stored in standard MongoDB format for easy restoration
• Flexible Configuration: Configure via environment variables, command-line flags, or config files
• Retention Policies: Configurable backup retention and cleanup strategies
• Detailed Logging: Rich logging with different formats (JSON, pretty, compact, console)

📋 Requirements

• Go 1.24+ (for building from source)
• MongoDB Tools (mongodump in PATH)
• Backblaze B2 Account with S3-compatible API enabled
• Kubernetes Cluster (for production deployment)

⚙️ Configuration

The application can be configured using environment variables, command-line flags, or by providing a configuration file:

Environment Variable	Flag	Description	Required	Default
MONGO_URI	--mongo-uri	MongoDB connection string URI	Yes	-
MONGO_DATABASE	--database	MongoDB database to backup (empty = all DBs)	No	(all databases)
ENVIRONMENT	--env	Environment (staging or production)	No	-
S3_ENDPOINT	--s3-endpoint	S3 endpoint URL for Backblaze	Yes	-
S3_REGION	--s3-region	S3 region	Yes	-
S3_BUCKET	--s3-bucket	S3 bucket name	Yes	-
S3_ACCESS_KEY	--s3-access-key	S3 access key	Yes	-
S3_SECRET_KEY	--s3-secret-key	S3 secret key	Yes	-
TEMP_DIR	--temp-dir	Temporary directory for backups	No	/tmp/mongodb-dumps
BACKUP_INTERVAL	--interval	Backup interval (1h, 6h, 24h)	No	(one-time run)
ONE_TIME	--one-time	Run a single backup and exit	No	false
LOG_FORMAT	--log-format	Log format: json, console, pretty, compact	No	pretty
-	--env-file	Path to .env file for environment variables	No	.env

🏃 Running Locally

From Source

# Clone the repository
git clone https://github.com/yourusername/mongodb-dumper.git
cd mongodb-dumper

# Build the application
go build -o dumper ./cmd/dumper

# Run with configuration
./dumper \
  --mongo-uri="mongodb://username:password@hostname:port" \
  --env=staging \
  --s3-endpoint="https://s3.backblazeb2.com" \
  --s3-region="us-west-001" \
  --s3-bucket="your-backup-bucket" \
  --s3-access-key="your-access-key" \
  --s3-secret-key="your-secret-key" \
  --log-format=pretty \
  --interval=1h

Using Configuration File

You can also create a .env file:

MONGO_URI=mongodb://username:password@hostname:port
ENVIRONMENT=staging
S3_ENDPOINT=https://s3.backblazeb2.com
S3_REGION=us-west-001
S3_BUCKET=your-backup-bucket
S3_ACCESS_KEY=your-access-key
S3_SECRET_KEY=your-secret-key
LOG_FORMAT=pretty
BACKUP_INTERVAL=1h

Then run:

./dumper --env-file=.env

🐳 Docker

Build and run using Docker:

# Build the Docker image
docker build -t mongodb-dumper:latest .

# Run the Docker container
docker run -d \
  -e MONGO_URI="mongodb://username:password@hostname:port" \
  -e ENVIRONMENT="staging" \
  -e S3_ENDPOINT="https://s3.backblazeb2.com" \
  -e S3_REGION="us-west-001" \
  -e S3_BUCKET="your-backup-bucket" \
  -e S3_ACCESS_KEY="your-access-key" \
  -e S3_SECRET_KEY="your-secret-key" \
  -e LOG_FORMAT="pretty" \
  -e BACKUP_INTERVAL="1h" \
  mongodb-dumper:latest

Using Docker Compose

# docker-compose.yml
version: '3'
services:
  mongodb-dumper:
    build: .
    restart: unless-stopped
    environment:
      - MONGO_URI=mongodb://username:password@mongodb:27017/database?authSource=admin
      - ENVIRONMENT=staging
      - S3_ENDPOINT=https://s3.backblazeb2.com
      - S3_REGION=us-west-001
      - S3_BUCKET=your-backup-bucket
      - S3_ACCESS_KEY=your-access-key
      - S3_SECRET_KEY=your-secret-key
      - BACKUP_INTERVAL=1h

Run with:

docker-compose up -d

☸️ Kubernetes Deployment

This application can be deployed to Kubernetes using the provided manifests in the k8s directory.

# Create the required secrets
kubectl create secret generic mongodb-dumper-secrets \
  --from-literal=MONGO_URI="mongodb://username:password@hostname:port" \
  --from-literal=S3_ACCESS_KEY="your-access-key" \
  --from-literal=S3_SECRET_KEY="your-secret-key"

# Deploy the application
kubectl apply -f k8s/mongodb-dumper.yaml

The provided Kubernetes manifest includes:

• A CronJob to run backups on schedule
• A ConfigMap for non-sensitive configuration
• A Secret for sensitive credentials
• Resource limits and requests

📦 Backup Details

Backup Format

The backups are stored in MongoDB's archive format (BSON), compressed as ZIP files, which can be easily restored using the mongorestore command.

Backup Naming Convention

Backups are stored with the following naming convention:

{environment}/{date}/{database}-{environment}-{timestamp}.zip

For example:

• staging/2023-04-15/my-database-staging-2023-04-15T12-00-00Z.zip
• production/2023-04-15/my-database-production-2023-04-15T12-00-00Z.zip

Backup Restoration

To restore a database from backup:

# Download the backup from Backblaze B2
# Unzip the backup archive
unzip my-database-staging-2023-04-15T12-00-00Z.zip -d ./extracted-backup

# Then restore using mongorestore:
mongorestore --uri="mongodb://username:password@hostname:port" ./extracted-backup

# Restore a specific collection
mongorestore --uri="mongodb://username:password@hostname:port/database?authSource=admin" \
  --nsInclude="database.collection" ./extracted-backup

🧪 Testing

Run the test suite:

go test -v ./...

Run integration tests (requires local MongoDB instance):

go test -v -tags=integration ./...

🔒 Security Considerations

• Always use MongoDB connection strings with minimal required permissions
• Store sensitive credentials as Kubernetes secrets or environment variables
• Consider using service accounts with restricted permissions for S3 access
• Encrypt backups at rest by enabling server-side encryption in your S3 bucket

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

📞 Support

For support, please open an issue on the GitHub repository or contact the maintainers directly.