feat: add custom encrypted GCS backup scripts

Add client-side encryption scripts for uploading backups to GCS using
age encryption. This provides an alternative for users who need to
encrypt data before it leaves their servers.

Scripts included:
- encrypted-upload.sh: compress, encrypt, and upload to GCS
- encrypted-download.sh: download, decrypt, and extract
- encrypted-list.sh: list remote backups with decrypted metadata
- encrypted-delete.sh: delete backups from GCS
- setup.sh: one-time installation and key generation
- config-encrypted-gcs.yml: sample configuration
- README.md: comprehensive documentation

Co-Authored-By: Claude Opus 4.5 <[email protected]>

Author: ns-janandaram

scripts/custom-encrypted-gcs/README.md

@@ -0,0 +1,342 @@
+# Custom Encrypted GCS Backup for clickhouse-backup
+
+Client-side encryption for ClickHouse backups before uploading to Google Cloud Storage.
+
+This solution uses [age](https://github.com/FiloSottile/age) encryption to encrypt backups locally before uploading to GCS, ensuring data is encrypted at rest with keys you control.
+
+## Features
+
+- **Client-side encryption** - Data is encrypted before leaving your server
+- **Age encryption** - Modern, secure, and simple encryption tool
+- **Key control** - You manage your own encryption keys
+- **Compatible** - Works with standard clickhouse-backup commands
+
+## Prerequisites
+
+| Tool | Installation | Purpose |
+|------|--------------|---------|
+| [age](https://github.com/FiloSottile/age) | `apt install age` / `brew install age` | Encryption |
+| [gsutil](https://cloud.google.com/sdk/docs/install) | Google Cloud SDK | GCS operations |
+| [jq](https://stedolan.github.io/jq/) | `apt install jq` / `brew install jq` | JSON processing |
+
+## Quick Start
+
+```bash
+# 1. Clone or navigate to the scripts directory
+cd scripts/custom-encrypted-gcs
+
+# 2. Run the setup script
+./setup.sh
+
+# 3. Configure your GCS bucket (add to ~/.bashrc or /etc/environment)
+export GCS_ENCRYPTED_BUCKET=gs://your-bucket/clickhouse-backups
+
+# 4. Authenticate with GCS
+gcloud auth application-default login
+# Or for service accounts:
+# export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
+
+# 5. Test the setup
+clickhouse-backup create test_backup
+clickhouse-backup upload test_backup
+clickhouse-backup list remote
+clickhouse-backup download test_backup
+clickhouse-backup delete remote test_backup
+```
+
+## Installation
+
+### Automated Setup
+
+```bash
+./setup.sh
+```
+
+This will:
+- Check for required dependencies
+- Create directories (`/opt/clickhouse-backup/scripts`, `/etc/clickhouse-backup`)
+- Generate an encryption key
+- Install scripts and configuration
+
+### Manual Setup
+
+1. Install dependencies:
+   ```bash
+   # Ubuntu/Debian
+   apt install age jq
+
+   # macOS
+   brew install age jq
+
+   # Install Google Cloud SDK
+   # https://cloud.google.com/sdk/docs/install
+   ```
+
+2. Generate encryption key:
+   ```bash
+   mkdir -p /etc/clickhouse-backup
+   age-keygen -o /etc/clickhouse-backup/encryption.key
+   chmod 600 /etc/clickhouse-backup/encryption.key
+   ```
+
+3. Copy scripts:
+   ```bash
+   mkdir -p /opt/clickhouse-backup/scripts
+   cp encrypted-*.sh /opt/clickhouse-backup/scripts/
+   chmod +x /opt/clickhouse-backup/scripts/*.sh
+   ```
+
+4. Copy configuration:
+   ```bash
+   cp config-encrypted-gcs.yml /etc/clickhouse-backup/config.yml
+   ```
+
+## Configuration
+
+### clickhouse-backup config (`/etc/clickhouse-backup/config.yml`)
+
+```yaml
+general:
+  remote_storage: custom
+  backups_to_keep_local: 2
+  backups_to_keep_remote: 5
+
+custom:
+  upload_command: "/opt/clickhouse-backup/scripts/encrypted-upload.sh {{.backup_name}}"
+  download_command: "/opt/clickhouse-backup/scripts/encrypted-download.sh {{.backup_name}}"
+  list_command: "/opt/clickhouse-backup/scripts/encrypted-list.sh"
+  delete_command: "/opt/clickhouse-backup/scripts/encrypted-delete.sh {{.backup_name}}"
+  command_timeout: "4h"
+```
+
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `GCS_ENCRYPTED_BUCKET` | `gs://your-bucket/clickhouse-backups` | GCS bucket and path for backups |
+| `ENCRYPTION_KEY_FILE` | `/etc/clickhouse-backup/encryption.key` | Path to age encryption key |
+| `CLICKHOUSE_BACKUP_PATH` | `/var/lib/clickhouse/backup` | Local backup directory |
+
+Set these in `/etc/environment`, `~/.bashrc`, or your systemd unit file:
+
+```bash
+# /etc/default/clickhouse-backup
+GCS_ENCRYPTED_BUCKET=gs://my-bucket/clickhouse-backups
+ENCRYPTION_KEY_FILE=/etc/clickhouse-backup/encryption.key
+```
+
+## Usage
+
+### Create and Upload Backup
+
+```bash
+# Create local backup
+clickhouse-backup create my_backup
+
+# Upload with encryption
+clickhouse-backup upload my_backup
+
+# Or do both in one command
+clickhouse-backup create_remote my_backup
+```
+
+### List Remote Backups
+
+```bash
+clickhouse-backup list remote
+```
+
+### Download and Restore
+
+```bash
+# Download and decrypt
+clickhouse-backup download my_backup
+
+# Restore
+clickhouse-backup restore my_backup
+```
+
+### Delete Remote Backup
+
+```bash
+clickhouse-backup delete remote my_backup
+```
+
+## Scripts Reference
+
+### encrypted-upload.sh
+
+Compresses the backup directory, encrypts it with age, and uploads to GCS.
+
+```
+Usage: encrypted-upload.sh <backup_name>
+
+Uploads:
+  - <backup_name>.tar.gz.age      # Encrypted backup archive
+  - <backup_name>.metadata.json.age  # Encrypted metadata (for list command)
+```
+
+### encrypted-download.sh
+
+Downloads encrypted backup from GCS, decrypts, and extracts.
+
+```
+Usage: encrypted-download.sh <backup_name>
+
+Downloads and extracts to:
+  /var/lib/clickhouse/backup/<backup_name>/
+```
+
+### encrypted-list.sh
+
+Lists all backups on GCS by reading and decrypting metadata files.
+
+```
+Usage: encrypted-list.sh
+
+Output: JSON objects (one per line) compatible with clickhouse-backup
+```
+
+### encrypted-delete.sh
+
+Deletes a backup from GCS (both archive and metadata).
+
+```
+Usage: encrypted-delete.sh <backup_name>
+```
+
+## GCS Bucket Structure
+
+```
+gs://your-bucket/clickhouse-backups/
+├── backup_2024_01_15.tar.gz.age           # Encrypted backup archive
+├── backup_2024_01_15.metadata.json.age    # Encrypted metadata
+├── backup_2024_01_16.tar.gz.age
+├── backup_2024_01_16.metadata.json.age
+└── ...
+```
+
+## Security Considerations
+
+### Key Management
+
+- **Back up your encryption key** - Without it, backups cannot be recovered
+- **Restrict key permissions** - `chmod 600 /etc/clickhouse-backup/encryption.key`
+- **Consider key rotation** - Periodically generate new keys for new backups
+- **Use a secrets manager** - For production, consider HashiCorp Vault, GCP Secret Manager, etc.
+
+### Key Backup
+
+```bash
+# Display key for backup
+sudo cat /etc/clickhouse-backup/encryption.key
+
+# The output looks like:
+# # created: 2024-01-15T10:30:00Z
+# # public key: age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+# AGE-SECRET-KEY-1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+```
+
+Store this securely (password manager, offline storage, etc.).
+
+### GCS Permissions
+
+Minimum required permissions for the service account:
+- `storage.objects.create` - Upload backups
+- `storage.objects.get` - Download backups
+- `storage.objects.list` - List backups
+- `storage.objects.delete` - Delete backups
+
+Example IAM role: `roles/storage.objectAdmin` (scoped to the backup bucket)
+
+## Limitations
+
+| Limitation | Description |
+|------------|-------------|
+| No resume support | If upload fails, you must restart from the beginning |
+| No incremental backups | `--diff-from` flag won't work as expected |
+| No parallel uploads | Single-stream upload (limited by encryption pipe) |
+| Metadata decryption | List command must decrypt each metadata file (slower for many backups) |
+
+## Troubleshooting
+
+### "Encryption key file not found"
+
+```bash
+# Check if key exists
+ls -la /etc/clickhouse-backup/encryption.key
+
+# Generate if missing
+age-keygen -o /etc/clickhouse-backup/encryption.key
+chmod 600 /etc/clickhouse-backup/encryption.key
+```
+
+### "gsutil: command not found"
+
+```bash
+# Install Google Cloud SDK
+curl https://sdk.cloud.google.com | bash
+exec -l $SHELL
+gcloud init
+```
+
+### "AccessDeniedException: 403"
+
+```bash
+# Check authentication
+gcloud auth list
+
+# Re-authenticate
+gcloud auth application-default login
+
+# Or check service account permissions
+gsutil iam get gs://your-bucket
+```
+
+### "age: error: no identity matched any of the recipients"
+
+The encryption key doesn't match. Ensure you're using the same key that was used to encrypt:
+
+```bash
+# Check which key is configured
+echo $ENCRYPTION_KEY_FILE
+cat $ENCRYPTION_KEY_FILE | head -3
+```
+
+### List command returns empty
+
+```bash
+# Check if backups exist in GCS
+gsutil ls gs://your-bucket/clickhouse-backups/
+
+# Test decryption manually
+gsutil cat gs://your-bucket/clickhouse-backups/backup_name.metadata.json.age | \
+  age --decrypt --identity /etc/clickhouse-backup/encryption.key
+```
+
+### Slow uploads/downloads
+
+For large backups, consider:
+- Using `gsutil -o GSUtil:parallel_composite_upload_threshold=150M` for parallel uploads
+- Increasing `command_timeout` in config
+- Running from a VM in the same GCP region as your bucket
+
+## Alternative: Using GPG Instead of Age
+
+If you prefer GPG over age, modify the scripts:
+
+```bash
+# Generate key
+openssl rand -base64 32 > /etc/clickhouse-backup/encryption.key
+
+# Encrypt (replace age command)
+gpg --batch --yes --passphrase-file "$ENCRYPTION_KEY_FILE" \
+    --symmetric --cipher-algo AES256
+
+# Decrypt (replace age command)
+gpg --batch --yes --passphrase-file "$ENCRYPTION_KEY_FILE" --decrypt
+```
+
+## License
+
+These scripts are provided as part of clickhouse-backup examples. Use at your own risk.

scripts/custom-encrypted-gcs/config-encrypted-gcs.yml

@@ -0,0 +1,39 @@
+# clickhouse-backup configuration for encrypted GCS storage
+# Copy this to /etc/clickhouse-backup/config.yml and customize
+
+general:
+  remote_storage: custom
+  backups_to_keep_local: 2
+  backups_to_keep_remote: 5
+  log_level: info
+  allow_empty_backups: false
+
+clickhouse:
+  username: default
+  password: ""
+  host: localhost
+  port: 9000
+  data_path: /var/lib/clickhouse
+  skip_tables:
+    - system.*
+    - INFORMATION_SCHEMA.*
+    - information_schema.*
+  timeout: 5m
+  secure: false
+  skip_verify: false
+
+custom:
+  # Path to scripts - adjust based on your installation
+  upload_command: "/opt/clickhouse-backup/scripts/encrypted-upload.sh {{.backup_name}}"
+  download_command: "/opt/clickhouse-backup/scripts/encrypted-download.sh {{.backup_name}}"
+  list_command: "/opt/clickhouse-backup/scripts/encrypted-list.sh"
+  delete_command: "/opt/clickhouse-backup/scripts/encrypted-delete.sh {{.backup_name}}"
+  command_timeout: "4h"
+
+# Environment variables to set (or export before running clickhouse-backup):
+#
+# GCS_ENCRYPTED_BUCKET=gs://your-bucket/clickhouse-backups
+# ENCRYPTION_KEY_FILE=/etc/clickhouse-backup/encryption.key
+# CLICKHOUSE_BACKUP_PATH=/var/lib/clickhouse/backup
+#
+# You can also set these in /etc/default/clickhouse-backup or your systemd unit file